Factorial invariance across gender of a perceived ICT literacy scale

Learning and Individual Differences 41 (2015) 79–85

Contents lists available at ScienceDirect

Learning and Individual Differences journal homepage: www.elsevier.com/locate/lindif

Factorial invariance across gender of a perceived ICT literacy scale Wilfred W.F. Lau a,⁎, Allan H.K. Yuen b a b

Department of Curriculum and Instruction, The Chinese University of Hong Kong, Hong Kong, China Faculty of Education, The University of Hong Kong, Hong Kong, China

a r t i c l e

i n f o

Article history: Received 28 May 2014 Received in revised form 11 January 2015 Accepted 10 June 2015 Keywords: Factorial invariance Measurement invariance Gender ICT literacy Secondary school education

a b s t r a c t This study used the framework of multiple-group confirmatory factor analysis (MGCFA) to test the factorial invariance (configural, measurement, and structural invariance) of a newly developed three-factor, 17-item perceived information and communication technology (ICT) literacy scale (3F-PICTLS) across gender, which includes the three subscales of information literacy (information), internet literacy (communication), and computer literacy (technology). Using a stratified random sample of 825 secondary school students (396 males) with ages ranging from 11 to 16 (mean = 13.16, SD = .773), the scale showed configural and partial measurement invariance but not structural invariance across gender. These findings highlight the importance of measurement invariance as a methodological challenge for researchers who attempt to make meaningful comparisons and interpretations across gender in a variety of contexts in the scholarship of ICT in education, which is largely ignored in existing literature. From a practical standpoint, we discuss the implications for teachers to assess and promote ICT literacy for students of different genders. Taken together, this study provides new methodological and pragmatic insights into a greater understanding of the issue of gender differences in ICT literacy for researchers and practitioners. © 2015 Elsevier Inc. All rights reserved.

1. Introduction The use of information and communication technology (ICT) in education can enhance learning and teaching, and increase connections and exchange between school and home (Bransford, Brown, Cocking, Donovan, & Pellegrino, 2000; Ezziane, 2007; Kelly-Salinas, 2000). However, since using ICT in education requires certain degree of ICT literacy and devices to access, such as computers and internet access lines, the differences in ICT literacy between demographic groups have been the focus of research in recent years (Hohlfeld, Ritzhaupt, Barron, & Kemker, 2008; Kim, Kil, & Shin, 2014; Luu & Freeman, 2011). Gender among others remains a salient theme in ICT literacy research. Research on gender differences in ICT literacy has extra significance over the years since females consistently participate less than males in the science, technology, engineering, and mathematics (STEM) fields in school and workplace settings. For example, in the U.S., it was reported that there were only 19% female AP Computer Science test-takers in 2013, 12% female Computer Science undergraduate degree recipients at major research universities in 2012, and 26% female computing professionals in workforce in 2013 (NCWIT, 2014). This gender digital divide is prevalent in schools and corporations where women are underrepresented in these localities (Cooper, 2006). Even after more than several decades of efforts to narrow the digital divide, the gender gap still continues to persist. The WGBH ⁎ Corresponding author. E-mail address: wwfl[email protected] (W.W.F. Lau).

http://dx.doi.org/10.1016/j.lindif.2015.06.001 1041-6080/© 2015 Elsevier Inc. All rights reserved.

Educational Foundation and the Association for Computing Machinery (2009) reported that college-bound females, regardless of race and ethnicity, show significantly less interest than males in computing. Females tend to associate computing with “typing”, “math”, and “boredom” while for males, they are more inclined to associate computing with “video games”, “design”, “electronics”, “solving problems”, and “interesting”. In fact, new manifestations of the divide were found in the use rather than access of technology between the genders (Lim & Meier, 2011). While the gender issue is complex and multi-faceted, relevant research can shed light on our understanding of the role that gender plays in ICT literacy, particularly within the background of measurement invariance, and provides possible avenues for us to approach the gender digital divide problem from a quantitative methodological perspective that aims to uncover some causes behind gender differences (Boeve-de Pauw, Jacobs, & Van Petegem, 2012). ICT literacy has been defined and conceptualized using various terminology and frameworks in the literature (see Section 2 for more details). In terms of measurement, self-report remains a common and valid method for assessing ICT literacy (Zelman, Shmis, Avdeeva, Vasiliev, & Froumin, 2011), and youth in school still constitute the major subjects in many studies as they consistently and heavily engage in ICT use. Against this background, this study aimed to test whether a self-reported perceived ICT literacy scale is invariant across gender in a group of adolescents at three levels of invariance: configural invariance, measurement invariance, and structural invariance, which are collectively known as factorial invariance (Dimitrov, 2010). While a

80

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85

review of literature shows that there are a variety of definitions of ICT literacy, this study defined it as follows: “ICT literacy is using digital technology, communication tools, and/or networks to access, manage, integrate, evaluate, and create information in order to function in a knowledge society” (ICT Literacy Panel, 2007, p. 2). Testing factorial invariance has become an increasingly important procedure for psychometric evaluation of measurement scale since it ensures that individuals from different groups ascribe the same meaning to the scale items and any group differences are the result of genuine differences across groups (Wu, Li, & Zumbo, 2007). 2. Gender differences in ICT literacy There has been ongoing research on the gender issue in ICT literacy examining the traditional view that males tend to be more ICT literate than females. Numerous studies have investigated the issue based on different conceptualizations and measures of ICT literacy, where literacy has been understood to include abilities, knowledge, skills, and competencies (Bawden, 2001). Voogt (1987) administered a Dutch version called “Computer Alfabetisme Schalen Twente” (CAST) of the Minnesota Computer Literacy Awareness Assessment to a group of secondary school students aged 12 to 16. In the study, computer literacy referred to the knowledge students displayed in a cognitive test comprising items from domains of programming and algorithms, software and data processing, and computer mystique and applications. Boys outperformed girls on the cognitive test of the CAST (p b .001). Tsai and Tsai (2010) developed an Internet Self-Efficacy Scale (ISES) to examine junior high school students' internet self-efficacy including the online exploration (explorative ISE) and online communication (communicative ISE) dimensions. The former subscale measured students' perceived ability to navigate or search information on the internet whereas the latter subscale assessed students' perceived ability to communicate through the internet. While the study found no statistically significant difference between the genders in the online exploration subscale, female students scored significantly higher (t = 2.055, p b .05) than male students in the online communication subscale. Zhao, Lu, Huang, and Wang (2010) adapted items from a General Internet Self-Efficacy (GISE) scale created by Hsu and Chiu (2004) to assess internet self-efficacy of high school students. Internet self-efficacy was conceptualized as students' self-assessed ability to accomplish some tasks using the internet. They showed that male students had higher internet self-efficacy than did female students (t = 10.649, p b 0.000), providing some evidence of inequality in internet skills. Hatlevik and Christophersen (2013) found no gender differences in digital competence and argued that the general assumption that boys are more digitally competent than girls should be reconsidered. Digital competence in their study was defined as students' ability to access, process, evaluate, produce, and communicate information with the aid of technology. Using the self-assessment scale about ICT use in the Program for International Student Assessment (PISA) surveys in 2003 and 2006, Zhong (2011) found that boys reported higher digital skills than did girls in the two years and digital skills here referred to students' perceived ability to finish some designated tasks on computers and the internet. Whereas the gender differences reported in the aforementioned studies may be due to the different measures that were used by the researchers, these studies appear not to sufficiently address the fundamental issue concerning measurement invariance when it comes to mean comparisons across groups. This could possibly be one of the reasons as to why mixed results are reported. The joint committee of the American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME) published a book entitled ‘Standards for Educational and Psychological Testing’ in 1999, which provides a sensible set of psychological test guidelines that were endorsed by major professional associations. The AERA et al.

(1999) suggested that validity is the evidence for inferences made about a test score, and there are three types of evidence, namely construct-related, criterion-related, and content-related. They also recommended 20 standards for reliability (AERA et al., 1999, pp. 31–36). Measurement instruments have been developed with reference to these guidelines. However, as Vandenberg and Lance (2000) noted, the prevailing focus on the measurement properties of observed variables based on their reliability and validity has not adequately dealt with the issue such as whether respondents from different demographic background and cultures interpret a given measure in a conceptually similar manner (p. 5). It has been advocated that the invariant properties of a measurement instrument have to be verified before it is administered to individuals, so that any differences of the latent means of constructs across groups stem from genuine differences but not methodological artifacts (Boeve-de Pauw et al., 2012). Dimitrov (2010) discussed the generalizability aspect of validity, which is related to whether properties and interpretations of scores can be generalized across population groups, settings, and tasks, and pointed out that factorial invariance is required to achieve this aspect of validity. Nimon and Reio (2011) also argued that ignoring measurement invariance could have significant implications for quantitative theory building and practices. It is equally inappropriate to recommend practices based on theory that is not tested for measurement invariance. These studies highlighted the importance of measurement invariance as a methodological challenge for researchers who attempt to make meaningful comparisons and interpretations across groups in a variety of contexts. 3. Method and results 3.1. Measure This study tested the factorial invariance properties of a perceived ICT literacy scale. The new scale (3F-PICTLS) (see Appendix A), which consists of 17 items with three subscales including information literacy (information), internet literacy (communication), and computer literacy (technology), was developed and validated in a prior study (Lau & Yuen, 2014). Findings of the study showed that the scale was internally consistent with Cronbach's alpha values of the factors ranging from .856 to .906 in exploratory factor analysis (EFA) in a calibration sample (n = 413) and from .844 to .908 in confirmatory factor analysis (CFA) in a validation sample (n = 386) respectively. Whereas there is ongoing discussion regarding how EFA should be conducted (Costello & Osborne, 2005; Schmitt, 2011), we used the principal component extraction method followed by a promax rotation in our EFA since our aim was to extract as much total variance as possible from the set of the manifest variables with the minimum number of dimensions, and it was expected that the latent factors were correlated (Kaplan, 2009; Widaman, 2007). The factor structure of the scale was supported through CFA (χ2/df = 2.244, CFI = .964, TLI = .958, and RMSEA = .057). Face, content, convergent, and discriminant validity were also affirmed. Face validity from the perspective of students and content validity as determined by the authors and teachers were considered good. Convergent validity of the scale was established based on the following conditions: (1) The factor loadings of most items were at least .7 on its respective construct (Carmines & Zeller, 1979); (2) Composite reliability values for the three constructs were all greater than .7 for ‘modest’ reliability in early stages of research by Nunnally (1978); and (3) The average variance extracted for the three constructs were .5 or above, which reached the minimum requirement of .5 suggested by Fornell and Larcker (1981). Discriminant validity was supported as chi-square difference tests showed statistically significant differences between the unconstrained model (the correlation between two constructs is free) and the constrained model (the correlation between the constructs is set to 1.0) for each pair of constructs one at a time (Bagozzi, Youjae, & Lynn, 1991). Taken

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85

together, this scale is considered to be reliable and valid. For more details about the validation of the scale, please refer to the study of Lau and Yuen (2014). 3.2. Participants The participants in this study were Secondary 2 (Grade 8) Chinese students in the 2011/2012 academic year in Hong Kong. All participating students in the survey obtained their parents' consent. These students were selected from 36 secondary schools using stratified random sampling with strata based on a broad categorization of general academic ability of students that reflects the profiles of all students in the territory. This ensures that students of various academic abilities are well represented in the sample. One intact class of the Secondary 2 level was invited to participate in the study from each of the sampled schools. In Hong Kong, since it is generally true that class sizes are very similar within the same school, we only selected one class out of all the classes at the level randomly from each sampled school. The initial dataset consisted of 1121 cases. The procedure to handle data elimination for inconsistencies was as follows. First, we checked for the inconsistencies of their use of ICT at home using the two sets of questions about ‘students use desktop computers’ and ‘students use notebook computers’. These two sets of questions were designed in hierarchical structure. Consistency check was conducted using the answers of the questions. Second, we looked into questions ‘in the last two weeks, how much time did you spend on using computer/the internet on average at home daily’ and ‘of which, how much time did you spend on learning’. Clearly, the second answer should not be greater than the first answer. The resulting sample included 826 students after data verification. Their ages ranged from 11 to 16 (mean = 13.16, sd = .773). 93.2% of the students were between the age of 12 and 14. There were 396 males and 429 females in the sample. One student did not indicate his or her gender in the questionnaire. 3.3. Procedure The procedure of the study was explained to each class of students in a session. Students were asked to complete an online survey instrument. It is a self-reported questionnaire about ICT use and related issues including students' perceived ICT literacy. Students were required to finish the survey during class in 20 min. 3.4. Data analytic strategy This study adopted a commonly used multistage procedure in the framework of multiple-group confirmatory factor analysis (MGCFA) (Byrne, 2004) for testing factorial invariance including configural invariance, measurement invariance, and structural invariance. Configural invariance means that the pattern of free and fixed model parameters is identical across groups (Dimitrov, 2010; Wu et al., 2007). In testing configural invariance, it is important to first identify a baseline model, which is estimated for each group separately. Typically, the most parsimonious model, which represents the most meaningful and best fitting model to the data for a group, is referred to as the baseline model for the group. Measurement invariance is considered at three levels: (a) metric invariance, i.e., equal factor loadings across groups, (b) scalar invariance, i.e., equal item intercepts across groups, and (c) invariance of item uniquenesses, i.e., equal item error variances/covariances across groups. These three levels are also referred to as weak, strong, and strict measurement invariance respectively. Finally, structural invariance is achieved when factor variances and covariances are invariant across groups. The forward approach (or sequential constraint imposition) to testing factorial invariance across groups is used, which involves a chisquare difference test (Δχ2) between two nested models: a constrained model (invariance assumed) and unconstrained model (no invariance

81

assumed) for specific parameters such as factor loadings and intercepts, is performed. Invariance of the parameters being tested is achieved if the chi-square difference (Δχ2 = Δχ2constrained − Δχ2unconstrained) is not statistically significant at a pre-specified level of significance (e.g., .05). The analysis starts with the unconstrained model (total lack of invariance) followed by subsequent imposition of restrictions for equality of specific parameters across groups. In the event of partial invariance, i.e., there are some specific parameters that are not invariant; Byrne (2010) suggested that one can test the invariance of all factor loadings at the subscale level separately. If the invariance of a factor is established, its specified equality constraints are retained and the invariance of other factors are tested cumulatively until all the invariant parameters are identified. Other researchers have argued that fit indexes could also be used to determine factorial invariance across groups. For instance, Cheung and Rensvold (2002) concluded that a decrease of .01 or more in CFI would signify a lack of invariance, i.e., ΔCFI b −.01.

3.5. Data screening and descriptive statistics MGCFA was performed on the data using AMOS 20.0.0. We used the maximum likelihood estimation method with the option to estimate means and intercepts in order to handle missing data. Before we tested factorial invariance, the assumptions of normality and absence of outliers were examined. Skewness and kurtosis of the variables ranged from −1.787 to .436 and −.951 to 2.300 respectively (see Table 1). According to Kline's (2005) suggestion that only variables with absolute values of skewness greater than 3 and absolute values of kurtosis greater than 10 are of concern, the data can be regarded as univariate normally distributed. The Mardia's coefficient, as a measure of multivariate normality, was 84.214, which was less than the recommended value (p(p + 2) = 17(19) = 323, where p is total number of observed indicators) by Raykov and Marcoulides (2008), and thus the assumption of multivariate normality was tenable. Univariate outliers were examined using the criterion of whether there are very large standardized scores (absolute values of z scores greater than 3.3). Fifty univariate outliers were found and removed. Mahalanobis distances of all the cases were evaluated and compared with the critical value of χ2(17) = 40.79 at the alpha level of .001 for the identification of multivariate outliers. Based on this criterion, 38 cases were removed from the data set. One case was also removed because the field about gender was missing and the final sample size was 737. In the end there were 344 males and 393 females in the sample. Since all the items are measured on a 5-point Likert scale (1: strongly disagree to 5: strongly agree), all students tend to perceive their ICT literacy to be fairly good. Table 1 Descriptive statistics of the scale items. Item

Male

Female

Mean SD INFL1 INFL2 INFL3 INFL4 INFL5 INFL6 INFL7 INTL1 INTL2 INTL3 INTL4 INTL5 COML1 COML2 COML3 COML4 COML5

3.625 3.820 3.805 3.637 3.564 3.767 3.741 4.375 4.503 4.328 4.314 4.372 4.009 3.922 3.869 3.701 3.640

Skewness Kurtosis Mean SD

.709 .436 .777 .099 .740 .111 .759 .313 .809 .189 .785 −.001 .829 −.071 .841 −.913 .756 −1.125 .854 −.767 .867 −.813 .827 −.875 .911 −.41 .911 −.354 .988 −.407 .993 −.161 1.139 −.364

−.592 −.903 −.763 −.633 −.400 −.473 −.23 −.624 −.328 −.951 −.747 −.691 −.812 −.628 −.623 −.763 −.663

3.501 3.786 3.791 3.560 3.448 3.733 3.626 4.417 4.728 4.527 4.580 4.504 4.168 3.982 3.995 3.746 3.751

.631 .742 .694 .672 .748 .730 .689 .807 .525 .703 .721 .697 .822 .879 .898 .932 1.153

Skewness Kurtosis .391 −.048 −.017 .186 .087 −.053 .127 −1.186 −1.787 −1.284 −1.713 −1.238 −.760 −.620 −.519 −.459 −.622

Note: 396 males and 429 females provided data for the scale items.

−.274 −.479 −.378 −.292 .084 −.363 −.347 .449 2.300 .702 2.296 .834 .259 .045 −.493 −.195 −.502

82

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85

To evaluate model fit between the data and the model, a number of indexes are referenced. The chi-square value should not be significant for a good model fit. However, it tends to increase with sample size, leading to the rejection of model fit. As such, it is suggested that if the ratio of chi-square value to degree of freedom (χ2/df) is less than 3, there is evidence of acceptable model fit (Carmines & McIver, 1981). Hu and Bentler (1999) recommended the following fit indexes be considered in model fit assessment such as the comparative fit index (CFI), Tucker–Lewis Index (TLI), standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA). They showed there is evidence of a reasonably good fit if the following conditions are satisfied: CFI ≥ .95, TLI ≥ .95, SRMR ≤ .08, and RMSEA ≤ .06. The CFA model, which was developed and validated before in our study (Lau & Yuen, 2014), was used as a baseline model and it was fitted to the data from each gender group (see Figs. 1 and 2). The goodness-of-

fit indexes show a good model fit for each group (see Table 2) except the RMSEA value of .07 which was the group of males. The standardized factor loadings of the baseline model for each group are all statistically significant (p b .001) as shown in Table 3. Therefore, configural invariance of the CFA model over the two groups of students was established. As models are nested with respect to the constraints imposed, a chisquare difference test (Δχ2) can be carried out to determine whether factorial invariance exists across groups and it was used to assess the invariance for each model in this study. After configural invariance was established, a baseline model for multiple-group comparison (M0) was formed in which there were no constraints imposed on any parameter of the model across the gender groups (see Table 4). However, as all the factor loadings were set to be equal for the two gender groups, results indicated that the assumption of invariance at this level was not supported (M1) based on the chisquare difference test criterion (Δχ2(14) = 28.503, p b .05). In the event of noninvariance in multiple-group comparison, it is advisable

Fig. 1. A three-factor CFA model of the 17-Item perceived ICT literacy scale for the male group.

Fig. 2. A three-factor CFA model of the 17-Item perceived ICT literacy scale for the female group.

3.6. Factorial invariance across gender

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85

83

Table 2 Model fit indexes for the baseline model of the two gender groups. 90% CI for RMSEA Group

χ2

df

χ2/df

p

CFI

TLI

SRMR

RMSEA

LL

UL

Male Female

320.133⁎⁎⁎ 199.660⁎⁎⁎

116 116

2.760 1.721

.000 .000

.953 .972

.944 .967

.038 .039

.072 .043

.062 .033

.081 .053

Note: CI = confidence interval; LL = lower limit; UL = upper limit. ⁎⁎⁎ p b .001.

to test invariance at the subscale level until all the invariant parameters are identified (Byrne, 2010). Following this procedure, the factor loading between INTL2 and INTL were freed and the model was estimated again. A non-significant result of the metric invariant model was obtained (M1P) (Δχ2(13) = 13.185, p N .05). The standardized factor loading for the male group was .935 with a standard error of .039, while that for the female group was .81 with a standard error of .057. This means that for one unit change in the item score of INTL2, the boys have a smaller change in the factor score of INTL compared with the girls. In testing scalar invariance, full invariance was not obtained (M2) (Δχ2(17) = 77.191, p b .001) and only the item intercepts for INFL2 to INFL7 were invariant after a series of tests, resulting in a partially scalar invariant model (M2P) (Δχ2(6) = 10.376, p N .05). When all the error variances were constrained to be equal, the model did not fit to the data well (M3) (Δχ2(17) = 52.596, p b .001). Again, tests were conducted to find the invariant error terms. It was found that by freeing the error variance of INTL1, invariance of item uniquenesses was achieved (M3P) (Δχ2(16) = 22.837, p N .05). Finally, further equality constraints were imposed on the factor variances and covariances to test structural invariance (M4) (Δχ2(6) = 28.008, p b .001). Subsequent tests showed no evidence of structural invariance after freeing different combinations of factor variances and covariances. As a further remark, it is noted from Table 4 that there are decreases in CFI when models are compared between M1 and M0, M2 and M1P, M3 and M2P, and M4 and M2P. This provides some support to the rejection of model invariance (Cheung & Rensvold, 2002). To conclude, there is configural invariance, partial weak measurement invariance, partial strong measurement invariance, and partial strict measurement invariance of the model across gender. However, structural invariance is not supported by the data. Table 3 Standardized factor loadings and standard errors by gender. Construct/item

INFL INFL1 INFL2 INFL3 INFL4 INFL5 INFL6 INFL7 INTL INTL1 INTL2 INTL3 INTL4 INTL5 COML COML1 COML2 COML3 COML4 COML5

Male

Female

λ

SE

λ

SE

.746 .835 .886 .778 .664 .794 .764

a .077 .073 .076 .082 .078 .083

.693 .744 .833 .712 .571 .764 .625

a .094 .089 .085 .093 .093 .086

.860 .933 .845 .790 .848

a .039 .049 .052 .047

.673 .810 .774 .692 .761

a .057 .076 .076 .075

.905 .850 .809 .694 .562

a .043 .049 .055 .068

.832 .816 .771 .578 .479

a .060 .061 .068 .086

Note: All factor loadings are statistically significant (p b .001); λ = standardized factor loading; SE = standard error. a — this value was fixed at 1.00 for model identification purpose and thus no critical ratio was calculated.

4. Discussion ICT literacy is recognized as an essential skill for 21st century education and learning. To successfully prepare students for lifelong learning, we need to incorporate ICT literacy into the school curriculum (Breivik, 2005; Selwyn, 2011). There is evidence to suggest the existence of gender specific preferences and patterns of use in respect of ICT, and in addition evidence that shows these become more prevalent as children grow older (Tømte, 2011). However, considering the findings of research studies in many countries, it is suggested that gender and attitudes toward ICT use, in particular when it comes to the development of ICT literacy scale, are very complicated issues and require further research and examination (Sarfo, Amartei, Adentwi, & Brefo, 2011). In the ICT in education literature, there is a lack of studies that aim to test factorial invariance of measurement instrument. This study contributes to research on establishing factorial invariance of a perceived ICT literacy scale. Based on the sample collected in this study, we conclude that the scale shows configural and partial measurement invariance but not structural invariance across gender. According to Wu et al. (2007), in practical terms, configural invariance means that different groups adopt the same conceptual framework to respond to measurement items. As configural invariance was established in this study, it can be reasonably assumed that the same constructs are measured among the male and female students. Weak invariance implies that a single unit change in the item score corresponds to an equivalent unit change in the factor score across groups so that the scale of the latent variable is identical across groups. This study shows that with the exception of INTL2, the postulates of weak invariance are valid. Strong invariance necessitates that the centers of the latent variables be scaled identically across groups. We found that partial strong invariance was obtained only after freeing all the intercepts of the items of INTL, COML, and the item intercept of INFL1. This result suggests that comparison of the means of the latent variables INTL and COML across gender should be made with caution. Strict invariance demands the equality of residual variances of all items across groups, and it indicates differences in reliability of the observed scores. The scale demonstrates partial strict invariance with all but the residual variance of INTL1. This implies that the scale is similar in terms of reliability across gender. Finally, as structural invariance does not hold, this suggests that the ranges of scores on the factors vary across groups as do the relationships between the factors (Boeve-de Pauw et al., 2012). Practically speaking, because configural and partial measurement invariance but not structural invariance across gender of the scale was supported, this indicates that male and female students differ in terms of the ways they express ICT literacy, and thus teachers should attempt to understand how this difference is generated and adopt fundamentally different strategies to assess and promote ICT literacy for students of different genders (Inglis et al., 2011). For example, male and female students were found to respond differently to the item INTL2, which is “I am able to search for information on the internet using a search engine (e.g., Yahoo, Google, Baidu)”. Is it because the males have more experience with or hold more positive attitudes toward technology than the females? Or is it because the males are able to employ more effective information seeking strategies such as searching, scanning, and browsing than the females? (Roy, Taylor, & Chi, 2003). From a broader

84

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85

Table 4 Testing factorial (measurement and structural) invariance across the two gender groups. Model

χ2

df

χ2/df

M0 M1 M1P M2 M2P M3 M3P M4

519.820 548.323 533.005 610.196 543.381 595.977 566.218 571.389

232 246 245 262 251 268 267 257

2.241 2.229 2.176 2.329 2.165 2.224 2.121 2.223

Δχ2

Model comparison

28.503⁎ 13.185 77.191⁎⁎⁎ 10.376 52.596⁎⁎⁎ 22.837 28.008⁎⁎⁎

M1 vs. M0 M1P vs. M0 M2 vs. M1P M2P vs. M1P M3 vs. M2P M3P vs. M2P M4 vs. M2P

Δdf

CFI

ΔCFI

RMSEA

14 13 17 6 17 16 6

.961 .959 .961 .952 .960 .955 .959 .957

−.002 .000 −.009 .008 −.005 .004 −.002

.041 .041 .040 .043 .040 .041 .039 .041

Note: M0 = baseline model (no invariance imposed); M1 = invariant factor loadings; M1P = partially invariant factor loadings (freeing the factor loading between INTL2 and INTL); M2 = partially invariant factor loadings and invariant intercepts; M2P = partially invariant factor loadings and partially invariant intercepts (freeing all the intercepts of the items of INTL, COML, and the item intercept of INFL1); M3 = partially invariant factor loadings, partially invariant intercepts, and invariant residual variances; M3P = partially invariant factor loadings, partially invariant intercepts, and partially invariant residual variances (freeing the residual variance of INTL1); M4 = partially invariant factor loadings, partially invariant intercepts, and invariant factor variances and covariances. a. ΔCFI ≤ −0.01 indicates lack of invariance of the respective comparison of nested models. ⁎ p b .05. ⁎⁎⁎ p b .001.

perspective, the mastery of ICT literacy is a critical component for students to successfully learn in the 21st century, and it can help to alleviate the digital divide, which is related to educational inequality and social exclusion. In particular, it is hoped that the prevailing gender digital divide is addressed partially through our informed understanding of gender difference in ICT literacy within the context of factorial invariance. There remain some issues that have the potential to be further examined in future studies. First, the absence of invariant intercepts in the items of INTL and COML points to the presence of item bias or differential item functioning (DIF) (Dimitrov, 2010). Apart from investigating the reasons for the occurrence of DIF across gender, it is also worthwhile to understand the types of DIF, whether it is uniform or non-uniform. In uniform DIF, the bias occurs uniformly at all levels of the latent trait whereas in non-uniform DIF, the bias may only occur at high or low level of the trait (Aguilar-Díaz, Page, Thomson, & Borges-Yáñez, 2013). Boeve-de Pauw et al. (2012) provided three options to deal with noninvariant items including making group comparisons based on all the items without consideration of measurement invariance, avoiding the use of any scales with noninvariant items, and selecting the invariant items while dropping the noninvariant items. Second, given that measurement invariance is largely ignored in the pertinent literature (e.g., Hatlevik & Christophersen, 2013; Tsai & Tsai, 2010; Voogt, 1987; Zhao et al., 2010; Zhong, 2011), it is plausible that observed gender differences in ICT literacy in previous studies are the results of methodological artifacts, i.e., they can be explained by measurement noninvariance of the items, it is imperative to examine the measurement invariance properties of the instruments employed in these studies. Finally, more research is needed to test measurement invariance of the scale across other groups defined by demographics such as age, socio-economic status, and culture. 5. Conclusion The measurement of ICT literacy has been a major research area over the years. Yet it is common to find tests that are biased toward certain groups of individuals because they lack invariance of the elements of the measurement structure. Measurement invariance is a prerequisite for cross-group comparisons and its establishment is needed to ensure homogeneity of parameter values of a model within sub-groups in a population. This study used the framework of MGCFA to test the factorial invariance of a perceived ICT literacy scale. The scale shows configural and partial measurement invariance but not structural invariance across gender. For the purpose of mean comparisons across groups, the scale should undergo further revisions to the items that are noninvariant so that they are free of bias. At the same time, measurement invariance should be considered as an additional criterion

together with reliability and validity when developing and validating a new scale. Appendix A The three-factor, 17-item perceived ICT literacy scale Factor

ICT literacy

INFL INFL1 INFL2 INFL3 INFL4

Information literacy I am able to identify appropriately the needed information from question. I am able to collect/retrieve information in digital environments. I am able to use ICT to process appropriately the obtained information. I am able to interpret and represent information, such as using ICT to synthesize, summarize, compare and contrast information from multiple sources. I am able to use ICT to design or create new information from information already acquired. I am able to use ICT to convey correct information to appropriate targets. I am able to judge the degree to which information is practical or satisfies the needs of the task, including determining authority, bias and timeliness of materials. Internet literacy I am able to set a homepage for an internet browser. I am able to search for information on the internet using a search engine (e.g., Yahoo, Google, Baidu). I am able to use email to communicate. I am able to use instant messaging software (e.g., MSN, QQ) to chat with friends. I am able to download files from the internet. Computer literacy I am able to set header/footer in word processor software (e.g., Microsoft Word). I am able to plot a graph and chart using spreadsheet software (e.g., Microsoft Excel). I am able to insert an animation in presentation software (e.g., Microsoft PowerPoint). I am able to edit a photo using image processing software (e.g., Photo Editor, Photo Impact, Photo Shop). I am able to set up a printer (e.g., installing printer driver).

INFL5 INFL6 INFL7

INTL INTL1 INTL2 INTL3 INTL4 INTL5 COML COML1 COML2 COML3 COML4 COML5

Note: All the items are scored on a 5-point Likert scale (1: strongly disagree to 5: strongly agree).

References Aguilar-Díaz, F. D. C., Page, L. A. F., Thomson, W. M., & Borges-Yáñez, S. A. (2013). Differential item functioning of the Spanish version of the Child Perceptions Questionnaire. Journal of Investigative and Clinical Dentistry, 4(1), 34–38. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Bagozzi, R. P., Youjae, Y., & Lynn, W. P. (1991). Assessing construct validity in organizational research. Administrative Science Quarterly, 36(3), 421–458. Bawden, D. (2001). Information and digital literacies: A review of concepts. Journal of Documentation, 57(2), 218–259.

W.W.F. Lau, A.H.K. Yuen / Learning and Individual Differences 41 (2015) 79–85 Boeve-de Pauw, J., Jacobs, K., & Van Petegem, P. (2012). Gender differences in environmental values: An issue of measurement? Environment and Behavior. http://dx.doi. org/10.1177/0013916512460761. Bransford, J. D., Brown, A. L., Cocking, R., Donovan, M., & Pellegrino, J. W. (2000). How people learn: Brain, mind, experience, and school. Washington DC: National Academy Press. Breivik, P. S. (2005). 21st century learning and information literacy. Change, 37(2), 20–27. Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling, 11, 272–300. Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications and programming. New York: Routledge. Carmines, E. G., & McIver, J. P. (1981). Analyzing models with unobserved variables: Analysis of covariance structures. In G. W. Bohrnstedt, & E. F. Borgatta (Eds.), Social measurement: Current issues (pp. 65–115). Beverly Hills, CA: Sage Publications. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Sage University paper series on quantitative applications in social sciences, No.07–17, Beverly Hills: CA: Sage. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255. Cooper, J. (2006). The digital divide: The special case of gender. Journal of Computer Assisted Learning, 22(5), 320–334. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1–9. Dimitrov, D. M. (2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149. Ezziane, Z. (2007). Information technology literacy: Implications on teaching and learning. Educational Technology & Society, 10(3), 175–191. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50. Hatlevik, O. E., & Christophersen, K. -A. (2013). Digital competence at the beginning of upper secondary school: Identifying factors explaining digital inclusion. Computers & Education, 63, 240–247. Hohlfeld, T. N., Ritzhaupt, A. D., Barron, A. E., & Kemker, K. (2008). Examining the digital divide in K-12 public schools: Four-year trends for supporting ICT literacy in Florida. Computers & Education, 51(4), 1648–1663. Hsu, M. H., & Chiu, C. M. (2004). Internet self-efficacy and electronic service acceptance. Decision Support Systems, 38(3), 369–381. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. ICT Literacy Panel (2007). Digital transformation: A framework for ICT literacy. Retrieved November 26, 2014, from http://www.ets.org/Media/Tests/Information_and_ Communication_Technology_Literacy/ictreport.pdf. Inglis, C. J., Marzo, J. C., Castejon, J. L., Nuñez, J. C., Valle, A., Garcia-Fernandez, J. M., et al. (2011). Factorial invariance and latent mean differences of scores on the achievement goal tendencies questionnaire across gender and age in a sample of Spanish students. Learning and Individual Differences, 21(1), 138–143. Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). Thousand Oaks, CA: SAGE. Kelly-Salinas, G. (2000). Different educational inequalities: ICT an option to close the gaps. Schooling for tomorrow: Learning to bridge the digital divide (pp. 21–36). OCED. Kim, H. -S., Kil, H. -J., & Shin, A. (2014). An analysis of variables affecting the ICT literacy level of Korean elementary school students. Computers & Education, 77, 29–38.

85

Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York: The Guilford Press. Lau, W. W. F., & Yuen, A. H. K. (2014). Developing and validating of a perceived ICT literacy scale for junior secondary school students: Pedagogical and educational contributions. Computers & Education, 78, 1–9. Lim, K., & Meier, E. B. (2011). Different but similar: Computer use patterns between young Korean males and females. Educational Technology Research and Development, 59(4), 575–592. Luu, K., & Freeman, J. G. (2011). An analysis of the relationship between information and communication technology (ICT) and scientific literacy in Canada and Australia. Computers & Education, 56(4), 1072–1082. NCWIT (2014). NCWIT's women in IT: By the numbers. Retrieved November 13, 2014, from http://www.ncwit.org/sites/default/files/resources/btn_02282014web.pdf. Nimon, K., & Reio, T. G., Jr. (2011). Measurement invariance: A foundational principle for quantitative theory building. Human Resource Development Review, 10(2), 198–214. Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill. Raykov, T., & Marcoulides, G. A. (2008). An introduction to applied multivariate analysis. New York: Taylor and Francis. Roy, M., Taylor, R., & Chi, M. T. H. (2003). Searching for information on-line and off-line: Gender differences among middle school students. Journal of Educational Computing Research, 29(2), 229–252. Sarfo, F. K., Amartei, A. M., Adentwi, K. I., & Brefo, C. (2011). Technology and gender equity: Rural and urban students' attitudes towards information and communication technology. Journal of Media and Communication Studies, 3(6), 221–230. Schmitt, T. A. (2011). Current methodological considerations in exploratory and confirmatory factor analysis. Journal of Psychoeducational Assessment, 29(4), 304–321. Selwyn, N. (2011). Schools and schooling in the digital age. London: Routledge. Tømte, C. (2011). Challenging our views on ICT, gender, and education. Nordic Journal of Digital Literacy, 6(Special Issue), 309–325. Tsai, M. J., & Tsai, C. C. (2010). Junior high school students' Internet usage and selfefficacy: A re-examination of the gender gap. Computers & Education, 54(4), 1182–1192. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70. Voogt, J. (1987). Computer literacy in secondary education: The performance and engagement of girls. Computers & Education, 11(4), 305–312. WGBH Educational Foundation and the Association for Computing Machinery (2009). New for computing image: Report on market research. Retrieved March 7, 2013, from www.acm.org/membership/NIC.pdf. Widaman, K. F. (2007). Common factors versus components: Principals and principles, errors and misconceptions. In R. Cudeck, & R. C. MacCallum (Eds.), Factor analysis at 100: Historical developments and future directions (pp. 177–203). Mahwah, NJ: Lawrence Erlbaum Associates. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multigroup confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research and Evaluation, 12(3), 1–26. Zelman, M., Shmis, T., Avdeeva, S., Vasiliev, K., & Froumin, I. (2011). International comparison of information literacy in digital environments. Retrieved January 2, 2015, from http://www.iaea.info/documents/paper_30e43f54.pdf. Zhao, L., Lu, Y., Huang, W., & Wang, Q. (2010). Internet inequality: The relationship between high school students' Internet use in different locations and their Internet self-efficacy. Computers & Education, 55(4), 1405–1423. Zhong, Z. J. (2011). From access to usage: The divide of self-reported digital skills among adolescents. Computers & Education, 56(3), 736–746.