Intranet satisfaction questionnaire: Development and validationof a questionnaire to measure user satisfaction with the Intranet

Computers in Human Behavior 25 (2009) 1241–1250

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Intranet satisfaction questionnaire: Development and validation of a questionnaire to measure user satisfaction with the Intranet Javier A. Bargas-Avila *, Jonas Lötscher, Sébastien Orsini, Klaus Opwis University of Basel, Faculty of Psychology, Department of Cognitive Psychology and Methodology, Missionsstrasse 62a, 4055 Basel, Switzerland

a r t i c l e

i n f o

Article history: Available online 8 August 2009 Keywords: Intranet Enterprise portal Questionnaire Survey Measure User satisfaction Usability

a b s t r a c t In recent years, Intranets have become increasingly important to their companies. Substantial investments have been made to provide crucial information and workflows to employees. In this context the question of quality assurance arises: how can user satisfaction with the Intranet be measured? This article presents the development of a questionnaire to measure user satisfaction with the Intranet. After a first validation of the instrument (18 items) in an international insurance company (N1 ¼ 881Þ, a final set of 13 items remained. These were tested with the Intranet of a national retail company (N 2 ¼ 1350Þ. The final version showed a high internal consistency (Cronbach aÞ of .89, good item difficulties (.36–.73) and discriminatory power coefficients (.48–.73), as well as a moderate average homogeneity of .44. An exploratory factor analysis revealed two factors, ‘‘Content Quality” and ‘‘Intranet Usability”, explaining 56.54% of the variance. Meanwhile, the survey was translated into 10 languages: Chinese, English, French, German, Italian, Japanese, Portuguese, Russian, Slovenian, and Spanish. Ó 2009 Elsevier Ltd. All rights reserved.

1. Introduction

2. Theoretical background

An increasing number of companies are developing internal information platforms, referred to as Intranets (Hoffmann, 2001). These portals support thousands of employees in their daily work. In this context, it is crucial to implement methods to measure, maintain and optimize the quality of the Intranet from the users’ point of view. A possible method of measuring user satisfaction is via a questionnaire. Older tools such as that developed by Bailey and Pearson (1983) were made in a very different technological situation and are not suited to Intranets. Newer questionnaires such as those devised by Doll and Torkzadeh (1988) or Lewis (1995) proved to be reliable, but were developed as generic tools. Their questions do not cover important aspects of internal information and working platforms (e.g. quality of communication or search features). The main goal of this work is to develop a questionnaire to measure user satisfaction with the Intranet. For the scale construction an explorative approach was chosen. The item analysis is based on classic test construction theories. This article begins with a short description of the theoretical background (Section 2). Then the first and second validation of the survey are presented (Sections 3 and 4). In the last Sections (5 and 6) further aspects and the results of the Intranet Satisfaction Questionnaire (ISQ) validation are discussed.

2.1. Intranet and user satisfaction

* Corresponding author. Tel.: +41 61 2673522; fax: +41 61 2670632. E-mail address: [email protected] (J.A. Bargas-Avila). 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.05.014

An ‘‘Intranet” is defined as a network of linked computers to which only a restriced group of an organizations members have access (Hoffmann, 2001). These links are usually based on common Internet technologies such as TCP/IP and HTTP protocols, naming conventions, and mark-up languages. There are three main factors that make these technologies very attractive to companies: (1) worldwide access through the global address system URI; (2) easy integration of different text, graphic, audio and video formats; and (3) it is easy to link these resources to each other with hyperlinks (Kaiser, 2000). These factors open up almost infinite possibilities for companies to provide new communication, information and knowledge pools that are targeted only to their employees. Intranets usually include common features such as news, forms, business processes or employee search, and sometimes also more advanced applications such as transaction processing systems, management information systems, decision support and expert systems. The construct ‘‘user satisfaction” in connection with computers is most often described as affective attitude. Bailey and Pearson (1983, P. 531) define user satisfaction as the ‘‘sum of one’s positive and negative reactions to a set of factors”. Doll and Torkzadeh (1988, P. 261) describe it as ‘‘the affective attitude toward a specific computer application by someone who interacts with the application directly”. Current authors also describe it as an affective

1242

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

reaction to a given information system (Fisher & Howell, 2004). For this work, user satisfaction is regarded as a ‘‘psychological tendency that is expressed by evaluating a particular entity with some degree of favor or disfavor” (Eagly & Chaiken, 1998, P.296). According to Huang, Yang, Jin, and Chiu (2004), user satisfaction is the most often used construct to measure the success of an information system. Nevertheless, there is little research that shows under which conditions user satisfaction arises. There is also little consensus as to how user satisfaction should be operationalized and many studies do not provide a theoretical basis for these operationalizations (Melone, 1990). The Expectancy Value Theory of Fishbein (1967) explains that an attitude can be understood as the sum of the products of a person’s expectations and evaluations (see Fig. 1). Adapted to this work, A0 stands for user satisfaction with the Intranet, bi are the expectations that the Intranet provides certain attributes, and ei are the evaluations of these attributes. In order to measure user satisfaction, it is therefore theoretically necessary to know all expectations (b’s). According to Gizycki (2002), web users have certain expectations regarding efficiency and effectiveness. If these expectations are fulfilled it is to be assumed that users will be satisfied with the system. In this concept, user satisfaction is regarded as the output of a comparison process of expectations and the perceived performance level of the application (Töpfer & Mann, 1999;Schneider, 2000). It is therefore expected that user satisfaction with the Intranet is reached, if the expectations of its users are fulfilled. The items of this questionnaire will focus on capturing the cognitive components of the user’s attitude toward the Intranet. Variables that may also influence user satisfaction, such as participation in the development of a system (McKeen, Guimaraes, & Wetherbe, 1994), usage frequency (Al-Gahtani & King, 1999), attitudes towards computers (Shaft, Sharfman, & Wu, 2004), and computer anxiety (Harrison & Rainer, 1996), are not covered.

his data revealed three factors influencing user satisfaction: system usefulness, information quality and interface quality. Other authors developed satisfaction scales for specific areas such as online-shopping (McKinney, Yoon, & Zahedi, 2002), company websites (Muylle, Moenaert, & Despontin, 2004), business-to-employee systems (Huang et al., 2004), mobile commerce interfaces (Wang & Liao, 2007), knowledge management systems (Ong & Lai, 2007), and the information systems of small companies (Palvia, 1996). Holzinger, Searle, Kleinberger, Seffah, and Javahery (2008) developed metrics specially targeted at the elderly population. A brief comparison of the items and scales of the named instruments reveals two main areas for consideration: (1) questions dealing with the information offered by the system and (2) items referring to the design of the human–computer interface. Therefore the development of this instrument will be based on these two topics (see Sections 2.3 and 2.4), referred to as ‘‘content quality” (1) and ‘‘Intranet usability” (2). 2.3. Quality of information An approach to determine data quality was made by Wang and Strong (1996). In their work, they found four levels of data quality: (1) Intrinsic data quality: this covers attributes that are embedded within the data such as accuracy, objectiveness or credibility. (2) Contextual data quality: this is given when the data are available in such a way that they fit into the workflow. According to Wang and Strong (1996), these are attributes such as relevance, added value, timeliness, and adequateness. (3) Representational data quality: this describes how easily the data can be understood and interpreted. (4) Accessibility data quality: this describes how easily access (search possibilities and information channels) to the information is granted.

2.2. User satisfaction questionnaires In this section a brief overview of existing questionnaires that were developed to measure user satisfaction in different contexts is provided. Bailey and Pearson (1983) developed a tool with 39 items to measure user satisfaction with computers. The questionnaire dates from over 20 years ago, when computers had very limited possibilities and were mostly used in data processing. Therefore several items deal solely with the satisfaction of the data-processing personnel. Technological advancements and the development of interactive software created a need for usable interfaces. Doll and Torkzadeh (1988) developed a questionnaire with 12 items designed to measure ease of use with specific applications. They postulated that user satisfaction is composed of five factors (content, accuracy, format, ease of use and timeliness). Harrison and Rainer (1996) confirmed the validity and reliability of this tool and showed that it could be used as a generic measuring tool for computer applications. Lewis (1995) developed and validated another questionnaire with 19 items, referred to as the ‘‘Computer Usability Satisfaction Questionnaires” (CSUQ). He regards usability as the prime factor influencing user satisfaction (hence the name). The analysis of

Fig. 1. Expectancy Value Theory (Fishbein, 1967).

According to Wang and Strong (1996, P. 22), high data quality can be described as follows: ‘‘High-quality data should be intrinsically good, contextually appropriate for the task, clearly represented, and accessible to the data consumer”. For the construction of the ISQ, it is assumed that users expect high data quality of information provided. This factor will subsequently be called ‘‘Content Quality”. 2.4. Quality of human–computer interface To conduct successful interactions with the Intranet, users have to form a correct mental representation of the dialogue system (Spinas, 1987). This means that they form a mental map of the content (information and functions) and the organization (structural criteria and access ways). Users should always be able to answer the questions ‘‘Where am I? What can I do here? How did I arrive here? Where can I go to? And how can I get there?” (Baitsch, Katz, Spinas, & Ulich, 1989). To ensure this, the dialogue has to be built in accordance with the ways that users process information (Spinas, 1987;Rauterberg, 1994;Herczeg, 1994;Rosson & Carroll, 2002;Holzinger, 2005). This topic is most commonly described as ‘‘usability”. There have been numerous attempts to determine generic rules and standards to ensure high usability in computer applications. The International Organization for Standardization (ISO) describes seven principles for designing usable dialogue systems: (1) suitability for the task, (2) self-descriptiveness, (3) controllability, (4) conformity with user expectations, (5) error tolerance, (6) suitability for individualization and (7) suitability for learning (ISO, 1998). For the construction of the ISQ it is assumed that the employees of a company expect a usable and efficient Intranet that enables

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

them to fulfill their tasks. If these expectancies are not met, users will often react with negative feelings. There are many authors who have developed items to measure the quality of the human– computer interface (e.g. Lewis, 1995). We will refer to this as ‘‘Intranet usability”. 2.5. Generic vs. specific items: the need for a new questionnaire The focus of most satisfaction scales lies in the development of application-independent instruments to enable their use in various contexts. Considering the broad range of tools and applications, this is often a very difficult task. However, it can be assumed that users have context-specific expectancies that arise depending on the information system used (here: the Intranet). In 2003, the consulting company Stimmt AG used the generic scale Computer System Usability Questionnaire (Lewis, 1995) to measure user satisfaction in 16 Intranets (Oberholzer, Leuthold, Bargas-Avila, Karrer, & Glaeser, 2003). They found that the instrument was only partially suited to use for Intranets. Many employees complained about the superficial questions, and several items were not applicable at all in this context. Intranet managers noted that the surveys left several important questions open (e.g. regarding the quality of the search engine, the navigation or the up-todateness of information). These observations led to the conclusion that there is a clear need for a new survey. The ISQ must include specific questions regarding topics that are relevant in the context of an Intranet.

3. Development and first validation This section describes how the first version of the ISQ was developed and validated. 3.1. Development of the ISQ 3.1.1. Scale Likert scales were used for all items. In this well established method, respondents specify their level of agreement or disagreement with a positive statement. Here, a six-point scale was used with labeled extreme points (1 = I strongly disagree, 6 = I fully agree). A higher number expresses a higher level of agreement, thus satisfaction. For the rating scale interval measurement is assumed, allowing the corresponding statistical validations. The assumption of interval measurement for a rating scale without prior empirical validation is a videly used research practice (Bortz, 1999). To ensure a high reliability and validity, use of between five and seven categories for a Likert scale is recommended (Borg, 2001). With the six-point scale, participants have three options for a positive and three for a negative attitude. Thus the user is ‘‘forced” to choose a direction (no neutral point is provided). For this instrument, an even number of options were chosen because research shows that a neutral/middle option has several disadvantages (Rost, 2004). According to Mummendey (2003), participants choose the middle option for multiple reasons: (1) they do indeed have a neutral attitude, (2) they do not know how to answer, (3) they think the question is irrelevant, (4) they refuse to answer, or (5) they want to express their dislike of the question (protest answer). The middle option does not therefore always measure the intended neutral attitude. Additionally, it has been shown that motivated participants often avoid using the neutral category (Rost, 2004). The questionnaire contains items that need certain usage experience with the Intranet to be answered. It cannot be ruled out, however, that some participants are unable to answer a question

1243

due to lack of knowledge. For such cases, an alternative option was introduced to the instrument (‘‘I can’t answer this question”). It can be argued here that this leads to similar problems as when an uneven Likert scale with a neutral option is used. Participants can choose this option to express different phenomenons: (1) they do not know how to answer the question or (2) they do indeed have a neutral attitude and therefore are not able to express their opinion. Despite this disadvantage, a six-point Likert scale with an alternative option was chosen, because it seemed to offer the best compromise. 3.1.2. Length The ISQ is intended to be used in a business context. Therefore it is crucial that participants do not have to spend more than 10–15 min answering the survey. This is in line with recommendations made by Batinic (1999). After reference to similar instruments (Lewis, 1995;Doll & Torkzadeh, 1988), a maximum of 20 items was chosen for the first version of the ISQ. 3.1.3. Item-generation for the first version Based on screening of the theoretical approaches and empirical data, a first item pool was generated by the authors and a full-time Intranet manager. These items were screened and unified into a first draft of the ISQ. This draft was revised by three Intranet managers. They were instructed to evaluate whether the questions were suited to measuring the construct ‘‘user satisfaction with the Intranet”, whether they were easy to understand, and to check whether important aspects had been missed. Based on their feedback, the first version of the ISQ, containing 18 items, was finalized. The version in Table 1 was used for the first validation of the ISQ. 3.2. Methodology 3.2.1. Experimental procedure To validate the ISQ, it was implemented as online survey and tested in cooperation with an international insurance company employing about 6000 people. To retain anonymity, the company will be called ‘‘Company A”. All registered data were logged electronically in a database of Company A. The questionnaire was conducted in three languages (German, French, Italian), but due to the small sample size of the French and Italian versions, only the German version will be considered here. The survey started with a short introductory text that highlighted the importance of employees’ feedback, the type of questions, the length of the survey and the anonymity of this enquiry. On the next pages, all 18 questions (see Table 1) were presented one by one. When submitting incomplete questions, users were forced by posterior error messages (Bargas-Avila, Oberholzer, Schmutz, de Vito, & Opwis, 2007) to choose one of the given options. Each item was provided with a free text commentary field. These qualitative data were recorded as an additional information source but could also be left blank. After the ISQ questions had been answered, the survey ended with some demographic items. The demographic part was positioned at the end to avoid the ISQ answers being influenced by concerns that feedback could be backtracked (Rogelberg, 2002). The survey was online in October 2005 for 18 days. After 14 days, the employees received a brief reminder by e-mail encouraging them to participate during the next 4 days. Such reminders have been shown to improve return rates (Batinic & Moser, 2005). 3.2.2. Sample In total, 2359 employees (users of the main Intranet) were asked by e-mail to participate in the survey, leading to 881 valid

1244

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

Table 1 First version of the ISQ (translated by the authors). No.

Item

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

The Intranet delivers relevant content for my work The content in the Intranet is up-to-date The Intranet is arranged in a manner that it is easy to orientate and to find the desired content The way information is communicated in the Intranet is clear and coherent I am familiar with the usage of the Intranet so I can apply it optimally for my needs The Intranet facilitates internal communication (e.g. via an employee-directory, messages from the management or discussion-forums etc.) When needed I can access the Intranet anytime I want I am satisfied with the quality of the search engine. It delivers good and useful results and is suited to find specific content or documents If something in the Intranet is outdated, wrong or incomplete, it is easy to contact the responsible person The Intranet enables me to work more efficiently (e.g. internal workflows, accessing support or information retrieval) If I want to publish a message or a document in the Intranet, I know how to proceed With the Intranet I can work fast (e.g. fast page and document download) The Intranet is easy to use (e.g. personalization, handling the employee-directory) I am satisfied with the help and support I get when I encounter a problem or have a question regarding the Intranet (e.g. online-help or help-desk) The Intranet constantly provides up-to-date company news I encounter the work-relevant information on the Intranet in a format I can easily handle I can rely on the information in the Intranet Overall I am satisfied with the Intranet

responses (27 responses had to be excluded for a variety of reasons; see Section 3.3). Due to the internal policies of Company A, only a reduced set of demographic variables could be requested. Of participants, 25.1% were aged below 30 years old, 31.3% were aged between 31 and 40 years, 27.9% were aged 41–50 years and 15.7% were aged 51 or more years. A total of 26.8% of the participants had worked for Company A for 1–5 years, 23.2% from 6 to 10 years and 41% for more than 10 years. Only 8.3% had been employed for less than 1 year, showing that the majority of participants had been able to use the Intranet for quite a while. 3.3. Results In total, 908 responses were registered. Four participants were excluded because they had not answered at least 11 items. One response was discarded, because the ‘‘I can’t answer it” option had been chosen in more than half of the items. Four people were excluded because they answered all 13 items exclusively with the best or the worst item score. Finally, 18 participants were excluded because the boxplot showed their answers to be clear statistical outliers. The sample size for the item analysis therefore consists of 881 participants. Table 2 gives an overview of the total missing values for each item after exclusion of the 27 participants, Table 3 shows the statistical parameters of the first validation. All missing values are due to the ‘‘I can’t answer it” option. Item 9 with 309 (= 35.1%) and item 14 with 239 (= 27.1%) missing cases differ strongly from the rest. These numbers are not very surprising: Both questions concern topics that may not have been needed by many employees (‘‘contact content owner” and ‘‘get support”). To avoid sample size reduction with the Listwise and Pairwise Deletion, the Expectation-Maximization Algorithm (EM) was used to replace the missing values. The replacement of missing values with EM has been proven to be a valid and reliable method and Table 2 Overview of missing values for each item (first version). Item

1

2

3

4

5

6

7

8

9

N Missing

878 3 0.3

858 23 2.6

881 0 0.0

877 4 0.5

877 4 0.5

857 24 2.7

880 1 0.1

814 67 7.6

572 309 35.1

Count %

Item

10

11

12

13

14

15

16

17

18

N Missing

851 30 3.4

753 128 14.5

842 39 4.4

845 36 4.1

642 239 27.1

871 10 1.1

839 42 4.8

875 6 0.7

880 1 0.1

Count %

Table 3 Statistical parameters of the first validation. Item

N

M

SD

Mode

S

K

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

881 881 881 881 881 881 881 881 881 881 881 881 881 881 881 881 881 881

4.82 4.77 3.37 4.41 4.35 4.35 5.15 3.3 3.93 4.21 3.27 4.25 4.38 4.66 4.89 4.27 4.9 4.49

1.17 0.93 1.27 0.99 1.11 1.16 1.06 1.38 1.06 1.1 1.64 1.13 1.07 0.95 0.96 1.05 1.11 1

5 5 4 5 5 5 6 4 4 4 1 5 5 5 5 5 5 5

1.14 0.84 0.07 0.86 0.72 0.62 1.72 0.14 0.41 0.54 0.11 0.8 0.81 1.01 1.11 0.67 1.21 0.8

0.98 1.04 0.73 0.93 0.18 0.08 3.41 0.9 0.41 0.06 1.09 0.39 0.65 1.54 1.67 0.44 1.36 0.66

pv 0.683 0.657 0.36 0.567 0.558 0.56 0.767 0.356 0.463 0.526 0.38 0.538 0.564 0.629 0.69 0.537 0.7 0.586

Missing values = EM; SES ¼ :082; SEK ¼ :165.

outclasses the listwise and pairwise deletion in many aspects (Schafer & Graham, 2002;Allison, 2001). There are virtually no differences between all values and the EM values (see Table 2). For interval-scaled item responses, it is advisable to calculate the discriminatory power with a product-moment correlation of the item score with the test score (Fisseni, 1997). Table 4 lists the discriminatory power and Cronbach a for each item. The discriminatory coefficients range between.30 (item 11) and.70 (item 10) with a mean of .54 ðSD ¼ :54Þ. Five items show a coefficient below .50 (item 1, 7, 8, 9 and 11). According to Borg and Groenen (1997), the lowest acceptable discriminatory power is .30. Item 11 falls in this category with a value of .30. The rest of the items are in an acceptable to good range. The homogeneity examines whether all items of the ISQ measure the same construct (‘‘user satisfaction”) and whether there are items that overlap (measure similar aspects of the construct). If the items of a test correlate with each other, it can be assumed that they measure similar aspects of the common construct. This topic can be explored in the intercorrelation matrix (see Table 5). It shows significant correlations for all items ðp < 0:01Þ with no negative correlations. The correlations with the global item 18 range from .22 to .62, with items 10, 13 and 16 showing the highest ðr ¼ :60 or higher). Mid-level correlations are found for items 3, 4, 6, 12, 14 and 15

1245

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

items of the ISQ form the basis for the second version of the ISQ (see Section 4).

Table 4 Discriminatory power and Cronbach a for each item (first version).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Corrected item – total correlation

Alpha if item deleted

.4977 .5651 .5659 .6396 .5399 .6367 .4403 .3683 .4553 .6931 .2994 .5724 .6240 .5960 .5837 .6700 .5082

.8818 .8799 .8792 .8772 .8802 .8765 .8837 .8882 .8832 .8747 .8945 .8790 .8774 .8789 .8792 .8758 .8814

3.4.1. Scale There is a clear tendency to use the ISQ scale in the upper part of the six-point scale (see Fig. 2). This finding is not surprising: It can be expected that an international insurance company will probably develop an acceptable Intranet. Additionally, this finding is in line with survey results gathered with the Computer System Usability Questionnaire (Lewis, 1995) by the consulting group Stimmt AG (Oberholzer et al., 2003;Leuthold et al., 2004), where the mean satisfaction level of 16 companies was 4.9 on a 7-point Likert scale. For the ISQ this means that the instrument differentiates well for the upper scale range.

aitem117 ¼ :8869. N = 881; Missing values = EM.

ð:50 < r < :60Þ, and finally items 11 ðr ¼ :22Þ; 7ðr ¼ :37Þ and 9 ðr ¼ :39Þ show very low correlations with the global item. The intercorrelations of items 1 to 17 are relatively moderate. The average homogeneity index for the scale is at .35 and the homogeneity indices for each item range from .19 to .49, with the lowest values for items 7, 8, 9 and 11. One explanation for the relatively moderate indices could lie in the complexity of the construct ‘‘user satisfaction”, a circumstance that requires the items to be heterogeneous to cover the whole spectrum. Cronbach a for the first version of the ISQ is relatively high ða ¼ :8869Þ, indicating a good reliability for this instrument. Item 18 represents the overall satisfaction expressed by users and was not taken into account in the calculation to avoid artificial inflation of Cronbach a. Table 4 shows that the internal consistency would be higher if items 11 and 8 were to be deleted (for item 8 only a small effect is stated). 3.4. Discussion The validation of the ISQ shows promising results. At the same time, it becomes clear that there are several problematic items that need to be modified or deleted. The discussion of the scale and the

3.4.2. Items In this section, only the problematic items will be discussed. An item can be regarded as being problematic if the statistical parameters show insufficient values (see Section 3.3) and/or if analysis of users’ comments (see Section 3.2.1) shows that participants misunderstood the corresponding item. Item 1. The frequency distribution of item 1 shows a minor ceiling effect. The homogeneity index and the discriminatory power are relatively low. An exclusion might have been considered, but due to the importance of the item it was decided to leave it in the pool for a second testing. Item 3. Although the statistical paramenters for these questions are satisfactory, the analysis of registered commentaries showed that many participants made comments about the search engine. Probably the last part of the item ‘‘. . .to find the desired content” triggered associations with the search engine in many users. This mix-up was not intended. To eliminate this problem the item was reformulated as the following statement: ‘‘The Intranet has a concise layout and a comprehensible structure”. Item 4. Compaired to the new formulation of item 3, the statement of item 4 is not sufficiently different. Whereas the first is intended to measure the comprehensibility of the structure, the latter deals with the understandability of the content. Therefore item 4 has been reformulated as ‘‘When I read something on the Intranet the content is clear and understandable”. Item 5. The statistical parameters of item 5 are in a mid-level range and provide arguments for both maintaining and eliminating the question. The item was intended to measure how familiar

Table 5 Intercorrelation matrix and homogeneity indices for item 1– 18 (first version). Item

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 H

1

2 .430 .241 .320 .436 .425 .310 .122 .125 .502 .103 .231 .267 .339 .389 .403 .378 .420 .320

3

4

5

6

7

8

9

10

11

12

1 .434 .250 .110 .195 .445 .309 .298 .382 .310 .347 .345 .308 .468 .338

1 .288 .264 .288 .543 .243 .392 .458 .386 .440 .455 .334 .528 .396

1 .160 .207 .331 .056 .358 .293 .340 .378 .286 .338 .371 .286

1 .307 .312 .127 .218 .257 .232 .164 .340 .161 .436 .241

1 .279 .253 .297 .303 .369 .285 .302 .308 .386 .286

1 .236 .446 .474 .425 .440 .568 .413 .614 .431

1 .241 .252 .203 .127 .252 .085 .221 .188

1 .487 .440 .345 .465 .317 .518 .362

13

14

15

16

17

18

1 .487 .332

1 .491

1 .293 .469 .336 .372 .396 .235 .302 .412 .085 .314 .303 .389 .491 .363 .461 .498 .362

1 .482 .368 .427 .165 .414 .319 .431 .251 .374 .485 .317 .297 .439 .186 .584 .357

1 .406 .456 .338 .247 .332 .464 .159 .407 .492 .411 .454 .509 .353 .565 .404

All correlations are significant ðp < :01Þ. H, homogeneity coefficient. N = 881; Missing values = EM.

1 .456 .413 .474 .285 .600 .393

1 .440 .444 .382 .533 .377

1 .416 .410 .503 .373

1 .433 .619 .418

1246

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

80

SD = .69 Mean = 4.32

Frequency

60

N = 881

40

tion in the Intranet is believable. A brief inspection of the comments registered on this topic showed that many participants were irritated by this question and made remarks like ‘‘Why? Shouldn’t I trust the information in the Intranet?” or ‘‘I really hope that if our company invests so much money in the Intranet, the information is at least to be relied on”. In consideration of the relatively weak statistical parameters and the irritation caused, this item was discarded. The validation of the 18 ISQ items led to the deletion of five (5, 7, 9, 11 and 17) and the modification of three (3, 4 and 6) questions. The second version of the scale therefore contains 13 items and was subjected to another validation (see next section).

20

4. Modification and second validation 0

0 .0 75 5. 50 5. 25 5. 00 5. 75 4. 50 4. 25 4. 00 4. 75 3. 50 3. 25 3. 00 3. 75 2. 50 2. 25 2. 00 2.

Mean scale for items 1 to 18 Fig. 2. Frequency distribution of the individual mean scale for items 1–18 (first version).

respondents are with using the Intranet. The analysis of the commentaries shows that many users made comments about the navigation and structure of the Intranet. This leads to the assumption that the item has strong overlaps with item 3. Therefore item 5 was discarded. Item 7. The discriminatory power of item 7 is sufficient, but relatively low. Additionally, it shows a high difficulty index (also reflected in the observed ceiling effect). The question was intended to investigate whether users can access the Intranet whenever needed. It is probable that new server and Internet technologies have now eliminated downtimes, turning the technological uptime of the Intranet into a daily commodity for most employees. Based on this reasoning, it was decided to eliminate item 7 from the ISQ. Item 8. Item 8 shows a relatively low discriminatory power and homogeneity index. Cronbach a could be slightly augmented by the elimination of this question. On the other hand, this item shows a larger variance, leading to a better differentiation between participants (Kranz, 1979). The search engine item also shows lower means, a finding that is in line with previous studies (Oberholzer et al., 2003;Leuthold et al., 2004). Due to the importance of the search engine, the item will remain in the ISQ, but will be shortened. The new formulation is ‘‘When I search something with the Intranet search-engine I find the desired information within a reasonable amount of time”. Item 9. Item 9 stands out due to the high number of users who chose the answer ‘‘I can’t answer this question” (35.1%). Additionally, the discriminatory power and homogeneity index are relatively low. The question was intended to investigate whether missing, outdated or wrong content could be easily notified to the content owner. It seems that many users do not know whether this can be done, probably because they have never done, or needed to do this. Due to the weak statistical parameters and the high percentage of missing answers this item will be discarded from future versions of the ISQ. Item 11. The statistical parameters of item 11 speak clearly for its elimination from the ISQ. It shows a low homogeneity, discriminatory power and reliability. The question was intended to investigate whether users know how to publish information in the Intranet. This is probably a task that is not applicable for many participants and so the item was deleted. Item 17. This item shows similar statistical parameters to item 1. It was intended to investigate whether users think the informa-

This section describes how the second version of the ISQ was developed and validated. 4.1. Modifications for the second version of the ISQ 4.1.1. Scale and length Considering the positive experiences and results of the first validation, it was decided to leave the scale in the ISQ unchanged from the first version (see Section 3.1.1). The first validation led to the elimination of five items, resulting in 13 questions for inclusion in the second version of the ISQ. This is in line with the requirement of having an instrument that needs only 10– 15 min to be completed. 4.1.2. Items for the second version Based on the validation of the first version (see Section 3.3), the second version of the ISQ contained 13 items (see Table 6). 4.2. Methodology 4.2.1. Experimental procedure To validate the ISQ, it was implemented as online survey and tested in cooperation with a national retail company employing about 45,000 people. To retain its anonymity, the company will be called ‘‘Company B”. All registered data were logged electronically in a database of the Department of Psychology. The questionnaire was conducted in three languages (German, French, Italian), but due to the small sample size of the French and Italian versions, again only the German version will be considered here. The survey started with a short introductory text that highlighted the importance of employees’ feedback, the type of questions, the length of the survey and the anonymity of this enquiry. On the next pages all 13 questions (see Table 6) were presented one by one. This time users were not forced to choose one of the given options. Each item was again provided with a free text commentary field. Due to the internal policies of Company B, it was not permissible to pose demograhic questions. The survey was online for 7 days. No reminder was sent because the return rate was very good. 4.2.2. Sample In total, 7000 employees were asked by news posting to participate in the survey, leading to 1350 valid feedbacks (25 feedbacks had to be excluded for various reasons; see Section 4.3). 4.3. Results A total of 1375 responses were registered. The missing data consist of unanswered items and ‘‘I can’t answer it” statements. Twenty

1247

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250 Table 6 Second version ot the ISQ (translated by the authors). No.

Item

1 2 3 4 5 6 7 8 9 10 11 12 13

The Intranet delivers relevant content for my work The content in the Intranet is up-to-date The Intranet has a concise layout and a comprehensible structure When I read something on the Intranet the content is clear and understandable The Intranet facilitates internal communication (e.g. via an employee-directory, messages from the management or discussion-forums etc.) When I search something with the Intranet search-engine I find the desired information within a reasonable amount of time The Intranet enables me to work more efficiently (e.g. internal workflows, accessing support or information retrieval) With the Intranet I can work fast (e.g. fast page and document download) The Intranet is easy to use (e.g. personalization, handling the employee-directory) I am satisfied with the help and support I get when I encounter a problem or have a question regarding the Intranet (e.g. online-help or help-desk) The Intranet constantly provides up-to-date company news I encounter the work-relevant information on the Intranet in a format I can easily handle Overall I am satisfied with the Intranet

250

participants were excluded, because they had not answered at least seven items. Another response was discarded, because the ‘‘I can’t answer it” option had been marked in more than half of the items. In the case of the last four excluded participants, it was assumed that they did not fill out the questionnaire seriously, because they answered all 13 items exclusively with the best or the worst item score. The sample size for the item analysis therefore consists of 1350 participants. Table 7 gives an overview of the total missing values for each item after the exclusion of the 25 participants. Most missing values are due to the ‘‘I can’t answer it” option. Item 10, with 375 missing cases (= 27.8%), differs strongly from the other items, which all have less than 7% missing values. Only eight of these 375 cases are blank; the rest of the participants marked the option ‘‘I can’t answer it”. These figures suggest that most of the participants have so far not required any help concerning the Intranet and therefore chose not to answer. Again, the Expectation-Maximization Algorithm (EM) was used to replace missing values, leading to virtually no differences between all values and the EM values. The average individual mean scale for the second ISQ is 4.30 ðSD ¼ :78Þ. The lowest mean is 1.15 and the highest 5.95. Fig. 3 shows the mean scales for items 1–13. The distribution of the mean scale is again skewed to the right (Skewness = .69, se ¼ :07Þ and shows a sharp peak (Kurtosis = .63, se ¼ :13Þ. Kolmogorov-Smirnov and Shapiro-Wilk tests confirm that the shape of the acquired data is not normally distributed ðp < :01Þ. The means for items 1–13 reside within a range of 1.59 given by the lowest score of item 3 ðM ¼ 3:46Þ and the highest score of item 11 ðM ¼ 5:05Þ with an average standard deviation of 1.13. Table 8 gives a brief overview of the most important statistical parameters. The frequency distributions show for most items a slightly asymmetric distribution to the right. All items show a negative skewness, although some are almost zero. Most items show a sharper peak than a normal distribution, with item 11 ðK ¼ 2:67Þ having the sharpest one. Exceptions are items 3, 6, 8 and 9 (slightly negative Kurtosis values). The item difficulty indices ðpv Þ, shown in Table 8, are again measured with consideration of the variances. They are distributed between .36 (item 6) and .73 (item 11). The average index is at .56 ðSD ¼ :11Þ.

SD = .78 Mean = 4.30

Frequency

200

N = 1350

150

100

50

0

Mean scale for items 1 to 13 Fig. 3. Frequency distribution of the individual mean scale for items 1–13 (second version).

Table 9 lists the discriminatory power and Cronbach a for each item of the second version. The coefficients range from .48 (item 6) to .73 (item 7) with a mean of .60 ðSD ¼ :54Þ. This time, no questions show worryingly low or high values. Regarding homogeneity, the intercorrelation matrix (see Table 10) shows significant correlations for all items ðp < 0:01Þ with no negative correlations. The correlations with the global item 13 range from a moderate to high level between .46 and .69. The average homogeneity index for the second version is now at .44 and the homogeneity indices for each item range from .34 to .57. Again, the relatively moderate indices are explained with the complexity of the construct ‘‘user satisfaction” that requires the items to be heterogeneous to cover the whole spectrum. Cronbach a for the second version of the ISQ is again relatively high ða ¼ :89Þ confirming the good reliability of this survey. Item 13 represents the overall satisfaction expressed by users and was

Table 7 Overview of missing values for each item (second version). Item

1

2

3

4

5

6

7

8

9

10

11

12

13

N Missing

1333 17 1.3

1330 20 1.5

1338 12 0.9

1348 2 0.1

1265 85 6.3

1306 44 3.3

1288 62 4.6

1307 43 3.2

1292 58 4.3

975 375 27.8

1330 20 1.5

1289 61 4.5

1340 10 0.7

Count %

1248

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

Table 8 Statistical parameters of the second validation. Item

N

M

SD

Mode

S

K

pv

1 2 3 4 5 6 7 8 9 10 11 12 13

1350 1350 1350 1350 1350 1350 1350 1350 1350 1350 1350 1350 1350

4.55 4.85 3.46 4.74 4.30 3.35 4.15 4.05 4.19 4.50 5.05 4.38 4.34

1.13 0.97 1.38 1.02 1.17 1.37 1.10 1.22 1.20 1.04 0.95 1.09 1.09

5 5 4 5 5 4 4 5 5 5 5 5 5

0.75 1.13 0.06 1.07 0.65 0.06 0.54 0.54 0.56 0.82 1.39 0.79 0.85

0.26 1.76 0.84 1.44 0.09 0.86 0.07 0.28 0.17 0.94 2.67 0.55 0.52

0.61 0.68 0.39 0.65 0.55 0.36 0.51 0.50 0.53 0.59 0.73 0.56 0.56

Missing values = EM; SES ¼ :067; SEK ¼ :133.

Table 9 Discriminatory power and Cronbach a for each item (second version).

1 2 3 4 5 6 7 8 9 10 11 12

Corrected item – total correlation

Alpha if item deleted

0.5149 0.5198 0.5821 0.5722 0.6649 0.4848 0.7389 0.5831 0.7058 0.6302 0.5882 0.6640

0.8864 0.8860 0.8839 0.8835 0.8783 0.8900 0.8746 0.8830 0.8759 0.8806 0.8830 0.8787

aitem1—12 ¼ :8908; N = 1350; Missing values = EM.

not taken into account in the calculation to avoid artificial inflation of Cronbach a. Table 9 shows that no item-deletion brings a relevant improvement to the internal consistency. In order to determine what factors were important for overall satisfaction (item 13), a linear regression was conducted, with the global item (13) as the dependent variable and the other 12 items as predictors. The model fit was R2 ¼ :741 (adjusted), with F(12, 1337) = 322, and a significance level of p < :005. In the linear regression model, the standardized beta coefficients reveal the importance of each item for the overall satisfaction level (see Table 11). For all items except item 4, beta coefficients are significant ðp < 0:05Þ. The most important item is the one concerning the comprehensible structure (.301), followed by ease of use (.156)

and reusable data format (.153). On the other hand, the item concerning content understandability (.007) is less important to predict the overall satisfaction value. Altogether the regression analysis shows satisfactory values for the ISQ. 4.4. Discussion The second validation of the ISQ reveals a stable measuring instrument. All statistical parameters (item difficulties, discriminatory power, homogeneity and internal consistency) are within reasonable to good levels. However, there are some items that must be critically discussed here. Item 10. With 27.8% missing values, this item is clearly the weakest point of the ISQ. Is it adequate to retain an item where more than a fourth of participants chose not to provide an answer? The statement of item 10 refers to the support and help quality employees receive, when they encounter problems in the Intranet (‘‘I am satisfied with the help and support I get when I encounter a problem”). Interactive applications should be designed in such a way that users are helped when they get to a dead end. Linderman and Fried (2004) refer to this topic as ‘‘Contingency Design”. It seems reasonable to claim that the level of help and support will influence satisfaction in the Intranet. At the same time, there will always be users who never need help and therefore will not be able to provide an answer to this question. Given the fact that all other statistical parameters are satisfactory and that the high level of missing values can be explained, it is justifiable to leave this item in the ISQ. Item 13. The global item was used to measure the internal validity of the construct. It can be argued, that after the successful validation of the survey this item now could be discarded. On the other hand, use of the ISQ in different Intranets has shown that it is often useful to have this item to cross-check the overall means of the survey. Therefore it is justifiable to leave this item in the ISQ. The first validation showed the necessity of modifying some items. These modifications were shown to be adequate: Item 3. The comments in the first version showed that many participants were referring to the search engine, rather than to the information architecture. Modification of the statement almost eliminated this problem. In the second validation, only sporadic comments about the search engine were registered in this item. Item 4. This question was modified to be clearly distinguishable from the modified item 3. There were no comments that suggested a possible mix-up. Item 6 (no. 8 in version 1). This item showed relatively low discriminatory power (0.37) and homogeneity (0.24) in the first vali-

Table 10 Intercorrelation matrix and homogeneity indices for item 1–13 (second version). Item

1

2

3

4

5

6

7

8

9

10

11

12

13

1 2 3 4 5 6 7 8 9 10 11 12 13 H

1 0.369 0.234 0.300 0.437 0.203 0.484 0.277 0.334 0.465 0.450 0.443 0.457 0.371

1 0.284 0.394 0.342 0.206 0.384 0.322 0.400 0.399 0.486 0.428 0.480 0.375

1 0.357 0.479 0.474 0.493 0.392 0.594 0.301 0.261 0.433 0.689 0.416

1 0.422 0.233 0.388 0.384 0.468 0.470 0.493 0.460 0.486 0.405

1 0.366 0.624 0.419 0.519 0.449 0.421 0.462 0.603 0.462

1 0.492 0.351 0.413 0.328 0.230 0.323 0.514 0.344

1 0.515 0.554 0.508 0.431 0.539 0.672 0.507

1 0.519 0.410 0.354 0.432 0.533 0.409

1 0.488 0.414 0.485 0.689 0.490

1 0.476 0.478 0.554 0.444

1 0.504 0.494 0.418

1 0.632 0.468

1 0.567

All correlations are significant ðp < :01Þ. H, homogeneity coefficient. N = 1350; Missing values = EM.

1249

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250 Table 11 Beta coefficients for items 1–12 (second version). Item

1

2

3

4

5

6

7

8

9

10

11

12

Stand. beta coefficient

.053

.084

.301

.007

.065

.087

.118

.051

.156

.094

.045

.153

dation. Therefore it was shortened to facilitate understanding. The second version showed improved values for both discriminatory power (0.48) and homogeneity (0.34). Due to the importance of the search topic, this item should remain in the ISQ, even if these values are only in a moderate range. 4.5. Exploratory factor analysis To analyze whether items can be grouped in factors, an exploratory factor analysis was conducted on the first 12 items of the ISQ (the global item was again not included). Two factors emerged with eigenvalues greater than 1.00, explaining 56.54% of the total variance. Accordingly, two factors were extracted and rotated using the Varimax with Kaiser Normalization method. The factor loadings for the extracted factors are shown in Table 12. Based on the item loadings and the theoretical background (see Section 2), the characterization of the two factors is not too difficult. The items that load on the first factor represent the quality of the content (e.g. relevance, up-to-dateness, understandability). Hence this factor is named ‘‘Content Quality”. The items that load on the second factor represent aspects of the Intranet’s usability (comprehensible structure, search engine quality, speed of work, etc.). Hence this factor is named ‘‘Intranet Usability”. In conclusion, the data show evidence that the ISQ is based on a bi-dimensional structure, comprising the factors that emphazise content quality and Intranet usability.

Table 12 Rotated factor loadings of the exploratory factor analysis for the ISQ.

Eigenvalues Relevant content (item 1) Up-to-dateness (item 2) Comprehensible structure (item 3) Content understandable (item 4) Facilitates internal communication (item 5) Quality of the search engine (item 6) Enables efficient work (item 7) Performance/speed (item 8) Ease of use (item 9) Support quality (item 10) Quality of company news (item 11) Data format is reusable (item 12)

Factor 1 Content Quality

Factor 2 Intranet Usability

5.593 .693 .688 .141 .621 .469 .044 .472 .361 .407 .663 .788 .625

1.191 .147 .150 .801 .297 .576 .766 .659 .580 .687 .330 .144 .413

5. Further aspects of the ISQ 5.1. Generalization The ISQ was validated with the Intranets of two different service-industry companies. At this point, it can be argued that some numbers could be biased toward certain aspects of the validation objects. To counter these doubts, the ISQ was used in the Intranets of five additional companies, varying in size and sectors. Table 13 summarizes the most important statistical parameters. It can be seen that the statistical parameters of the ISQ – if applied in different Intranets – show similar values. This indicates that it can be used as a generic survey in different Intranets of different companies and sectors. Again, the highest percentage of missing values (approximately 30-40%) was found for item 10 (see Section 4.4) in all companies. In one case, item 10 showed only 15.7% of missing values. In this company, the ISQ revealed in general very low satisfaction values. It was reported that there were severe problems with the usability and stability of this portal, which probably led to an increased need for help and support. 5.2. Language effects All ISQ validations were made with the German version of the survey. The translations of the survey into Chinese, English, French, Italian, Japanese, Portuguese, Russian and Spanish were made by professional language services. The Slovenian version was translated by the language service of a Slovenian company. At the moment, there are insufficient statistical data to make direct comparisons of different languages within the same company. A qualitative analysis of the registered free text commentary fields showed no feedbacks that could point to problematic translation effects. In the future it would be important to conduct multilingual surveys within large companies to make direct statistical comparisons and to examine possible language biases. 6. Conclusions Both validations of the ISQ show high Cronbach a values, evidence of excellent internal consistency. Furthermore, the second validation indicates that Cronbach a cannot be substantially increased by item-exclusion. The homogeneity indices have been increased to a moderate level in the second validation. Given that the construct of user satisfaction is complex and implies heterogeneous items, these values can be regarded as satisfactory. Thus, the overall reliability and validity of the ISQ are good. The items were corrected and approved by several Intranet managers, mak-

Extraction method: principal component analysis. Rotation method: Varimax with Kaiser normalization.

Table 13 Statistical parameters of ISQ surveys in other companies. Employees

Sector

N

% mis (min)

% mis (max)

pv (min)

pv (max)

RIT (min)

RIT (max)

H (min)

H (max)

a1—12

2500 11,000 6000 19,000 8000 2000

Technology Government Technology Chemical Finance Technology

95 663 108 105 57 292

0 0 0 0 0 0

31.6 38.2 15.7 35.2 38.6 29.5

.436 .584 .345 .495 .407 .487

.727 .804 .637 .683 .760 .751

.4675 .5528 .4732 .4598 .4871 .4416

.7978 .6955 .7041 .7968 .8073 .7172

.346 .405 .327 .320 .370 .321

.617 .583 .535 .579 .596 .583

.9041 .9014 .8873 .8878 .9114 .8969

mis, missing values; pv , item difficulty; RIT , discriminatory power; H, homogeneity; a, internal consistency.

1250

J.A. Bargas-Avila et al. / Computers in Human Behavior 25 (2009) 1241–1250

ing it very likely that the most important aspects of user satisfaction in the Intranet were considered and leading to a good content validity. The criterion-related validity is measured by the correlations of the global item as an external criterion; this also shows satisfactory results. The objectivity concerning the independency of the experimenters can be disregarded in an online study. The validation of the ISQ seems to be company-independant, because comparison with other companies shows similar results. In this work, a clear and identifiable need of the business sector was adressed: how can we measure user satisfaction with the Intranet? With an explorative approach 18 items were generated and tested. The validation showed that they could be reduced to 13 items. The resulting measuring instrument was again validated, turned out to be stable, and can be recommended for usage in the Intranet. In November 2006, the ISQ was offered via www.Intranetsatisfaction.com in various languages for free on the Internet. Since then, over 500 companies from around the world have downloaded the tool and dozens have already made use of it. This clearly confirms the need for such an instrument. An Intranet is not a static construct. It is in constant modification and grows organically. New technologies open more and more possibilities of developing useful services for employees. On the other hand, the market changes and brings new challenges to companies that lead to new requirements for the Intranet. Due to these conditions, we consider that the current version of the ISQ will have a lifespan of 3–5 years. It will be necessary to monitor current and future trends, and at a given time to develop and validate a subsequent version. References Al-Gahtani, S. S., & King, M. (1999). Attitudes, satisfaction and usage: Factors contributing to each in the acceptance of information technology. Behaviour & Information Technology, 18(4), 227–297. Allison, P. D. (2001). Missing data. Vol. 136 of Sage University Papers: Quantitative applications in the social sciences. Sage Publications. Bailey, J., & Pearson, S. (1983). Development of a tool for measuring and analyzing computer user satisfaction. Managment Science, 29(5), 530–545. Baitsch, C., Katz, C., Spinas, P., & Ulich, E. (1989). Computerunterstützte Büroarbeit. Ein Leitfaden für Organisation und Gestaltung. Zürich: vdf. Bargas-Avila, J., Oberholzer, G., Schmutz, P., de Vito, M., & Opwis, K. (2007). Usable error message presentation in the world wide web: Do not show errors right away. Interacting with Computers, 19(3), 330–341. Batinic, B. (1999). Online research: Methoden, Anwendungen und Ergebnisse. Internet und Psychologie. Göttingen: Hogrefe, Verl. für Psychologie. Batinic, B., & Moser, K. (2005). Determinanten der Rücklaufquote in Online-Panels. Zeitschrift für Medienpsychologie, 17(2), 64–75. Borg, I. (2001). Mitarbeiterbefragungen. In H. Schuler (Ed.), Lehrbuch der Personalpsychologie. Goettingen: Hogrefe, Verlag für Psychologie. Borg, I., & Groenen, P. (1997). Modern multidimensional scaling: Theory and applications. Springer. Bortz, J. (1999). Statistik für Sozialwissenschaftler (5th ed.). Berlin: SpringerLehrbuch, Springer. Doll, W. J., & Torkzadeh, G. (1988). The measurement of end-user computing satisfaction. MIS Quaterly, 12(2), 259–274. Eagly, E. A., & Chaiken, S. (1998). Attitude structure and function. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (4th ed., pp. 269–322). New York: Mc Graw-Hill. Fishbein, M. (1967). Readings in attitude theory and measurement. New York: John Wiley. Fisher, S. L., & Howell, A. W. (2004). Beyond user acceptance: An examination of employee reactions to information technology systems. Human Resource Management, 43(2 & 3), 243–258. Fisseni, H.-J. (1997). Lehrbuch der psychologischen Diagnostik: Mit Hinweisen zur Intervention (2nd ed.). Göttingen: Hogrefe, Verlag für Psychologie.

Gizycki, G. v. (2002). Usability – Nutzerfreundliches Web-Design. In M. Beier & V. Gizycki (Eds.), Usability. Nutzerfreundliches Web-Design (pp. 1–17). Berlin: Springer. Harrison, A. W., & Rainer, K. R. (1996). A general measure of user computing satisfaction. Computers in Human Behavior, 12(1), 29–92. Herczeg, M. (1994). Software-Ergonomie: Grundlagen der Mensch-ComputerKommunikation. Bonn: Addison-Wesley Publishing. Hoffmann, C. (2001). Das Intranet: ein Medium der Mitarbeiterkommunikation. Vol. 9 of Medien und Märkte. UVK, Konstanz. Holzinger, A. (2005). Usability engineering methods for software developers. Communications of the ACM, 48(1), 71–74. Holzinger, A., Searle, G., Kleinberger, T., Seffah, A., & Javahery, H. (2008). Investigating usability metrics for the design and development of applications for the elderly. In Proceedings of the 11th international conference on computers helping people with special Needs (pp. 98–105). Springer. Huang, J.-H., Yang, C., Jin, B.-H., & Chiu, H. (2004). Measuring satisfaction with business-to-employee systems. Computers in Human Behavior, 20, 17–35. ISO (1998). Ergonomics requirements for office work with visual display terminals (VDTs). International Standardization Organisation (ISO). Kaiser, T. M. (2000). Methode zur Konzeption von Intranets. Difo-Druck OHG, Bamberg. Kranz, H. T. (1979). Einführung in die klassische Testtheorie. Vol. 8 of Methoden in der Psychologie. Frankfurt a.M: Fachbuchhandlung für Psychologie. Leuthold, S., Hah, E.-J., Karrer, L., Glaeser, M., Guth, I., Bargas-Avila, J., Oberholzer, G., & Springer, A. (2004). Stimmt Intranet Report 2004. Tech. rep., Stimmt AG. Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57–78. Linderman, M., & Fried, J. (2004). Defensive design for the Web. New Riders, Indianapolis, Ind. McKeen, J. D., Guimaraes, T., & Wetherbe, J. C. (1994). The relation between user partricipation and user satisfaction: An investigation of four contingency factors. MIS Quaterly, 18(4), 427–451. McKinney, V., Yoon, K., & Zahedi, F. M. (2002). The measurement of web-customer satisfaction: An expectation and disconfirmation approach. Information Systems Research, 13(3), 296–315. Melone, N. P. (1990). A theoretical assessment of the user-satisfaction construct in information system research. Managment Science, 36(1), 76–91. Mummendey, H. D. (2003). Die Fragebogen-Methode: Grundlagen und Anwendung in Persönlichkeits-, Einstellungs- und Selbstkonzeptforschung (4th ed.). Göttingen: Hogrefe, Verlag für Psychologie. Muylle, S., Moenaert, R., & Despontin, M. (2004). The conceptualization and empirical validation of web site user satisfaction. Information & Management, 41(5), 543–560. Oberholzer, G., Leuthold, S., Bargas-Avila, J., Karrer, L., & Glaeser, M. (2003). Stimmt Intranet Report 2003. Tech. rep., Stimmt AG. Ong, C., & Lai, J. (2007). Measuring user satisfaction with knowledge management systems: Scale development, purification, and initial test. Computers in Human Behavior, 23(3), 1329–1346. Palvia, P. C. (1996). A model and instrument for measuring small business user satisfaction with information technology. Information & Management, 31, 151–163. Rauterberg, M. (1994). Benutzerorientierte Software-Entwicklung: Konzepte, Methoden und Vorgehen zur Benutzerbeteiligung. Vol. 3 of Mensch Technik Organisation. Zürich: Verlag der Fachvereine. Rogelberg, S.G. (2002). Handbook of research methods in industrial and organizational psychology. Vol. 1 of Blackwell handbooks of research methods in psychology. Oxford: Blackwell. Rosson, M.B., & Carroll, J.M. (2002). Usability engineering: Scenario-based development of human–computer interaction. The Morgan Kaufmann series in interactive technologies. San Francisco: Morgan Kaufmann. Rost, J. (2004). Lehrbuch Testtheorie-Testkonstruktion (2nd ed.). Bern: Hans Huber. Schafer, J., & Graham, J. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. Schneider, W. (2000). Kundenzufriedenheit: Strategie, Messung, Management. Lerch: Verlag Moderne Industrie. Shaft, T., Sharfman, M., & Wu, W. (2004). Reliability assessment of the attitude towards computers instrument (ATCI). Computers in Human Behavior, 20(5), 661–689. Spinas, P. (1987). Arbeitspsychologische Aspekte der Benutzerfreundlichkeit von Bildschirmsystemen. Zürich: Administration & Druck AG. Töpfer, A., & Mann, A. (1999). Kundenzufriedenheit als Messlatte für den Erfolg. In A. Töpfer & E. Bauer (Eds.), Kundenzufriedenheit messen und steigern. 2nd Edition (pp. 59–110). Luchterhand: Hermann. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–34. Wang, Y., & Liao, Y. (2007). The conceptualization and measurement of mcommerce user satisfaction. Computers in Human Behavior, 23(1), 381–398.