User-centered E-Government in practice: A comprehensive model for measuring user satisfaction

Government Information Quarterly 26 (2009) 487–497

Contents lists available at ScienceDirect

Government Information Quarterly j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / g ov i n f

User-centered E-Government in practice: A comprehensive model for measuring user satisfaction Pieter Verdegem ⁎, Gino Verleye Research Group for Media and ICT (MICT) — Department for Communication Studies, Ghent University (UGent) — Interdisciplinary Institute for Broadband Technology (IBBT), Belgium

a r t i c l e

i n f o

Available online 28 April 2009 Keywords: User-centered E-Government Government internet services Acceptance of information technology and services Measurement of satisfaction Benchmarking User research

a b s t r a c t New information and communication technologies (ICT) offer the government new possibilities for providing citizens and businesses with better, more efficient services. Criticism about the provision of EGovernment services, however, has proposed a more user-oriented approach. The user needs to be placed at the center of the development and the provision of electronic public services. Simultaneously, it is important to investigate the impact of the new services on customers. User satisfaction may indeed have a decisive influence on large-scale adoption and use of E-Government services. This article describes the development of a comprehensive model for measuring user satisfaction in the context of E-Government. It rethinks the e-strategies of government and subsequently presents a conceptual model derived from ICT acceptance theory. Both quantitative as well as qualitative research have been carried out in order to elaborate the model and to formulate adequate indicators for measuring user satisfaction. The measurement model has been tested using data from five Flemish E-Government websites. Structural equation modeling (SEM) was applied in order to investigate the goodness of fit of the model and the underlying indicators. Based on the results, it becomes possible to reduce the list of indicators to nine items, still covering the full conceptual model. With this shortlist of satisfaction determinants, E-Government suppliers would be able to get a clear view on acceptance of their internet services by the users. © 2009 Elsevier Inc. All rights reserved.

1. Introduction The rapid development of new telecommunication and computerized technologies in the last decades has had an enormous impact on how we are living and working. New information and communication technologies (ICT) are changing our society completely. The production, management and consumption of information and knowledge are seen to be the core of societal development and economic productivity, illustrated in the rise of concepts such as the information society and the so-called knowledge economy (Castells, 2000; Freeman & Soete, 1997). Policy makers cannot stand aloof from these trends as they are forced to implement innovations as well as to explore new opportunities. New possibilities offered by ICT give government chances to rethink ways of working and providing services for citizens and businesses (Bekkers & Homburg, 2007; Heeks, 2003; Prins, 2001). In this changing world, government authorities simultaneously face two challenges: the importance of fulfilling the new needs and expectations of their citizens and the reality of reduced budgets (Bertot & Jaeger, 2008). The new service

⁎ Corresponding author. Korte Meer 7/9/11, B-9000 Ghent, Belgium. Fax: +32 9 264 69 92. E-mail addresses: [email protected] (P. Verdegem), [email protected] (G. Verleye). 0740-624X/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.giq.2009.03.005

delivery must provide greater satisfaction with higher efficiency (West, 2004). E-Government strategies have been increasingly examined and questioned (Jaeger & Thompson, 2003; Heeks & Bailur, 2007; Titah & Barki, 2006; OECD, 2005). Many critics claim that the development of electronic public services has until now been primarily guided by supply side factors and that technological possibilities rather than user needs have determined all too often the design of online public services (Bertot & Jaeger, 2006; Kunstelj, Jukic, & Vintar, 2007; Reddick, 2005b; Schedler & Summermatter, 2007; Gareis, Cullen, & Korte, 2004; Ebbers, Pieterson, & Noordman, 2008). In reaction to this, the plea for (more) user-centered E-Government strategies becomes more prominent. A demand-oriented approach is proposed to improve the supply and take-up of electronic public services. Therefore, an extensive study of the needs and expectations of users (also regarding to services in the future) is essential (van Dijk, Peters, & Ebbers, 2008; Bertot & Jaeger, 2008; Centeno, Van Bavel, & Burgelman, 2004). The goal of the presented study is to meet the call for a more thorough understanding of citizens' needs and expectations towards E-Government. In this article, the user is at the center of the research which has been conducted in order to develop a comprehensive model for measuring user satisfaction of online public services. Enhanced quality of service provision has been a major component of

488

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

public administration reform over the last decades and the use of ICT to generate improvements has been a primary driver for E-Government activity (Jaeger, 2003; West, 2004; OECD, 2005; Bekkers & Homburg, 2007). Online public services are increasingly seen as part of a broader service (improvement) strategy, with important customer and efficiency benefits. As users of public services can be obliged to interact with their governments,1 user dissatisfaction with the quality of the services may quickly become a major political issue (OECD, 2003). The discussed model for measuring user satisfaction is based on a research track concerning the needs and preferences of citizens towards electronic service provision. It consisted of a literature review and both quantitative research (large-scale survey, n = 1651) as well as qualitative in-depth research (focus group interviews, n = 28) that have been carried out for the development of the measurement model. Afterwards, the model has been validated by advanced statistical testing (structural equation modeling, SEM) based on a sample of 5590 respondents. The results led to the reduction of the measurement instrument (consisting of satisfaction indicators of online E-Government services) to a short list of nine items, still covering the full conceptual model. The article starts with theoretical considerations concerning the rethinking of E-Government. The successive steps in the development of the measurement tool are presented and illustrated with the most important research results. Finally, the validation of the measurement tool is discussed, together with conclusions and recommendations drawn from the research. 2. Towards user-centered E-Government 2.1. Objectives of electronic public services There are many definitions of E-Government and the term itself is not universally used. The differences are not merely semantic and may reflect priorities in government strategies (Heeks & Bailur, 2007; Yildiz, 2007; Relyea, 2002; Evans & Yen, 2006). Moreover, definitions and terms adopted by individual countries have shifted, as priorities have changed and as progress has been made towards particular objectives. This is as it should be: the area is a dynamic one and policies, as well as definitions, need to remain relevant. The Organisation for Economic Co-Operation and Development (OECD) defines E-Government as: “The use of information and communication technologies, and particularly the internet, as a tool to achieve better government” (OECD, 2003: 23). This definition emphasizes the outcome opportunities of the delivery of electronic public services. New services can indeed offer (more) advanced possibilities and will be more multifunctional. The new technological innovation impulses, however, do not (yet) take into account the impact on the customers (both citizens and businesses) of implementing such public services (IPTS, 2004; van Deursen, van Dijk, & Ebbers, 2006). One of the objectives of E-Government is to make the government and its policies more efficient: providing citizens with quicker and better access to public information and the ability to use services in a more personal and cost-effective manner (Bekkers & Zouridis, 1999; Heeks, 2003; Prins, 2001). The increase in efficiency would strengthen the quality of services and the ambition to achieve more effective outcomes in key policy areas (Relyea, 2002). E-Government would also push the reform agenda (the modernization of the administrations) and simultaneously promote economic policy objectives (OECD, 2003). In addition, public service delivery via ICT would also lead to an improved relationship between government and citizens or 1

In most countries, citizens are not bound to use electronic public services. Some services, however, start to offer certain advantages for people who make use of EGovernment. For instance, the Belgian tax office (Tax-on-Web) provides users with an immediate estimate of their costs or reimbursement.

businesses (Millard, 2003). Finally, E-Government would fortify democracy and reduce the distance between citizens and government (Macintosh, Robson, Smith, & Whyte, 2003). 2.2. Challenges to electronic public service delivery Although E-Government is high on the agenda and many resources are set aside for it, E-Government still faces significant challenges as it continues to develop (Jaeger, 2003; Jaeger & Thompson, 2003; Traunmüller & Wimmer, 2004). In addition, the success of the electronic service delivery is not always clear (yet). Adoption and use of the new services is still rather limited in most countries (van Deursen et al., 2006; van Dijk et al., 2008; Kunstelj et al., 2007; Bertot & Jaeger, 2008; Ebbers et al., 2008) and needs to be stimulated (European Commission, 2006). Low uptake becomes problematic as greater efficiency and returns on investments are only possible with a widespread use of the E-Government services (Norris & Moon, 2005; Jaeger, 2003). In the following paragraphs we seek to understand the criticism on current e-strategies. First, electronic public services have been often primarily guided by supply side factors (Bertot & Jaeger, 2006; Reddick, 2005b; Kunstelj et al., 2007; Gareis et al., 2004). In this approach, governments often start to expand their portfolio of services2 by creating an electronic version of the existing (offline) services (van Dijk et al., 2008; LeeKelley & Kolsaker, 2004). Second, technological possibilities rather than user needs have determined all too often the design of online public services. Too much attention is paid to the technology itself, rather than to the real needs and expectations of users (Bertot & Jaeger, 2008; van Dijk et al., 2008; Ebbers et al., 2008; IPTS, 2004; Pieterson & Ebbers, 2008; Reddick, 2005a). These two tendencies cross the strategy of government to put the user – ‘the individual’ – at the center of the new technological developments towards an inclusive knowledge society (Gareis, 2004). In response to this failing, the focus shifted towards (new) paradigms such as ‘user-centered E-Government’ and ‘multichannel government’ (OECD, 2005). Public services have to evaluate the impact of their strategies on the customers (citizens and businesses) and have to consider the new emerging needs and expectations for electronic services in the future. Furthermore, one of the biggest barriers for the development of E-Government is the digital divide (Reddick, 2005a; Reddick, 2005b; van Dijk et al., 2008; Dugdale, Daly, Papandrea, & Maley, 2005; OECD, 2003). Keeping in mind that a large part of the population does not use the internet (or has no access to it) (Eurostat, 2005), government authorities must consider channels other than the internet for their electronic service delivery (Ebbers et al., 2008), otherwise they risk excluding a major portion of their citizens from benefits offered by E-Government. Improved online access for the public at large will thus increase the pool of potential users of electronic services, while simultaneously delivery of (integrated) public services via other (new) channels must be explored (Pieterson & Ebbers, 2008; Reddick, 2005a; Wimmer, 2002). Moreover, this shift to the development of a multichannel government must be combined with the tendency to make E-Government more user-centric (Bertot & Jaeger, 2006; Lau, 2005). 2.3. Towards user-centric E-Government strategies The tendency to progress to a more user-oriented E-Government approach is influenced by a number of circumstances. In the late 80s, the ‘reinventing government’ movement (Osborne & Gaebler, 1992) 2 Policy makers are driven by the goal to reach the highest possible percentage of online available services. International rankings such as ‘Leadership’ surveys (Accenture), ‘Online availability of public services’ (Cap Gemini Ernst & Young) or ‘UN EGovernment survey’ (UNPAN) put government under pressure to do better than other countries or to evolve from the previous ranking.

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

489

Fig. 1. Basic concept underlying user acceptance models (Venkatesh, Morris, Davis, & Davis, 2003).

already stressed the rethinking of government services and proposed an ‘outward-looking approach’ (starting from the citizens' needs) instead of an ‘inward-looking’ approach (starting from the public services themselves) (Tat-Kei Ho, 2002). In addition, the public sector increasingly considered practices and concepts used in the private sector, such as CRM or CCSD. CRM (Customer Relationship Management) was introduced to reorganize the relationship with the customers in a more efficient way (Whiteley, 2001). CCSD (CitizenCentered Service Delivery) is mainly a Canadian government3 approach to base its e-strategy fundamentally on citizens' needs and expectations (Brown, 2005). It can be stated that a user-centric approach should be an integral part of governmental e-strategies. Continuous inquiry of public service customers is a must, as both services and users (and their expectations) are changing permanently. Especially in the world of ICT where new innovations pass on in a rapid succession (Centeno et al., 2004), more attention to the investigation of needs, perceptions and experiences of users towards technology and its applications is required (van Dijk et al., 2008; Kunstelj et al., 2007; Moore & Flynn, 2004). Interesting examples in this field are the Canadian ‘Citizen First Surveys’ (Roy, 2006), the ‘Use of and Satisfaction with E-Government Services’ studies from the Australian Government (2005) or the ‘EGovernment Trendbarometer’ carried out in Switzerland (Berner Fachhochschule & Unisys, 2005). Furthermore, it is important to bear in mind that there is no such a thing as ‘the’ citizens. Users of electronic public services are very diverse and heterogeneous (Bertot & Jaeger, 2006; Bicking, Janssen, & Wimmer, 2006). Each user group has its own preferences towards services and channels. This comes with the observation that a ‘one size fits all’ approach is no longer feasible (OECD, 2005). 3. The construction of a model for explaining E-Government acceptance and satisfaction Given the increasing focus on efficiency and effectiveness in the public sector (Bertot & Jaeger, 2008; Eyob, 2004; McLaughlin, Osborne, & Ferlie, 2002), it is not surprising that the study of performance measurement is increasing. A majority of these studies focuses on policy and administrative procedures and draws on the new public management paradigm (Bouckaert & Thijs, 2003; Flynn, 2007; Brignall & Modell, 2000). The perspective of this article is, however, slightly different as performance measurement is studied from a user-centered viewpoint. The proposed approach aims to respond to the need for more demand side oriented impact studies. As van Dijk et al. (2008) state, there is a lack of data and investigations into the demand side of public or government electronic services (especially in comparison with multiple classifications and benchmarks lists describing the E-Government supply side). Summarizing, the goal of this study is to respond to this call by developing a comprehensive model for studying user satisfaction, hereby taking into account people's needs and expectations towards online public services. 3 Canada is still considered to be one of worldwide frontrunners in E-Government development (Roy, 2006).

3.1. A basic model from ICT acceptance theory Research into ICT acceptance has been concerned with how and why people adopt new information technologies. Different theoretical frameworks have been arising during the last decades. On the one hand, ‘Diffusion of Innovations’ by Rogers (2003) provided insights into the adoption and diffusion of innovations within the population. It is a frequently used theory to assess adoption potential (Wejnert, 2002). Rogers' view is that technology appropriation passes through different successive stages: knowledge, persuasion, decision, implementation and confirmation. The success of an innovation can be estimated using five innovation characteristics: relative advantage, compatibility, complexity, trialability and observability. The second school in technology research, on the other hand, is comprised of the so-called Social Shaping of Technology theories (Lievrouw, 2006), in which Domestication theory (Silverstone & Haddon, 1996) is one of the most important theoretical exponents. The latter starts from a sociological perspective and stresses the implementation of new technologies in the daily routines of everyday life. Domestication theory has originated partly from criticism that was formulated with regard to Diffusion theory, as this theory is too technologydeterministic and assumes that technology adoption is mainly a linear process (Punie, 2004). Both theoretical schools are general frameworks; they are too broad to apply to most determining variables that influence the acceptance and use of E-Government services (van Dijk et al., 2008). The theoretical frameworks, however, offer the opportunity to derive some general concepts and lines of thought. Diffusion theory, with a focus on innovation related characteristics, has been extended with insights from social psychology studies aiming to broaden the scope to attitudinal (adoption) determinants (Plouffe, Hulland, & Vandenbosch, 2001). This has resulted in a greatly elaborated framework for studying ICT and service acceptance, comprising theories and models such as Theory of Reasoned ActionTRA (Ajzen & Fishbein, 1980), Theory of Planned Behavior-TPB (Ajzen, 1985), Technology Acceptance Model-TAM (Davis, Bagozzi, & Warshaw, 1989) and Social Cognitive Theory (Bandura, 2002). Fig. 1 presents the basic conceptual framework underlying the class of models explaining individual acceptance of information technology and services that forms the starting point for the model presented in this article. The baseline of the figure is that usage (of technology or services) is the key dependent variable, while intention is seen as a predictor of behavior (i.e. usage) (Taylor & Todd, 1995). 3.2. Application to the domain of E-Government services: how to measure E-Government satisfaction? Predicting user acceptance and measuring user satisfaction can be seen as two separate domains. With regard to the goal of this article, however, we wanted to bridge the two frameworks in order to have an integrated conceptual basis at our disposal. Starting from the perspective of social psychology and intention-based models (Fig. 1), in the following paragraphs several models and studies in the field of quality measurement as well as E-Government are

490

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

discussed as they have provided input for the development of the underlying model for measuring E-Government satisfaction. A lot of studies in user satisfaction stress the importance of the relationship between attitudes and perceptions towards (the usage of) electronic services. MORI (2002) elaborates on the role of expectations. They overcome the antithesis between two schools – on the one hand, the “service quality school,” which views satisfaction as an antecedent of service quality; on the other hand, the “satisfaction school” which holds that assessments of service quality lead to an overall attitude towards the service that they call satisfaction – by placing expectations centrally in their conceptual model. MORI states that both concepts are strongly related: if the experience of the service greatly exceeds the expectations clients had of the service, the satisfaction will be high and vice versa. The same line of reasoning is behind SERVQUAL (a multiple-item scale for measuring perceptions of service quality) originated by Parasuraman, Zeithaml, and Berry (1988) and CMT (Common Measurement Tool) which was originally developed by the Canadian Institute for CitizenCentered Service (ICCS, 2003). Zeithaml, Parasuraman, and Malhotra (2000) extended the SERVQUAL model – which consists of four gaps in explaining customer satisfaction: design gap, information gap, communication gap and fulfilment gap – with an awareness gap between objective user needs and effective user requirements.4 CMT also highlights the differences between expectations and experiences, as this difference is perceived to be the key for understanding the fulfilment of public services. In addition, CMT also considers the role of surrounding factors (experience, intensity of use, etc.) as they can be used in order to introduce process improvements. Traditional models in the field of quality and satisfaction measurement, such as SERVQUAL, are originally developed by marketing researchers to assess service quality in general. This model is widely used in different domains, however, its applicability in the field of ICT has come under criticism. Findings indicate that SERVQUAL suffers from a number of conceptual and empirical difficulties (Van Dyke, Kappelman, & Prybutok, 1997; Jiang, Klein, & Carr, 2002). The weak operationalization of the service quality construct5 in particular raises cautions about applying this model for assessing the satisfaction of information systems. Summarizing, some quality measurement schools' lines of thought are incorporated in the proposed conceptual model (see 3.3), however, they were combined with conceptions from ICT acceptance theory as well as E-Government research. In the European IST research project eUser (2004) much attention was given to design and delivery of user-centered online public services (E-Government, e-health, e-learning). At the center of its conceptual framework was the match (or ‘mismatch’) between demand side and supply side. Therefore, eUser takes into account both drivers and barriers for using/not using public services in relation to the satisfaction. For instance, lack of access to ICT in general (or to the online service more particularly) may be an important barrier and can thus cause a negative perception towards the electronic service provision. Especially for public services, issues like accessibility and usability are extremely important (Jaeger & Bowman, 2005; Bertot & Jaeger, 2006). In the same context, van Dijk et al. (2008) have tested a model for explaining the acceptance and use of electronic government services. Starting from the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003), they have adapted a multidisciplinary model comprising elements of both demand side as supply side. Based on the results it can be concluded that the supply of E-Government services is a precondition for people to develop the

4 People might simply not be aware of their needs, which means they do not demand services related to this need from providers. 5 The perception of service quality – according to SERVQUAL and related models – is seen as the result of an evaluation process whereby “the customer compares the perceived service against the expected service” (Grönroos, 1984: 37).

Fig. 2. Extended conceptual model.

intention to use these services. Supply as such, however, does not lead to a direct proportional increase of the actual use. van Dijk et al. (2008) stress the importance of knowledge or availability of EGovernment services. The study also revealed that the interrelated factors of digital media preferences, access, and experience, explain the intentions to use E-Government services better than mental processes and socio-demographic factors. A conclusion from the research is that E-Government acceptance should be seen as a dynamic learning process whereby people will stick to their habits of using traditional (offline) public services unless they learn of a better electronic alternative that is offering real added value. 3.3. Conceptual model Based on the literature review, the basic conceptual model underlying user acceptance models was extended and adapted into a more specific model for studying E-Government acceptance. This is the basis of the framework for measuring user satisfaction of online public services. Fig. 2 depicts this conceptual model. The conceptual model considers the different phases that the user of E-Government services must undergo. The dotted building blocks and arrows are new in comparison with the basic model. It starts from the individual reactions to (using) E-Government services. It is important to take this into account as expectations will influence people's attitudes during the whole process and consequently may have an impact on both usage of and satisfaction with the services (ICCS, 2003; MORI, 2002). After the starting point, the following steps can be distinguished: • Awareness: the user must be aware of the existence of electronic public services (in order to fulfil his or her needs) before the intention to use the E-Government services can arise (van Dijk et al., 2008; Kunstelj et al., 2007). This extra step is important as it is necessary in order to continue and go through the next phase. If people do not know that electronic public services are there, they will not use them even if those services have an added value for

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

them. Raising awareness about E-Government is crucial (Reddick, 2005b) as research has shown that many citizens do not know which services actually do exist (Australian Government, 2005; Berner Fachhochschule & Unisys, 2005) (see also 4.2); • Intentions to use: the user will develop the intention to use the EGovernment services as he or she will consider whether to use the service or not.6 This consideration (of the perceived utility of the service) can be seen as a first evaluation of the service in terms of the fulfilment that the services can offer or not (Titah & Barki, 2006). Usefulness and ease of use can both have a decisive impact on user intentions (Davis, 1993; Venkatesh et al., 2003); • Access: the actual behavior (i.e. usage of the E-Government services) is in the extended conceptual model divided across two separate phases. After awareness of the service and positive evaluation of the utility (resulting in intentions to use), the user will attempt to gain access to the service. As awareness and especially accessibility deserve special attention in a governmental context (Jaeger, 2004), this phase is also an addition in comparison with the original model. It points the attention not only to accessibility of E-Government services as such, but also to the problem of the digital divide (Reddick, 2005a; Reddick, 2005b; Pieterson & Ebbers, 2008; Helbig, Gil-Garcia, & Ferro, 2009); • Usage is the phase in which the user arrives when he or she has developed the intention to use the service and has gained access to that particular service. In sum, the previous stages must be passed before the user can (try to) use the services to fulfil his or her needs. The last step, and of major importance for this article, is the feeling of (dis)satisfaction that the user may potentially have regarding the particular E-Government services. User satisfaction should be measured during or after usage (the last step in the process), but that does not mean that it stops there (Muylle, Moenaert, & Despontin, 2003). The model is dynamic as satisfaction can have an impact on the individual's decision whether or not he or she will reuse the service. (Dis)Satisfaction can also influence individual reactions to (use) E-Government services. A second observation about Fig. 2 concerns the trajectory from access to satisfaction. We state that access can be seen not only as a condition for usage of the service, but also as having a direct impact on the satisfaction of the E-Government services. Government authorities are forced to offer not only usable services, but also accessible services. Other than private companies, they cannot ignore certain groups in the society (van Dijk et al., 2008; Ebbers et al., 2008). So, accessibility, usability and functionality should go in hand with each other (Bertot & Jaeger, 2006). The questions remain: access to what? And what significantly influences user satisfaction in the field of EGovernment? 4. A multi-method research track concerning user-centered E-Government Bringing the concept of “user-centered E-Government” into practice, a multi-method study has been set up with a double goal: first of all, we wanted to inquire about the needs and expectations of Flemish citizens7 towards the electronic service provision. This would allow for refinement or adjustment of the conceptual model which is developed from our literature research. Secondly, we also wanted to

6 Research has shown that there is a manifest gap between intentions to use EGovernment services and actual use of them (van Dijk et al., 2008; Kunstelj et al., 2007). Therefore, E-Government studies should focus on the factors influencing this gap. 7 The research has been carried out in Flanders, the Northern part of Belgium and home to the Dutch speaking community.

491

detect the most important determinants for measuring E-Government satisfaction starting from a user point of view. The first research phase, which consisted of a large-scale quantitative survey (4.1), has been complemented with qualitative research (focus group interviews, 4.2) in order to dispose of in-depth information about the real needs and expectations of citizens towards E-Government services. 4.1. Quantitative research: a broad view on needs and expectations towards E-Government 4.1.1. Methodological base The first empirical phase of the research project consisted of a large-scale quantitative survey. Data were collected in 2006 in Flanders by a combined online panel (73.6%) and offline data collection (26.4%) resulting in a total sample of 1651 respondents. The research sample consisted of 49.3% men and 50.7% women, originating from different regions in Flanders. Respondents were well divided across different age categories, although older respondents (people aged 65 and over) are underrepresented. Due to this underrepresentation, we decided to reweight the sample according to official Flemish population information (SVR, 2008) for the ‘gender versus age’ distribution. The distribution over education levels (3.3% had received only primary education; 39.9% finished secondary school; 33.2% followed higher education and 24.2% obtained a university degree) deviates from the overall Flemish population: this sample is overrepresented by higher educated people (Chi-Square = 558.906; df = 3; p = 0.00). Respondents were asked about their possession and usage of ICT in order to investigate their “techno-profile.” Furthermore, they were questioned about their knowledge of and attitudes towards EGovernment. Flemish citizens most often use local E-Government websites and are in general quite satisfied with the service delivery. Electronic public services from regional government (Flanders) and federal government (Belgium) were less often consulted in comparison with municipality websites. Parts of the questionnaire examined satisfaction determinants in more detail (see further) and also inquired into channel preferences. More particularly, the actual and future use of E-Government was assessed by comparing provision of service via non-electronic channels (counter, mail and telephone) with that via electronic channels (internet, digital television and mobile applications). As the main focus of this article is on the development of a model for measuring user satisfaction of online public services, we will not elaborate on the multichannel part of the research. 4.1.2. Knowledge and attitudes towards E-Government More than once in a lifetime, citizens have to consult different public services. The results of the survey demonstrated that 64.1% of the respondents are familiar with the concept of E-Government. They described it by means of several key words such as “electronic counter,” “reducing the administrative burden,” “outside business hours,” “public service delivery via internet,” “online forms,” “faster,” “tax-on-web,” and “time saving.” In general, E-Government was mainly seen as a public service delivery on the internet. 4.1.3. Most important results on user satisfaction indicators A central goal of the presented research was to enrich the theoretically based model about E-Government acceptance (3.3) with determinants of user satisfaction. The formulation of adequate indicators, however, is quite complicated. Quality is a notion with a lot of different meanings that can have a subjective interpretation. The quality indicators have been determined through iterative cycles of theoretical and pragmatic consensus. In order to get an adequate view of user satisfaction concerning electronic public services, it is important to find out what people judge as important. We confronted the respondents with a list of

492

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

15 factors. The indicators were based on the work of Johnston and MORI. Johnston (1995) listed 18 factors which are often used in empirical research related to measuring user satisfaction of service delivery. MORI (2002) employed the same list of indicators in performance management studies in the UK. The presented determinants are related to functionality, accessibility and usability (Bertot & Jaeger, 2006). The respondents were asked to rate the indicators (as a single item) on an 11-point Likert scale (varying from “not important at all” to “very important”) in terms of their perceived importance. Each of the indicators was accompanied by an explanatory example so that it was clear for to respondents what was meant by the concept. The results show that the importance (mean) scores are close to one another. However, pairwise t-tests of those means after weighting reveal that 85 out of 105 (15 × 14/2) pairwise combinations are statistically significant. The indicator “reduce the administrative burden” received the highest average score of importance (Mean = 8.79; SD = 1.41). Flemish citizens seem to be demanding on this matter (Kampen, Snijkers, & Bouckaert, 2005). A second place is reserved for the determinant “reliability” (Mean = 8.63; SD = 1.61). In addition to these two, we notice a number of criteria with comparable average scores for importance: “usability” (Mean = 8.55; SD = 1.58), “cost effective” (Mean = 8.54; SD = 1.68), “ease of use” (Mean = 8.52; SD = 1.68), “security” (Mean = 8.50; SD = 1.79), “content readability” (Mean = 8.46; SD = 1.77). “Privacy/personal information protection” (Mean = 8.37; SD = 1.86), “courtesy” (Mean = 8.34; SD = 1.71), “content quality” (Mean = 8.30; SD = 1.68), “transparency” (Mean = 8.29; SD = 1.68), “responsiveness” (Mean = 8.23; SD = 1.71) and “accessibility” (Mean = 8.20; SD = 1.62) also have average scores which lie close to one another. Beneath the list, we see “flexibility” (Mean = 7.85; SD = 2.14) and “personal contact” (Mean = 7.61; SD = 2.03). Although these two last parameters got the lowest average scores, the mere fact that the means are in the 8 on a 10-point scale area implies that the respondents still perceive them as important. A preliminary conclusion is that all of the presented determinants are important for citizens and have to be taken into account in the measurement of satisfaction. This research, however, has to be considered as a purely quantitative survey that stressed the importance of all listed determinants. It does not provide much space for (more) gradations. In addition, in-depth research was perceived to be necessary in order to investigate whether other indicators should also be selected. Simultaneously, a prolonged qualitative study enabled better interpretation of satisfaction determinants in terms of their measurability. 4.2. In-depth qualitative research 4.2.1. Methodological base The qualitative research design dealt in an explorative manner with several questions, more specifically about the users' views on government and electronic public services. Three focus group interviews were organized in 2006, with a total of 28 respondents. They were recruited with the help of a field research agency, based on a list of rigid sampling criteria. As a result, heterogeneity in gender, age and education level was obtained. The focus group interviews were led by an experienced moderator, assisted by a researcher. The interviews were recorded by tape and video as support for the transcription. This enhanced the thorough analysis of the data. Data were analyzed using the constant comparison technique (Glaser & Strauss, 1967). For the purpose of this article, all the interview data were handled by one analyst reading all the transcripts to gain an overall sense of the data. The data were then read again and open coded to produce an initial code list (with a focus on the most important indicators for E-Government satisfaction). Afterwards, the data were selectively recoded in terms of categories identified with the initial code list.

The focus group interviews consisted of three main parts. After a short introduction, the attitudes towards public government authorities and particularly the electronic public services were discussed. The second part of the group interviews aimed at reflecting ideas about the most important conditions and experiences towards electronic public service delivery. By doing so, it became possible to evaluate the conceptual model (3.3) in terms of its strength while respondents also indicated which indicators they would have liked incorporated in the measurement model. 4.2.2. Government authorities and electronic public service delivery By and large, citizens' attitudes towards government in the focus group interviews are none too positive. Even worse, the government is seen as a scapegoat for a variety of shortcomings. Moreover, the respondents condemn the cumbersome and bureaucratic structure of the government. It is sometimes seen as an anonymous, bureaucratic superpower. Reducing the administrative burden would be helpful to bridge the divide between policy and citizens. The government also needs a more efficient mode of communication in order to better address the issue that most citizens consider the information given by the government as incomplete, unclear, and unreliable. Above all, the government must become more user-friendly and take into account the different needs of its citizens. In its service delivery, personal contact still remains important. According to the respondents, E-Government has potential in the future, but currently they would rather wait to see which way the wind blows. E-Government is considered unsafe and to offer too little in return. After all, according to the respondents, government authorities have a lot of information at their disposal, and could develop a more pro-active service delivery. Special points of interest to the respondents concerning electronic public service delivery are a better legislative framework and clear information and communication about these services. Although there already exist, a lot of electronic services are still too unfamiliar to the audience. Many of the respondents also stress the importance of a centralized website (one-stop government) (Wimmer, 2002), in which information and services can be ranked according to different life-events. And, in addition to the online environment, an offline contact point (such as a helpdesk by telephone) seems to be of vital importance. 4.2.3. Most important results on user satisfaction indicators During the interviews different determinants were often mentioned by all respondents as having a decisive impact on satisfaction with electronic public services. Based on the analysis of the data, the following aspects are perceived as extremely important with regard to E-Government service delivery: communication about services, recency of information, security, help or guidance, personal contact and centralization/integration. The indicators can be clustered in three groups: 1) access to the service; 2) use of the service; 3) impact of the service. • Access to the service: the respondents stress that services must be easily findable. More information and communication about which services are being offered is therefore essential. Furthermore, these services have to be easily accessible. Also people possessing fewer computer skills must be able to access the services in an uncomplicated way. Above all, these services must be cost friendly. • Usage of the service: the usage of the service largely depends on the content it has to offer. The available information must be easily comprehensible, reliable and up-to-date. In general, the usability of the service is tremendously important. Particularly, the underlying technical system must be stable and reliable so that no malfunctions can arise. The respondents also pay attention to

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

safety issues: government has to be careful with all the data it possesses (identity, financial data, etc.). Again, people emphasize the importance of a kind of helpdesk, or the possibility to have personal contact if necessary. • Impact of the service: very important amongst the goals of any E-Government service is reducing the administrative burden. The latter is, in view of the respondents, strongly connected with the realization of more customer-friendly services. In addition, all respondents point to the necessity of having one central contact point, which is a clear example of integrated EGovernment. After all, people are often confused about which government (federal, regional, local) is responsible for what kind of service. The centralization of services and the accessibility by means of one window, or making these services accessible via a functional guide or search robot, are considered to be very important. The focus group interviews eventually enabled us to complete the list of indicators of satisfaction. It also allowed us to reformulate some of the (descriptions of the) indicators, which is important for practical measurement. Moreover, it became possible to structure and to establish priority in the results of the respondents, i.e. those indicators that have less or more importance for the user satisfaction of e-services. Table 1 offers an overview of key and sub indicators for both access and usage (comprising also impact factors). This table is the result of the coding process of the interview data after several iterations. The determinants were clustered and given an importance rate according to the respondents' opinions. The indicators are listed that have been selected for further use in the research. Key indicators are split up by underlying determinants. Information about their importance (in view of the respondents) is provided (as a result of data analysis): indicators with ‘+++’ are perceived as the most important, followed by ‘++’ and ‘+’. In the most right column, item names are also listed. Summarizing, the qualitative research enabled us to complete or drop some indicators

Table 1 Results of qualitative research: indicators and importance. Key indicator

Indicator

Importance

Item name

Infrastructure

Skills Hardware Software Anytime Anyplace Communication Findability Affordability Speed of the system Technical reliability Compatibility Integration of the different services Reduce of the administrative burden Possibility of personal contact User-oriented Flexibility Customization/personalization Acknowledge the receipt of transaction Protection of personal information Security of the transaction Identification Content recency Content readability Content credibility Content usefulness Content sufficiency Help/guidance Ease of navigation Lay-out and design

+++ + + + + ++ + + +++ +++ + +++ +++ +++ ++ + + +++ ++ + + +++ ++ ++ + + +++ + +

Infra 1 Infra 2 Infra 3 Avai 1 Avai 2 Aware 1 Aware 2 Cost Tech 1 Tech 2 Tech 3 CFr 1 CFr 2 CFr 3 CFr 4 CFr 5 CFr 6 SecPri 1 SecPri 2 SecPri 3 SecPri 4 Cont 1 Cont 2 Cont 3 Cont 4 Cont 5 Usa 1 Usa 2 Usa 3

Availability Awareness Cost Technical aspects

Customer friendliness

Security/privacy

Content

Usability

493

for the measurement of user satisfaction. In addition, the rating of the indicators gave the facility to set up a clear structure. With this information, it became possible to start (re)formulating the indicators into a questionnaire. 5. From conceptual model to measurement tool After constructing a conceptual model and formulating applicable indicators on the basis of quantitative and qualitative research, these indicators finally had to be turned into measurable variables. In other words: how can asking precise questions concerning the different aspects of the service lead to the assessment of a certain user's satisfaction? Based on the indicators collected in the research, the measurement model was constructed by integrating the indicators into a questionnaire. The qualitative research design provided information about how to formulate these indicators precisely, because it had to be ensured that the indicators are understood correctly and exactly the same by all the users. It is indeed important that what is being measured exactly corresponds with what we aim to measure (validity). Therefore, each indicator (single item) was illustrated with an example. To take into account the differences (or possible discrepancies) between expectations towards the service and the perceptions of the service, it was also necessary to measure the importance of a certain aspect or indicator in view of the user (Kunstelj et al., 2007). After all, some people may evaluate the lay-out of a website as very poor, but on the other hand they do not consider this as an important satisfaction criterion. The average score of this concept should then receive a lower weight of importance. Different techniques exist to link satisfaction scores to impact values. In this study it was opted, for research purposes, to include questions concerning both satisfaction and importance in the questionnaire. The construction of the final questionnaire was quite a timeconsuming task, which happened in different iterations. After developing a first version, a pre-test was conducted within a group of 10 respondents who gave feedback for further elaboration or refinement of the measurement tool. Next, the questionnaire was evaluated by a team of E-Government experts (employees of CORVE – Coordination Unit of the Flemish E-Government – partner in the research project). The final step was to bring the measurement instrument into practice. 6. Validation of the research model In order to assess the fit of the conceptual model, structural equation modeling (SEM) was applied. This statistical technique allows for estimation of the goodness of fit of the hypothetical model given the data at hand (Bollen, 1989). Estimating measurement models to validate conceptual models have a long tradition in marketing and consumer research since the first publications by Bagozzi (1980) and Bagozzi (1982). SEM offers a sub model (measurement model) to test assumptions regarding the strength of the relationships between indicators (items in the questionnaire) and the latent variables (the concepts), with simultaneous estimation of the correlations between the concepts. Data collection in the validation study was organized using pop-up messages – in cooperation with a company which specialized in internet research – inviting visitors of five Flemish government websites to participate in an online survey. Since the major goal of this research project was the assessment of the fit of the conceptual model with the data in practice, eventual (unit) non-response was not an issue. Answer categories “non applicable” and blanks were treated as missing information in the FIML AMOS estimation solution (Arbuckle, 2005). This approach yields very reliable estimates (Verleye, 1999). In total 5590 respondents were willing to participate in the study. 75.7%

494

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

Fig. 3. Results of structural equation modeling (SEM).

of the respondents were female8 and the age distribution shows an overrepresentation of young adults compared to a random sample from the Flemish population (SVR, 2008): respondents were aged 16– 24: 22.0%; 25–34: 41.0%; 35–44: 20.9%; 45–54: 13.4%; 55–64: 2.5%; 65–74: 0.2% and finally older than 74 years: 0.1%. Fig. 3 provides information about the measurement model estimated with SEM. In this picture, the concepts that were measured are represented by ellipses while the observed variables (items in the questionnaire) are represented by squares. Arrows from ellipses towards squares represent the relationship between both, estimated as standardized regression coefficient. Covariation between concepts is represented by bidirectional arrows containing the Pearson product-moment correlation. As can be seen in the figure, most assumed indicators do a fine job in measuring the concepts. The concepts are mutually correlated, but the link between concepts and items has on average higher values than the mutual correlations, a necessary characteristic for any sound measurement model.

The pop-up measured two data series following the structure shown in Table 1: the first 29 items have measured the satisfaction rate on a 11-point Likert scale (varying from “not satisfied at all” to “very satisfied”), while the second time the importance was measured on a 5-point Likert scale (varying from “not important at all” to “very important”). Data modeling was carried out on both the satisfaction and importance sub series and revealed twice acceptable goodness of fit (Table 2). For the “cost” measure, only one item was used in the questionnaire. Table 2 contains RMSEA and two incremental fit indices. Incremental fit indices are informative on the fit of the hypothesized model compared to a reasonable base line model. In this case, the latter is a model that suggests that no factors underlie the observed items and that the correlations between the observed items are zero in the Table 2 Goodness of fit indicators for both satisfaction and importance measures.

8

The overrepresentation of female respondents can easily be explained because one of the five E-Government websites deals with family – mother and child – related questions and services.

Satisfaction model Importance model

RMSEA

TLI

CFI

0.062 0.058

0.906 0.902

0.926 0.923

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497 Table 3 Reduced measurement instrument. Item name

Operationalization of indicators

R2

Infra 2

How satisfied are you with software that is needed to get access to this E-Government website? (e.g. internet browser) How satisfied are you with the opportunity to access the services on this E-Government website anytime you want? (e.g. 24h/24, 7d/7) How satisfied are you with the findability of this E-Government website? (e.g. via portal website or search robot) How satisfied are you with the cost of getting access to this E-Government website? (e.g. cost of computer infrastructure, internet connection) How satisfied are you with loading speed of the pages of this E-Government website? How satisfied are you with the possibility of getting personal help/assistance on this E-Government website? (e.g. special telephone number or helpdesk) How satisfied are you with the security of transactions that can be performed via this E-Government website? (e.g. payment via secured access) How satisfied are you with the usefulness of the information that is available through this E-Government website? How satisfied are you with the flexibility that is being offered through this E-Government website? (e.g. both information, communication or transaction possibilities are provided according to specific questions from users)

0.904

Avai 1

Aware 2

Cost

Tech 1 CFr 5

SecPri 3

Cont 4

Usa 2

a

0.688

0.520

/a

0.700 0.580

0.733

0.761

0.773

No R2 value for this item as it has no latent variable.

population. For acceptance of hypothesized models, such incremental fit indices should in general have values higher than 0.9. RMSEA should be smaller than 0.08 for acceptance (Arbuckle, 2005). For application purposes a shortlist with the best items (largest R2 per latent variable) is needed, still covering the full conceptual model; this study yields the following shortlist with nine items — measuring satisfaction (Table 3). In general, it can be concluded that the large-scale validation survey confirmed the conceptual model derived from both quantitative and qualitative research. Furthermore, the results also enable us to derive a core satisfaction questionnaire with substantial predictive power towards overall satisfaction rates. 7. Discussion and recommendations E-Government strategies are evolving in maturity and are increasingly investigated. An important criticism, however, is that electronic public services still lack a user-centric approach (Bertot & Jaeger, 2006; van Dijk et al., 2008; Reddick, 2005b; Kunstelj & Vintar, 2004). The absence of large-scale take-up of E-Government services could – to some extent – be explained by the fact that too much attention is given to technology as well as the governments' tendencies to start all too often from existing ways of working (in terms of the services that are being provided) (van Deursen et al., 2006; Ebbers et al., 2008). In this article, an attempt is made to bring the user-oriented paradigm into practice. More specifically, research is presented concerning the measurement of user satisfaction with (online) E-Government services. The model, based on a literature review and quantitative as well as qualitative research, proved to be a good fit with regard to the measurement of perceptions of actual users towards E-Government. Nevertheless, some considerations and recommendations can be formulated. First, although the validation research mainly intended to investigate whether the measurement instrument corresponds with the theoretical model and indicators concerning E-Government acceptance and satisfaction, some conclusions can be drawn from the test phase. With an overall satisfaction score of 8.00 on 10 (from

495

five E-Government websites that have participated in the research), respondents seem to be quite satisfied with the services that are being tested. Similar results were detected in the quantitative research (see 4.1) concerning the needs and expectations of Flemish citizens as well in other areas of research (Australian Government, 2005; Berner Fachhochschule & Unisys, 2005; Reddick, 2005a). It raises the question of how contradictions can be explained between satisfied E-Government users on the one hand, and citizens having very critical and sometimes very negative attitudes towards government and public services, on the other hand (Millard, 2006). The same can be said about the gap between the interest of potential users and actual use of E-Government (van Dijk et al., 2008; Kunstelj et al., 2007). Second, user-centric research on electronic public services should also pay attention on E-Government acceptance as such. More specifically, a thorough understanding into the motivations of nonusage of E-Government seems to be necessary (van Deursen et al., 2006). The research presented shows that a large part of the respondents explain their E-Government non-usage simply through their unawareness of which services are being offered. Additionally, a lack of awareness of services prevents citizens from getting familiar with the added value that electronic public services could offer in their everyday lives. Summarizing, efforts are needed to improve service delivery, but simultaneously, investigation of which services could really address citizens' needs (including via which channels) seems to be vital (Centeno et al., 2004). Research has shown that channel and media preferences and experience regarding different types of services have a decisive impact on take-up (van Dijk et al., 2008; Ebbers et al., 2008; Pieterson & Ebbers, 2008). Therefore, governments should take into account these channel-related impacts and rethink their electronic service delivery accordingly. This should also be accompanied with more effective information and communication campaigns regarding the existing services (Reddick, 2005b). Third, user research concerning citizens' preferences should be an integral part of e-strategies of government. Inquiring into user needs and expectations only generates added value when it can be organized on a continual basis (Roy, 2006; Bertot & Jaeger, 2008). The latter is important as new developments succeed at an increasing tempo. The plea for repeated investigations with citizens and businesses concerning E-Government should be supported by longitudinal research tracks that provide the opportunity to detect trends and developments for policies in the long-term. Fourth, satisfaction measurements can offer added value when it comes to evaluating existing services (Gupta & Jana, 2003). These measurements (evaluation of beta services) can also be important before launching new E-Government services. Especially with benchmarking techniques, it becomes possible to determine minimal satisfaction scores that should be achieved before actually offering the new service to the overall population. Benchmarking means that all parts of a specific service can be evaluated, but satisfaction can also be compared with other public services in the same domain. Finally, in times which require makers of public policies to rethink their services in terms of efficiency and effectiveness, service delivery via new (electronic) channels cannot stay behind in this performance management debate (Gouscos, Kalikakis, Legal, & Papadopoulou, 2007; Bertot & Jaeger, 2008). Only when E-Government services get the same attention in terms of the quality assessment of their service delivery, will users evaluate online services of equal value as their traditional (offline) equivalents. And this is a prerequisite for stimulating not only the potential interest, but also the actual use of E-Government (Bertot & Jaeger, 2006; van Dijk et al., 2008). These recommendations are not simple to address, but policy makers should consider some of them when evaluating their estrategies. After all, all efforts towards increasing budgets for EGovernment and people management for realizing progressive public policies in the 21st century deserve a larger take-up of E-Government services than occurs today in many countries.

496

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497

8. Conclusion Recently, the development of electronic public service delivery has been increasingly questioned. Therefore, a more user-oriented approach is often suggested. In this article an overview is given of research that has been carried out to develop a framework for measuring user satisfaction of E-Government services. This article discusses the different research steps – reviewing ICT acceptance theory complemented with quantitative as well as qualitative research concerning the needs and expectations of citizens towards electronic public services – that were necessary to formulate adequate satisfaction indicators within an integrated framework. The model takes into account determinants concerning accessibility, usability and functionality. Afterwards, the theoretical measurement model was brought into practice in order to investigate the goodness of fit between the perceptions of citizens towards E-Government services and their actual satisfaction. Advanced statistical testing (structural equation modeling) enabled not only validation of the theoretical model, but also reduction of the list of 29 indicators into a measurement instrument of nine key indicators still covering the full conceptual model. With this concise list of items – comprising determinants of infrastructure, availability, awareness, cost, technical aspects, customer friendliness, security/privacy, content and usability – and, simultaneously, covering the full range of quality assessment of public services, policy makers are provided with a practical tool for bringing the user-centric paradigm into practice. Acknowledgments The research in this paper was carried out within the project DashGov, funded by the Interdisciplinary Institute for Broadband Technology (IBBT). Partners in the consortium are: Flemish government, research groups from Ghent University (UGent), Free University Brussels (VUB), Catholic University Leuven (KUL) and two (research) companies. The authors would also like to thank Laurence Hauttekeete for her support in the research project. References Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice Hall. Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. In J. Kuhl & J. Beckman (Eds.), Action-control: From cognition to behaviour (pp. 11−39). Heidelberg: Springer. Arbuckle, J. L. (2005). AMOS 6.0 user's guide. Chicago, IL: SPSS. Government, Australian (2005). Australians' use of and satisfaction with E-Government services. Barton: Commonwealth of Australia. Bagozzi, R. P. (1980). Causal models in marketing New York: Wiley. Bagozzi, R.P. (eds.) (1982). Special issue on causal modelling. Journal of Marketing, 19(4) (403–584). Bandura, A. (2002). Social cognitive theory in cultural context. Applied Psychology and International Review, 51(2), 269−290. Bekkers, V., & Homburg, V. (2007). The myths of E-Government: Looking beyond the assumptions of a new and better government. The Information Society, 23(5), 373−382. Bekkers, V., & Zouridis, S. (1999). Electronic service delivery in public administration: Some trends and issues. International Review of Administrative Sciences, 65(2), 183−196. Berner Fachhochschule & Unisys. (2005). E-Government trendbarometer Thalwil: Unisys (Schweiz) AG. Bertot, J. C., & Jaeger, P. T. (2006). User-centered E-Government: Challenges and benefits for government web sites. Government Information Quarterly, 23(2), 163−168. Bertot, J. C., & Jaeger, P. T. (2008). The E-Government paradox: Better customer service doesn't necessarily cost less. Government Information Quarterly, 25(2), 149−154. Bicking, M., Janssen, M., & Wimmer, M. (2006). Scenarios for governments in 2020: Towards a roadmap for future E-Government research in Europe. In P. Cunningham & M. Cunningham (Eds.), Exploiting the knowledge economy: Issues, applications and case studies (pp. 407−415). Amsterdam: IOS Press. Bollen, K. A. (1989). Structural equations with latent variables New York: Wiley. Bouckaert, G., & Thijs, N. (2003). Kwaliteit in de overheid. Een handleiding voor kwaliteitsmanagement in de publieke sector obv een internationaal comparatieve studie Brussels: Academia Press.

Brignall, S., & Modell, S. (2000). An institutional perspective on performance measurement in the ‘new public sector’. Management Accounting Research, 11(2), 281−306. Brown, D. (2005). Electronic government and public administration. International Review of Administrative Sciences, 71(2), 241−254. Castells, M. (2000). The rise of the network society: Economy, society and culture. Oxford: Blackwell Publishing. Centeno, C., Van Bavel, R. & Burgelman, J.C. (2004). eGovernment in the EU in the next decade: Vision and key challenges. European Commission, DG JRC, Institute for Prospective Technological Studies. Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982−1003. Davis, F. D. (1993). User acceptance of information technology: System characteristics, user perceptions and behavioural impacts. International Journal of Man-Machine Studies, 38(3), 475−487. Dugdale, A., Daly, A., Papandrea, F., & Maley, M. (2005). Accessing E-Government: Challenges for citizens and organizations. International Review of Administrative Sciences, 71(1), 109−188. Ebbers, W. E., Pieterson, W. J., & Noordman, H. N. (2008). Electronic government: Rethinking channel management strategies. Government Information Quarterly, 25 (2), 181−201. European Commission (2006). i2010 eGovernment action plan: Accelerating eGovernment in Europe for the benefit of all. Communication from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions, COM(2006), 173 Final. Eurostat (2005). The digital divide in Europe. Statistics in focus. Retrieved September 18, 2008 from http://epp.eurostat.cec.eu.int/cache/ITY_OFFPUB/KS-NP-05-038/ EN/KS-NP-05-038-EN.PDF eUser, (2004). Evidence-based support for the design and delivery of user-centered online public services-Workpackage 1: Conceptual and Analytical Framework, part A), All project partners. Retrieved September 11, 2008 from http://www.euser-eu.org/ Evans, D., & Yen, D. C. (2006). E-Government: Evolving relationship of citizens and government, domestic, and international development. Government Information Quarterly, 23(2), 207−235. Eyob, E. (2004). E-Government: Breaking the frontiers of inefficiencies in the public sector. Electronic Government, an International Journal, 1(1), 107−114. Flynn, N. (2007). Public sector management London: Pine Forge Press. Freeman, C., & Soete, L. (1997). The economics of industrial innovation London: Continuum. Gareis, K. (2004). Towards user-centered E-Government. Understanding potential demand for online public services. Paper presented at the Telecities and MUTEIS conference ‘Urban impacts of the information society: Facts, fiction and policies’, March 17–19, 2004, The Hague. Gareis, K., Cullen, K., & Korte, W. (2004). Putting the user at the center. Implications for the provision of online public services. In P. Cunningham (Ed.), eAdoption and the knowledge economy. Issues, applications, case studies (pp. 611−618). Amsterdam: IOS Press. Glaser, B. G., & Strauss, L. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine. Gouscos, D., Kalikakis, M., Legal, M., & Papadopoulou, S. (2007). A general model of performance and quality for one-stop E-Government service offerings. Government Information Quarterly, 24(4), 860−885. Grönroos, C. (1984). A service quality model and its marketing implications. European Journal of Marketing, 18(4), 36−55. Gupta, M. P., & Jana, D. (2003). E-Government evaluation: A framework and case study. Government Information Quarterly, 20(4), 365−387. Heeks, R. (2003). Reinventing government in the information age. International practise in IT-enabled public sector reform London: Routledge. Heeks, R., & Bailur, S. (2007). Analyzing E-Government research: Perspectives, philosophies, theories, methods, and practice. Government Information Quarterly, 24(2), 243−265. Helbig, N., Gil-Garcia, J. R., & Ferro, R. (2009). Understanding the complexity of electronic government: Implications from the digital divide literature. Government Information Quarterly, 26(1), 89−97. ICCS (2003). Common Measurement Tool. CMT Questions bank. Retrieved September 11, 2008 from http://www.iccs-isac.org/eng/cmt-about.htm IPTS, (2004). E-Government in EU in 2010: Key policy and research challenges, in the Experts Workshop on E-Government and e-Health in ACCs. Retrieved September 18, 2008 from http://fiste.jrc.es/download/2.%20IPTS_Future%20%20challenges% 20for%20e-gov1.pdf Jaeger, P. T., & Thompson, K. M. (2003). E-Government around the world: Lessons, challenges, and future directions. Government Information Quarterly, 20(4), 389−394. Jaeger, P. T. (2003). The endless wire: E-Government as global phenomenon. Government Information Quarterly, 20(4), 323−331. Jaeger, P. T. (2004). Beyond section 508: The spectrum of legal requirements for accessible E-Government web sites in the United States. Journal of Government Information, 30(4), 518−533. Jaeger, P. T., & Bowman, C. A. (2005). Understanding disability: Inclusion, access, diversity and civil rights. Westport, CT: Praeger. Jiang, J. J., Klein, G., & Carr, C. L. (2002). Measuring information system service quality: SERVQUAL from the other side. MIS Quarterly, 26(2), 145−166. Johnston, R. (1995). The determinants of service quality: Satisfiers and dissatisfiers. International Journal of Service Industry Management, 6(5), 53−71. Kampen, J. K., Snijkers, K., & Bouckaert, G. (2005). Public priorities concerning the development of E-Government in Flanders. Social Science Computer Review, 23(1), 136−139.

P. Verdegem, G. Verleye / Government Information Quarterly 26 (2009) 487–497 Kunstelj, M., & Vintar, M. (2004). Evaluating the progress of E-Government development: A critical analysis. Information Polity, 9(3–4), 131−148. Kunstelj, M., Jukic, T., & Vintar, M. (2007). Analysing the demand side of E-Government: What can we learn from Slovenian users? Lecture Notes in Computer Science, 4656, 305−317. Lau, E. (2005). E-Government and the drive for growth and equity. Proceedings of the Conference from E-Gov to I-Gov. Retrieved September 13, 2008 from http://www. belfercenter.org/files/lau-wp.pdf Lee-Kelley, L., & Kolsaker, A. (2004). E-Government: The fit between supply assumptions and usage drivers. Electronic Government, an International Journal, 1(2), 130−140. Lievrouw, L. (2006). New media design and development: Diffusion of innovations versus social shaping of technology. In L. Lievrouw & S. Livingstone (Eds.), The handbook of new media (updated student edition) (pp. 246−265). London: Sage Publications. Macintosh, A., Robson, E., Smith, E., & Whyte, A. (2003). Electronic democracy and young people. Social Science Computer Review, 21(1), 43−54. McLaughlin, K., Osborne, S., & Ferlie, E. (2002). New public management: Current trends and future prospects London and New York: Routledge. Millard, J. (2003). ePublic services in Europe: Past, present and future. Research findings and new challenges. Final draft-paper. Prepared for IPTS, not published. Millard, J. (2006). User attitudes to E-Government citizen services in Europe. International Journal of Electronic Government Research, 2(2), 49−58. Moore, T. & Flynn, P. (2004). Future challenges for E-Government. The changing role of multi-channel service delivery. Discussion Paper no. 4. Australian Government Information Management Office. 2004. Retrieved from September 6, 2008 from http://www.finance.gov.au/Publications/future-challenges-for-egovernment/ docs/AGIMO-FC-no4.pdf MORI, (2002). Public service reform. Measuring and understanding costumer satisfaction. A Mori Review for the Office of Public Services Reform. London: The Prime Minister's Office of Public Services Reform. Muylle, S., Moenaert, R., & Despontin, M. (2004). The conceptualization and empirical validation of web site user satisfaction. Information and Management, 41(5), 543−560. Norris, D. F., & Moon, M. J. (2005). Advancing E-Government at the grassroots: Tortoise of hare? Public Administration Review, 65(1), 64−75. OECD. (2003). The E-Government imperative Paris: OECD E-Government Studies. OECD. (2005). E-Government for better government Paris: OECD E-Government Studies. Osborne, D., & Gaebler, T. (1992). Reinventing government: How the entrepreneurial spirit is transforming the public sector. Reading: Addison-Wesley. Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). Servqual: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1), 12−40. Pieterson, W., & Ebbers, W. (2008). The use of service channels by citizens in the Netherlands: Implications for multi-channel management. International Review of Administrative Sciences, 74(1), 95−110. Plouffe, C., Hulland, J., & Vandenbosch, M. (2001). Richness versus parsimony in modelling technology adoption decisions — Understanding merchant adoption of a smart card-based payment system. Information Systems Research, 12(2), 208−222. Prins, J. E. (2001). Designing E-Government. On the crossroads of technological innovation and institutional change. The Hague: Kluwer Law International. Punie, Y. (2004). Een theoretische en empirische benadering van adoptie, gebruik en betekenis van informatie- en communicatietechnologie in het dagelijkse leven. In N. Carpentier, C. Pauwels, & O. Van Oost (Eds.), Het on(be)grijpbare publiek: een communicatiewetenschappelijke exploratie van publieksonderzoek (pp. 175−200). Brussels: VUB Press. Reddick, C. G. (2005a). Citizen-initiated contacts with government: Comparing phones and websites. Journal of E-Government, 2(1), 27−53. Reddick, C. G. (2005b). Citizen interaction with E-Government: From the streets to servers? Government Information Quarterly, 22(1), 38−57. Relyea, H. C. (2002). E-gov: Introduction and overview. Government Information Quarterly, 19(1), 9−35.

497

Rogers, E. M. (2003). Diffusion of innovations, (5th ed.) New York: The Free Press. Roy, R. (2006). E-service delivery and new governance capacities: ‘Service Canada’ as a case study. International Journal of Services Technology and Management, 7(3), 253−271. Schedler, K., & Summermatter, L. (2007). Customer orientation in electronic government: Motives and effects. Government Information Quarterly, 24(2), 291−311. Silverstone, R., & Haddon, L. (1996). Design and the domestication of information and communication technologies: Technical change and everyday life. In R. Silverstone & R. Mansell (Eds.), Communication by design: The politics of information and communication technologies (pp. 44−74). Oxford: Oxford University Press. SVR (2008). Totale bevolking naar leeftijd en geslacht per gewest. Brussel: Studiedienst Vlaamse Regering. Retrieved October 2, 2008 from http://www4.vlaanderen.be/dar/ svr/cijfers/Exceltabellen/demografie/1_Bevolking/1_Gewesten/5_Bevolking_naar_ leeftijd_en_geslacht_per_gewest.xls Tat-Kei Ho, A. (2002). Reinventing local governments and the E-Government initiative. Public Administration Review, 62(4), 434−444. Taylor, S., & Todd, P. (1995). Understanding information technology usage: A test of competing models. Information System Research, 6(4), 144−176. Titah, R., & Barki, H. (2006). E-Government adoption and acceptance: A literature review. International Journal of Electronic Government Research, 2(3), 23−57. Traunmüller, R., & Wimmer, M. (2004). E-Government: The challenges ahead. Lecture Notes in Computer Science. 3183, 1–6. van Deursen, A., van Dijk, J., & Ebbers, W. (2006). Why E-Government usage lags behind: Explaining the gap between potential and actual usage of electronic public services in the Netherlands. Lecture Notes in Computer Science, 4084, 269−280. van Dijk, J., Peters, O., & Ebbers, W. (2008). Explaining the acceptance and use of government internet services: A multivariate analysis of 2006 survey data in the Netherlands. Government Information Quarterly, 25(3), 379−399. Van Dyke, T. P., Kappelman, L. A., & Prybutok, V. R. (1997). Measuring information systems service quality: Concerns on the use of the SERVQUAL questionnaire. MIS Quarterly, 21(2), 195−208. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425−478. Verleye, G. (1999). Missing data in linear modelling. KM Journal of the Dutch Statistical Society, 62, 95−110 special issue on missing data. Wejnert, B. (2002). Integrating models of diffusion of innovations: A conceptual framework. Annual Review of Sociology, 28(1), 297−326. West, D. M. (2004). E-Government and the transformation of service delivery and citizen attitudes. Public Administration Review, 64(1), 15−27. Whiteley, R. C. (1991). The customer driven-company: Moving from talk to action. Massachusetts: Addison-Wesley. Wimmer, M. A. (2002). Integrated service modelling for online one-stop government. Electronic Markets, 12(3), 149−156. Yildiz, M. (2007). E-Government research: Reviewing the literature, limitations, and ways forward. Government Information Quarterly, 24(3), 646−665. Zeithaml, V.A., Parasuraman, A. and Malhotra, A. (2000). A conceptual framework for understanding e-service quality: Implications for future research and managerial practice. MSI Monograph, Report #00-115 (Working Paper Series). Pieter Verdegem is a Ph.D. student and researcher at the Research Group for Media and ICT (MICT-IBBT), Ghent University. His research focuses on ICT acceptance, EGovernment, information society, ICT policy and e-inclusion. He holds a master's degree in Communication Sciences and E-communications Studies. He is preparing a Ph.D. on government strategies particularly E-Government and e-inclusion. Gino Verleye owns a Ph.D. in psychometrics and teaches Methodology and Statistics at Ghent University, Department of Social Sciences. He is currently research coordinator at the Research Group for Media and ICT (MICT-IBBT), specialized in the area of research on new broadband applications.