Using balanced scorecards for the evaluation of “Software-as-a-service”

Information & Management 50 (2013) 553–561

Contents lists available at ScienceDirect

Information & Management journal homepage: www.elsevier.com/locate/im

Case studies in research

Using balanced scorecards for the evaluation of ‘‘Software-as-a-service’’ Sangjae Lee a, Sung Bum Park b,*, Gyoo Gun Lim c a

School of Business Administration, Sejong University, Republic of Korea National Information Society Agency, Republic of Korea c School of Business, Hanyang University, Republic of Korea b

A R T I C L E I N F O

A B S T R A C T

Article history: Received 17 August 2012 Received in revised form 24 January 2013 Accepted 11 July 2013 Available online 1 August 2013

To overcome the problem of limited resources, increasing numbers of small- and medium-sized companies (SMEs) are adopting ‘‘Software-as-a-service’’ (Saas) as an efficient tool for IS implementation. The balanced scorecard (BSC) has been adopted by SMEs to evaluate Saas via four measures: learning and growth, internal business processes, customer performance, and financial performance. The survey results for 101 Software-as-a-service adopters indicate that learning and growth, internal business processes, and customer performance are causally related to financial performance. The results show that these four key elements for Saas success are interrelated, supporting the core premise of the BSC. ß 2013 Elsevier B.V. All rights reserved.

Keywords: Software-as-a-service (Saas) Balanced scorecard (BSC) Causal relationships

1. Introduction IS maintenance costs comprise a major portion (70%) of the total IS implementation costs. Companies choose to invest their resources and manpower in their core capability to provide products or services. The demand for IT outsourcing and the Software-as-a-service (Saas) model, which integrates network, hardware, and software, is increasing as IT sophistication itself increases [31]. Saas can be defined as applications and computer-based services delivered and managed from a remote center to multiple customers via the Internet or a VPN. Saas shares common themes with On-Demand Service [27]. There is a growing use of other related and advanced platform services, such as cloud computing, infrastructure-as-a-service (Iaas) and platform-as-a-service (Paas), representing a large pool of usable resources, such as hardware and software, that are easily accessible via the Internet [11]. It is estimated that by 2013, the cloud market will have reached $8.1 billion [3]. Industry leaders predict that revenues from cloud computing enterprises will grow to $160 billion. Experts describe cloud computing as ‘‘an emerging IT development, deployment and delivery model, enabling real-time delivery of products, services and solutions over the Internet’’ [16].

* Corresponding author. E-mail addresses: [email protected] (S. Lee), [email protected] (S.B. Park), [email protected] (G.G. Lim). 0378-7206/$ – see front matter ß 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.im.2013.07.006

These attempts at ‘‘utility computing’’ are taking off due to the availability of sufficient bandwidth for such services. Broadband communication has become cheap and plentiful enough for utilities to deliver computing services with the speed and reliability that businesses previously enjoyed from their local machines. Typical Iaas offerings include Amazon’s Elastic Compute Cloud (EC2) and Simple Storage Service (S3), Joyent’s Accelerator, and Rackspace’s Mosso. The possible reasons for adopting cloud computing include (1) avoiding capital expenditure in hardware, software, and IT support and (2) using the flexibility and scalability of IT resources. The major issues of cloud computing include integrity of services and data, confidentiality of corporate data, and reliability perceptions, to name a few [5,8]. The Saas provider acts as a mediator, mediating services between independent software vendors (ISVs). Saas customers do not possess, manage or maintain the applications, but only use them as final products by accessing services with IT support. While Saas is advantageous in that it reduces the repair costs of application-based construction and maintenance, the risk of data leakage becomes a major disadvantage because application servers are constructed by outside companies. Saas is one type of ASP (application service provider service). After being introduced between 1998 and 1999, the ASP service market increased rapidly from 2000 to 2001 due to excessive expectations and ASP service provider mergers and acquisitions (M&A). While the number of customers and the market size are continuously growing, the growth rate is plateauing. As the ASP market worsens, it is essential to improve ASP planning and management, as it is now harder for many ASP providers to survive

554

S. Lee et al. / Information & Management 50 (2013) 553–561

[10]. For ASP services to be successful, system and service qualities must be well prepared [32]. In small- and medium-sized companies (SMEs), both ASP service and information quality are significant factors in enhancing user satisfaction, trust, and the intention to use [38]. ASPs are said to have achieved considerable success with big businesses, but their success has been less notable with SME markets [1]. For example, lack of ASP customization options and concerns about financial stability, service reliability, and functional capability flexibility are problems for SMEs. For these reasons, SMEs are less likely to adopt ASP services than large companies, making it necessary to develop empirical studies to examine how specialized measures may be applied to SMEs. Outcome assessment in the traditional field of business administration places excessive emphasis on financial performance measures [49]. Because it is impossible to assess an organization in a competitive environment based merely on financial performance, it is necessary to measure Saas performance by using the balanced scorecard (BSC) approach, which balances leading and lagging indicators, as well as by using financial and non-financial measures [29]. To provide a balanced approach to the measurement of organizational performance, including sub-areas, such as knowledge management (KM), business processes, and financial performance, BSC measures four categories: learning and growth, internal business processes, customer performance, and financial performance. Small companies develop multiple scoreboards, each tailored to the strategy and goals of a specific subunit. To survive in today’s global and volatile business environment, SMEs are using newer management systems, such as BSCs, to clarify their vision and strategy and to translate them into action [35]. BSCs also include a financial perspective because such a perspective can easily summarize previous financial activities and yield predictable economic outcomes. The financial perspective indicates whether a strategic operation contributed to net profit improvement. The customer category assesses the extent to which the target market was captured. The internal business processes category focuses on core processes aimed at customer satisfaction and at financial objective achievement. The learning and growth category assesses the construction of necessary longterm growth and improvement infrastructure [30]. BSCs have been used to measure the performance of a wide variety of businesses. For instance [71], demonstrated the application of BSCs at an institutional level in a collaborative effort to develop a performance measurement framework for the Food Research Institute (FRI). Withanachchi et al. [65] applied BSCs to evaluate an organizational development program (TQM) that was implemented at a tertiary-care public hospital. Moreo et al. [46] suggested that BSCs could be used by managers to quantify the environmental and financial impact of a company and to help environmental quality to stakeholders, including hospitality owners and stockholders. Smandek et al. [55] developed and implemented a BSC system for IP management to optimize licensing income generation, cut costs, and keep inventors’ motivations high. Homburg et al. [24] tested and applied BSCs to marketing performance management to show the comprehensive relationship between a marketing performance measurement system and firm performance as conditional on marketing alignment and market-based knowledge. Taylor and Baines [60] applied BSCs to evaluate the performance management of higher education in terms of the formation, monitoring, and evaluation of strategy and policy, as well as issues of motivation. While BSCs have been applied in various contexts, empirical studies on the application of BSCs to specific IT services, such as Saas, are lacking. Further, while SMEs have difficulty rationalizing their operational practices and strategic processes, there is a dearth of comprehensive performance management system applications

by SMEs [18]. Despite the wide scope of BSC application as a decision-supporting instrument, cause-and-effect relationships and time-delayed elements between measures are still elusive [47]. While causal relationships among the four BSC measurements are the core focus of BSC, empirical studies (via the testing of causal relationships among categories) examining how well BSCs can be applied are almost nonexistent. This study intends to fill this gap. This study suggests some measures to evaluate Saas and tests these measures using data collected from companies that have adopted Saas. The discussion and implications are included in the study. 2. Theoretical background 2.1. ASP service evaluation models The criteria considered for ASP service selection are credibility, appropriateness, and efficiency, which further includes ‘‘prior experience,’’ ‘‘ASP service expectation,’’ ‘‘perceived provider performance,’’ and ‘‘expectation-disconfirmation.’’ Kern et al. [31] suggested six propositions based on resource dependency theory, resource-based theory, transaction cost theory, and agency theory. These propositions are as follows: (1) the use of an ASP service is a strategic decision to supplement necessary parts of IS; (2) ASP customer service depends heavily on the ASP service; (3) ASP service generally has a lower cost; (4) ASP service prices increase over time; (5) SMEs are more interested in ASP service than are large businesses; and (6) the suitability of an ASP service is determined by the capability of the customer and the variety of the ASP service. Kern et al. articulated specific characteristics derived from the relationship between ASP service providers and their customers. This framework for measuring the SME benefits and risks of Internet-based applications is designed for companies that do not have any specific team or methodology for measuring proposed IT outsourcing. Currie [10] suggested five categories for measuring the risk and benefit of applications provided by an ASP service, including delivery and enablement, integration, management and operation, business transformation, and customer/vendor relationship. This framework provides a difference in viewpoint between ASPs and their customers regarding key performance indicators (KPIs) of ASP services. Susarla et al. [58] suggested that perceived provider performance has a positive impact on user satisfaction with an ASP. Further expectations about ASP services have a significant influence on any performance evaluation of ASPs. Leam and Lee [37] proposed items for auditing and verifying the reliability of ASP service through a survey of 35 Korean companies. By using categories, including network, data center, application, and security and customer support, Leam and Lee evaluated three ASP vendors in Korea. Leam and Lee derived the ASP life cycle and auditing items based on the evaluation results. The items for auditing ASPs were classified by function and performance into 12 items, and the relative importance of those 12 items was investigated. The items for auditing an ASP focus on assessing usefulness, extensibility, and application usability. Zviran et al. [72] suggested that perceived usefulness is one of the factors affecting user satisfaction with enterprise resource planning (ERP) systems, which are typical examples of ASP services. Kim et al. [33] identified three characteristics of ASPs that affect the satisfaction of ASP customers: stability, IT infrastructure, and service flexibility. Kim and Kim [32] suggested five characteristics of ASPs that affect ASP customer satisfaction: credibility, system currency, security, acceptability, and system support. Susarla et al. [59] used prior literature in transaction cost economics (TCE) to posit that the contract design for an ASP service should consider such factors as uncertainty in specifying service requirements,

S. Lee et al. / Information & Management 50 (2013) 553–561

interdependence between the ASP application and the IT systems in the client organization, and the need for specific investments to address transaction costs that result due to contractual incompleteness and opportunism. Schwarz et al. [53] suggested 10 attributes that firms consider when deciding on the outsourcing of applications and indicated that the three most significant drivers of an IT application service choice were cost, risk, and vendor capability. Yao et al. [70] suggested that firms who specialize in providing software applications to other firms over the Internet should be evaluated in terms of measures, such as the possibility of using an ASP service, software customization requirements, financial stability, and exit strategies. However, these measures are limited, as they are based only on the BSC customer’s perspective. Other BSC perspectives should also be added to Saas evaluation measures. 2.2. The balanced scorecard The need for a comprehensive performance measurement, including knowledge components, has motivated the development of the balanced scorecard (BSC). The BSC attempts to integrate all interests of key stakeholders, such as shareholders, customers, and employees, on a scoreboard [30]. Recent studies have been conducted on BSC application in various fields, including the fields of organizational and IS performance evaluation. Lee et al. [39] proposed an approach based on the fuzzy analytic hierarchy process (FAHP) and balanced scorecard (BSC) processes for evaluating an IT department in the Taiwanese manufacturing industry. The results offered guidance for Taiwanese IT departments in the manufacturing industry regarding strategies for improving department performance. Huang [25] suggested the use of an analytic hierarchy process (AHP) to prioritize the measures and strategies of a BSC framework. This was done with the intention of developing an intelligent BSC planning and management decision support system for strategic planning. Wu et al. [69] suggested a Fuzzy Multiple Criteria Decision Making (FMCDM) approach for banking performance evaluation based the four perspectives of BSCs. Wu et al. summarized performance indexes to fit the banks to construct a hierarchical framework of performance evaluation; Asosheh et al. [2] adopted an integrated use of the balanced scorecard (BSC) and data envelopment analysis (DEA), and proposed a new approach to IT project selection. This approach used BSCs as a comprehensive framework for defining IT project evaluation criteria and adopted DEA as a nonparametric technique for ranking IT projects. Velcu [63] tested the interrelations between strategic alignment, management of ERP implementations, process changes, and the business performance of companies implementing ERP, and their study was based on a balanced scorecard approach to analyzing business performance. Kunz and Schaaf [36] suggested an expert system for health care management for the formalization of BSC evaluation. They provided a general definition of an indicator system for each BSC category for clinical application. BSCs will be able to represent and deliver strategies according to each business, rather than offering the simple combination of financial and non-financial indicators sorted by individual categories. The strategy of BSC organization is represented by performance connected through causal relationships and ongoing performance indicators. In general, final performance indicators, or ‘‘lagging indicators,’’ are represented by improved financial performance, which is the ultimate goal of the strategy. Meanwhile, ongoing performance indicators, or ‘‘leading indicators,’’ inform all organization members of what will be performed in learning and business processes [30]. Because a strategy is the hypothetical representation of cause and effect regarding the capability of an organization to correspond to market changes to

555

achieve a goal, such a strategic causal relationship can form a set of causes and effects. Thus, the result (lagging indicator), focused on each business’ strategy, may be appropriately connected to the performance cause (leading indicator), allowing all indicators in each BSC category to be related to the financial goal. 3. Research model Despite the wide scope of BSC application as a decisionsupporting instrument, few empirical studies on the application of BSC to specific IT services such as Saas have been conducted (Fig. 1). Further, it is necessary to support SMEs with specialized performance measurement systems such as BSCs. As a way to accomplish these research objectives, this paper examines causal effects in each BSC category to determine how well BSCs can be applied to Saas. BSC proponents expect that companies who continuously improve their capabilities (e.g., by implementing advanced workplace practices, monitored via innovation and learning) should achieve better performance for their customers. All such efforts should lead to improved financial performance. Sim and Koh [54] positively verified a causal relationship between several individual performance indicators among the four BSC categories through relationship and regression analysis. They used data obtained from 83 electronics companies in the U.S. BSC can be used as a tool to align business processes with new strategies, moving away from cost reduction and toward growth opportunities based on more customized, value-adding products and services. Wu [68] used a multiple criteria analysis tool to determine causal relationships between key performance indicators for each BSC perspective to create a visualized strategy map with logical links to improve banking performance. If cause-and-effect relationships are not adequately reflected on a balanced scorecard, the scorecard will not translate to a company’s vision and strategy. Sohn et al. [56] posited that a BSC’s merit is that it seeks a balance between financial and nonfinancial measures, categorized into financial, customer, and internal business processes, and innovation and learning factors. Firms using IS must determine which specific BSC measures to focus on and which to ignore. However, relatively few studies have been carried out on the specific weights of each BSC perspective. Northcott and Taulapapa [48] indicate the need for improved theorization on several issues that present particular challenges for BSC practice in the public sector: BSC measure modification, measure design that captures important qualitative outcomes, ‘‘customer’’ suggestion and achievement of a genuine multistakeholder approach, and mapping BSC causality relationships. In an era of global economic competition, organizations should be creative in learning. Organizational capability for learning and creating knowledge is the cornerstone for operating businesses. Many researchers in the field of knowledge management (KM) have emphasized the role of learning in knowledge management systems (KMS) based on the interrelatedness of learning and knowledge. Organizational knowledge is defined as the improvement of actions through knowledge acquisition. Organizational

Learning and Growth

H2 Customer’s Performance

H1

Financial H4 Performance

H3 Internal Business Process

Fig. 1. Research model – causal relationship among Saas measures.

556

S. Lee et al. / Information & Management 50 (2013) 553–561

learning occurs when individuals and subunits acquire knowledge after understanding the possibility of organizational change. Organizational learning improves the potential capability for effective actions of organizations and individuals through improved business processes and customer service. In KM, organizational learning is established across entire organizational hierarchies when the skills and knowledge acquired by individuals are internalized into organizational entities. Organizations become efficient and skillful initiators of change by creating, acquiring, and transferring knowledge. Organizational improvement is determined by learning processes that require the addition or change of mental models. Organizations can create knowledge continuously from inquiry into understanding of new environments [13]. Improvement in customer service depends on the effectiveness of situational behaviors in their contingencies. Under the rapid, innovative, and discontinuous change of environments, the assumptions lying beneath work practices stored in a knowledge repository should be continuously improved for better customer service. Thus, organizations that do not engage in creative learning through KM increase entropy, which ultimately leads to failure in customer service. Top management’s view of the business environment should be adjusted in a timely manner according to changes in environments. Successful customer service companies have better and more rapid creative learning. Mason [43] indicated that one crucial measure, besides financial performance, is learning, which is necessary in assessing the extent to which organizations acquire new knowledge to maintain competitiveness. Pfeffer [51] showed that organizational perception can determine organizational performance and that creative learning can result in innovation and dramatic performance improvement. This section examines the effects of learning and growth on customer service improvement for Saas performance. The learning and growth of companies is a fundamental force driving customer service performance and customer relationship management. For example, better staff skills will reduce the frequency of bugs in an application. An application with fewer bugs will be more likely to meet end-user expectations. This, in turn, will enhance the support of customer service processes. Nonfinancial indicators, such as the customer satisfaction index and customer loyalty, are prerequisites and pre-conditional indicators for financial indicators [4,26]. Customer performance has been proven to be a decisive factor in financial performance, as learning and growth affect financial performance through customer performance. For instance, using data from multinational corporations, Gonzalez-Padron et al. [21] used BSCs to assess how organizational learning affects actions relating to global marketing strategy and subsequent financial performance. Thus, organizational change based on learning is a crucial factor for improved organizational process and customer service. Top management’s ability to understand and learn in uncertain and competitive environments is important for business processes and customer service. Top management interprets information on behalf of organizations, and the creative learning of top management likely affects the creative learning of organizations. Thus, organizations should acquire knowledge of business environments and perform learning processes faster than competitors to maintain good business and customer relations. This increases the demand for efficient processes based on KM tools and new infrastructure capability [6]. Hypothesis H1. Learning and growth positively affect internal business processes in the client’s organization after Saas is offered. Hypothesis H2. Learning and growth positively affect the customer’s performance after Saas is offered.

In addition, how the effect of internal business processes on both customer and financial performance, as well as customer performance, affect financial performance can be investigated. The goals of internal business processes in the BSC model are to innovate and improve the process of identifying and satisfying customer demand, as well as to provide excellent customer management service afterward. Customers can then recognize that the service company provides the best customer value, and thus customer satisfaction increases. The customer’s performance facilitates market share and customer profitability in the target market, allowing financial goals to be achieved. Previous studies have suggested that customer and financial performance BSC measures are causally interrelated. For instance, Ittner and Larcker [26] studied the relationship between customer satisfaction and financial performance by using various data sorted by company, business, and customer. As a result, the measured value of customer satisfaction had a significant impact on future financial performance. Behn and Riley [4] demonstrated, in their research, the announcement that customer satisfaction had an effect on American airline companies’ future financial performance, i.e., that non-financial indicators function as leading indicators of quarterly financial performance. The ultimate aim of many balanced IS scorecards is to support IS management, such that the overall financial outcomes of the enterprise are improved [42]. This study suggests, based on this premise, that balanced IS scorecards should improve financial performance. To realize financial performance, customer performance should be improved. The determinants for customer performance are, in turn, internal business processes, and learning and growth [14]. Gonzalez-Padron et al. [21] confirmed that customer performance affects financial performance. Learning and growth facilitate internal business processes. These facts lead to the following hypotheses: Hypothesis H3. Internal business processes positively affect customer performance after Saas is offered. Hypothesis H4. Customer performance positively affects financial performance after Saas is offered. 4. Research method 4.1. Measurement of variables Most of the variables in the model are measured by items written in the form of a statement with which the respondent is to agree or disagree on a 7-point Likert-type scale. The latent variables and measurement items used in this study are shown in Table 1. The measurement items were measured by partially revising the leading and lagging indicators suggested in the BSC model. The research variables consisted of four measures, including learning and growth, internal business processes, customer performance, and financial performance. The measures are based on and adapted from previous studies on intellectual capital or BSCs, such as [7,14,22,42,44,40,50,57], and so on. The measure of learning and growth represents the extent to which using Saas continued employee learning and development, as well as knowledge sharing, to make efficient use of business resources. Saas and knowledge sharing can promote learning and development in an organization. Internal business processes indicate the extent to which value is brought to the customer via efficient use of business resources. Customer performance indicates the extent to which the customer market and service are improved. Financial performance indicates return on investment and created shareholder value.

S. Lee et al. / Information & Management 50 (2013) 553–561

557

Table 1 Research variables. Latent variables

Measurement items

Sources

Learning and Growth (LG)

The work process is processed extensively through Saas (LG1). Extensive knowledge sharing within the organization exists after Saas is adopted (LG2). Extensive knowledge sharing with other organizations (trading partners) exists after Saas is adopted (LG3).

Roos and Roos [52] Epstein and Manzoni [14] Kaplan and Norton [29,30] Harvey and Lusch [22] Lipe and Salterio [40] Kim et al. [33]

Internal Business

The customer order processing time is greatly reduced after Saas is adopted (IP1).

Martinsons et al. [42]

Process Performance (IP)

The customer request time limit is strictly followed after Saas is adopted (IP2).

Olve et al. [49] Kim et al. [33]

Customer Performance (CP)

Customer service is greatly improved after Saas is adopted (CP1). The company experiences a great annual average increase in its customer market (CP2).

Hoffecker and Goldenberg [23] Butler et al. [7] Frigo et al. [17] Epstein and Manzoni [14] Harvey and Lusch [22] Martinsons et al. [42] Petty and Guthrie [50] Lipe and Salterio [40] Stewart [57]

Financial Performance (FP)

The ordering cost is greatly reduced after Saas is adopted (FP1). The company’s revenue is greatly increased every year after Saas is adopted (FP2).

Hoffecker and Goldenberg [23] Butler et al. [7] Johnson [28] Epstein and Manzoni [14] Harvey and Lusch [22] Kaplan and Norton [29,30] Lipe and Salterio [40] Mendoza and Zrihen [45]

Learning and growth include the percentage of work processed through Saas, and knowledge sharing within an organization and with other organizations (trading partners) after Saas is offered. The category of internal business processes is composed of customer order processing time reduction, observance of customer request time limits, and the improvement of customer service after Saas is offered. Customer performance consists of the annual average increase in the customer market, cost reduction in ordering, and revenue increase per year after Saas is offered. 4.2. Data collection Five-hundred interviewees were selected randomly among SMEs registered by the Small & Medium Business Administration in Korea. Data were collected using a structured questionnaire, in Korean, through telephone and personal interviews with IT staff members who used Saas. The sample was composed of companies that are diverse in terms of business area, corporate size, and so on. Among the 500 corporations, 101 corporations were included in the final sample, after excluding missing values and incomplete responses. IT corporations accounted for 51% (51 corporations) of the 101 corporations that responded, while non-IT corporations made up 49% (49 corporations) of the sample. The descriptive statistics of the sample data are shown in Table 2. Organizations within the sample had an average of 79 employees and had adopted Saas for an average of three years. The average percentage of Saas use was 45.2%. The majority of applications using Saas were core businesses (sales, production, purchasing, and logistics applications) (71%). 5. Results 5.1. Assessment of measurement model To ensure convergent and discriminant validity, exploratory factor analysis with Varimax rotation was performed to produce a

Table 2 Profile of sample. Number of companies

Percentage (%)

22 19 51 8 100

22 19 51 8 100

Size of company (Number of employees) (mean = 79) 1–10 39 11–50 42 More than 50 19 Total 100

39 42 19 100

Years of Saas adoption (years) (mean = 3) 1–2 3–4 More than 4 Total

43 38 19 100

Industry of the company Manufacturing industry Retail industry Service industry Others Total

43 38 19 100

Percentage of Saas use in total processes (%) (mean = 45.2) 0–10 22 11–50 41 More than 50 37 Total 100

22 41 37 100

Monthly Saas usage fee (dollar) (mean = 85) 0–10 11–30 31–100 More than 100 Total

41 20 24 15 100

41 20 24 15 100

36

36

71

71

32 100

32 100

Applications using Saas Supporting applications (accounting, personnel, and administration) Core business (sales, production, purchasing, and logistics applications) Business-to-business applications Total

S. Lee et al. / Information & Management 50 (2013) 553–561

558 Table 3 Exploratory factor analysis.

LG1 LG2 LG3 IP1 IP2 C1 C2 F1 F2

Learning and Growth (LG)

Internal Business Process Performance (IP)

Customer’s Performance (CP)

Financial Performance (FP)

0.814 0.929 0.851 0.674 0.752 0.716 0.701 0.586 0.384

0.383 0.651 0.014 0.954 0.966 0.825 0.754 0.612 0.505

0.372 0.683 0.013 0.711 0.85 0.966 0.965 0.682 0.571

0.346 0.462 0.037 0.557 0.593 0.624 0.674 0.949 0.927

The bold values are the largest factor loading in the low.

cross-loading matrix. The matrix shows a correlation between latent variables and measurement items, suggesting that the measurement items load highly on their theoretically assigned latent variables, while not loading highly on other factors [19]. Table 3 shows the cross-loadings of individual items compared across all latent variables. Each item is assigned highly to a respective latent variable, thus ensuring convergent and discriminant validity. Data analysis was performed using the partial least squares (PLS) method. The use of PLS has increased in IS fields [9]. PLS, a component-based structural equation modeling technique, is similar to regression but simultaneously models paths among variables. A large sample size is required, and standard distribution is assumed for SEM. In contrast, PLS, introduced by Wold et al. [66], focuses on maximizing the variance of dependent variables explained by an independent variable. The PLS approach places minimal restrictions on sample size and residual distribution. In addition, a PLS algorithm allows each indicator to vary in how much it contributes to the composite score of the latent variable instead of assuming equal weights for all indicators of a scale. Tobia [61] posited that PLS is useful in screening out negligible factors affecting the dependent variable. A reliability test was performed using Cronbach’s Alpha and composite reliability (CR). Table 4 shows that there is no significant defect in internal consistency. CR greater than 0.70 implies that a construct retains both internal consistency and convergent validity [64]. Table 4 additionally illustrates how each item contributes to the total reliability of each construct. Tests were only conducted for the construct variable ‘‘LG’’, which has more than three indicators. To test validity, factor loading and average variance extracted (AVE) were examined. AVE measures the variance percentage captured via a construct by showing the variance sum ratio captured by the construct and the measurement variance. It is acceptable when an individual item factor loading is greater than 0.70 and an AVE exceeds 0.50 [20].

Table 4 shows individual item factor loading and each latent variable’s CR and AVE. All loadings are significant at a = 0.001. Because the CRs of all eight-measurement items shown in Table 4 are higher than 0.70, all latent variables can be said to have convergent validity. The factor loadings of all items exceed 0.70. Therefore, the research variables have strong reliability and validity. Additionally, the AVE square root can be used to examine discriminate validity. As suggested by Fornell and Larcker [15], diagonal elements should be larger than the entries in corresponding rows and columns. Table 5 indicates that the AVE square root of a latent variable (the entry in the diagonal of Table 5) is larger than the correlation of other latent variables. The correlations among all constructs are well below 0.70, suggesting that all constructs are distinct from one another. 5.2. Testing of research hypotheses The structural model estimation and research hypotheses tests were performed using the PLS method, as shown in Fig. 2 and Table 6. In the entire sample, three of four path coefficients are higher than 0.5 (LG-IP: 0.746; IP-CP: 0.610; CP-FP: 0.672). All of the paths in the research model are significant and positive. All four research hypotheses are acceptable for the entire sample, as well as for the non-IT corporation sample. This acceptability indicates that causal

Table 5 Correlations among latent variables. (The numbers in diagonals are the square root of the average variance extracted.). Latent variables

LG

IP

Learning and Growth (LG) Internal Business Process Performance (IP) Customer’s Performance (CP) Financial Performance (FP)

0.834 0.734

0.960

0.746 0.526

0.818 0.6

CP

FP

0.966 0.672

0.960

Table 4 Reliability and validity of research variables. Measurement items Learning and Growth (LG) LG1 LG2 LG3

Loading

0.8290 0.8537 0.8175

Internal Business Process Performance (IP) IP1 0.9603 IP2 0.9603

Composite Reliability (CR)

LG2

LG1

LG3

0.852**

0.814**

0.826**

Learning and Growth

0.872

H2 0.610**

H1 0.746**

H3

0.960

H4 0.672**

Customer’s Performance

Financial Performance

0.279* 0.956**

0.965**

0.950**

0.927**

Internal Business

Customer’s Performance (CP) CP1 CP2

0.9659 0.9659

0.965

Financial Performance (FP) FP1 FP2

0.9603 0.9603

0.937

CP1

Process

CP2

FP1

FP2

0.954** 0.966**

IP1

IP2

Fig. 2. PLS analysis results for the entire sample. *p < 0.05, **p < 0.01.

S. Lee et al. / Information & Management 50 (2013) 553–561

559

Table 6 Test of research hypotheses. Research hypotheses H1: H2: H3: H4: * **

Learning and Growth ! Internal Business Process Performance Learning and Growth ! Customer’s Performance Internal Business Process Performance ! Customer’s Performance Customer’s Performance ! financial performance

Entire sample **

0.746 0.279* 0.610** 0.672**

Sample of IT companies **

0.746 0.107 0.760** 0.576**

Sample of non-IT companies 0.761** 0.452** 0.467** 0.773**

p < 0.05. p < 0.01.

relationships exist among learning and growth, internal business processes, customer performance, and financial performance. This implies that for Saas customer companies, the financial category of BSC measurement focuses on generating satisfactory return on investment and creating shareholder value, while the three other categories can be explained as determinants of financial success. Customer performance represents value creation for customers and is important in explaining how customers perceive their performance. The important way to increase customer service and market is to have massive numbers of employees servicing customers, or have fewer employees who use time management more efficiently and who are supported by excellent IT-based processes. Creating value for customers only translates into shareholder value if it is based on effective and efficient key internal processes. Finally, the study results indicate that to ensure that the Saas customer company will be appreciated by future customers and will continue to make excellent use of its processes, the organization and its employees must continue organizational learning growth. The learning and growth category should thus group indicators capturing the company’s performance through the use of Saas with respect to learning based on information sharing and innovation. In IT corporations, learning and growth fail to affect customer performance. In contrast, learning and growth significantly influence customer performance in non-IT corporations. Therefore, corporation type – IT vs. non-IT – moderates and affects the relationship between learning and growth, and customer performance. For non-IT companies, learning and growth are more important for the improvement of customer service and market than they are for IT companies. Knowledge sharing and Saas use are considered natural and taken for granted by IT companies, as these processes have a lesser influence on customer service and market. For non-IT companies, there exists more opportunity for improvement in customer service and market through learning based on information sharing and use of Saas. 6. Implications 6.1. Implications for researchers Despite rapid growth in the Saas market, no research has been conducted on Saas measurement based on BSCs. This study provides an empirical study on BSC application to Saas and shows the application of comprehensive performance management systems for SMEs, considering that SMEs have a hard time rationalizing their operational practices and strategic processes. This study indicates that significant causal relationships exist among BSC measures in the Saas context, which strengthens the justification for BSC application as a decision-supporting instrument, given that the causal relationships among the four BSC measures are the focus of BSC assessment. By showing these causal relationships, this paper provides a rationale for SMEs to use BSCs to assess Saas. This study extends previous studies on ASP success in that the causal relationships among Saas financial performance determinants are investigated. This study provides an assessment of comprehensive performance management systems for SMEs of

specific IT services, such as Saas. Further, this study provides an empirical basis for using BSC as a decision-supporting instrument by illuminating cause-and-effect relationships and time-delay elements among measures (these relationships are still elusive [47]). This study uses BSCs as a lens for suggesting four measures. The intention was to determine the leading and lagging indicators. This determination improves the understanding of leading and lagging indicators of BSC categories based on the balance between financial and non-financial analysis. 6.2. Implications for practitioners SMEs, which comprise the majority of the firms in the study sample, consider investment in enterprise applications to be costly and tend to avoid taking efforts away from their main business area. To focus on main business activities, SMEs are more likely to use Saas, as it can provide a viable alternative to costly in-house information systems. The high use of Saas and the high percentage of its use in core applications show that SMEs use Saas strategically to fill a gap in IS resources and capabilities, given that Saas provides a cost-efficient and rapid solution to these difficulties. Knowledge acquisition on Saas was often informal and ad hoc, usually induced by vendor hype rather than through a rigorous evaluation of vendor capabilities. To address this problem, BSCs can provide a template for evaluating the performance of application outsourcing. The results of this study suggest that BSCs effectively assess four causally interrelated measures, i.e., learning and growth, internal business processes, customer performance, and financial performance. To increase financial measures, customer performance should be promoted. This, in turn, requires the enhancement of learning and growth and internal business processes. Financial measurement is the most lagging indicator and is effective in summarizing previous financial activities into easily predictable economic results. Customer performance assesses the extent to which the target market is captured. To increase financial measures, customer performance should be improved and promoted. To increase customer performance, employee learning and internal business process efficiency should be improved. Internal business processes are focused on customer satisfaction and financial objective achievement. Learning and growth assess the necessary infrastructure constructed for long-term growth and improvement. In a knowledge economy society, learning knowledge is the basis for creating new capital, and the purpose of business is to create new knowledge. Thus, continuous learning and experimentation are necessary to produce new ideas and products; it is critical to stress the importance of an organizational culture supportive of learning. The designers and developers of KMS should focus on developing an integrated KM system that facilitates knowledge acquisition and application, the two subprocesses affecting creative organizational learning. KMS should enhance both the efficiency of existing knowledge use by processing information and the effectiveness of new knowledge creation [41]. This study reinforces arguments that KMS developers should

560

S. Lee et al. / Information & Management 50 (2013) 553–561

concentrate on both creation of new knowledge and the use of existing knowledge. 7. Conclusion Because it is impossible, in a competitive environment, to assess an organization merely according to financial performance measures, it is necessary to measure the performance of Saas by using a BSC, which balances leading and lagging factors, and which also considers financial and non-financial factors. This paper examines the causal relationships among the four BSC categories that explain the performance of Saas. Some issues for future research exist. First, practical case studies of corporations that have adopted BSC-based performance monitoring systems are a promising avenue for study. This line of research can practically prove that key performance measures of learning and growth improve other measures, such as customer performance and internal business processes, as time passes. Second, in analyzing moderating effects, other moderating variables, such as corporate strategy type, market positioning type, and main service and product type, can be used. Further, it is necessary to carry out future studies to validate the reason that key measures of learning and growth do not affect customer performance in IT corporations. This study has some limitations. First, due to the insufficient sample size, it is not possible to further sub-categorize the sample to test moderating effects. More detailed categories must be considered in terms of IT usage or sophistication, rather than just two categories (i.e., IT corporations versus non-IT corporations), as used in this study. Second, there could be missing or unconsidered measurement items for learning and growth, internal business processes, customer performance, and financial performance, even though this study developed or adapted each item based on previous literature. Each Saas-adopting corporation has its own main business field and performance characteristics, and this can also affect the response data. References [1] F. Altaf, D. Schuff, Taking a flexible approach to ASPs, Communications of the ACM 53 (2), 2010, pp. 139–143. [2] A. Asosheh, S. Nalchigar, M. Jamporazmey, Information technology project evaluation: an integrated data envelopment analysis and balanced scorecard approach, Expert Systems with Applications 37 (August (8)), 2010, pp. 5931–5938. [3] BBC News, Cloud computing for business goes mainstream, 06 May, 2010 http:// www.bbc.co.uk/news/10097450. [4] B.K. Behn, R.A. Riley, Using non-financial information to predict financial performance: the case of the U.S. airline industry, Journal of Accounting, Auditing and Finance 14, 1999, pp. 29–56. [5] T.S. Behrend, E.N. Wiebe, J.E. London, E.C. Johnson, Cloud computing adoption and usage in community colleges, Behaviour & Information Technology 30 (2), 2011, pp. 231–240. [6] J. Bergman, A. Jantunen, J.M. Saksa, Managing knowledge creation and sharing – scenarios and dynamic capabilities in inter-industrial knowledge networks, Journal of Knowledge Management 8 (6), 2004, pp. 63–76. [7] A. Butler, S.R. Letza, B. Neale, Linking the balanced scorecard to strategy, Long Range Planning 30 (2), 1997, pp. 242–253. [8] D. Catteddu, G. Hogben, Cloud computing: benefits, risks and recommendations for information security, European Network and Information. Security Agency (ENISA) 2009, pp. 1–125. [9] D.R. Compeau, C.A. Higgins, Application of social cognitive theory to training for computer skills, Information System Research 6 (2), 1995, pp. 118–143. [10] W.L. Currie, A knowledge-base risk assessment framework for evaluating webbased application outsourcing projects, International Journal of Project Management 21 (3), 2003, pp. 207–217. [11] M. Cusumano, Technology strategy and management: cloud computing and SaaS as new computing platforms, Communications of the ACM 53 (4), 2010, pp. 27–29. [13] A. DuToit, Knowledge: a sense making process shared through narrative, Journal of Knowledge Management 7 (3), 2003, pp. 27–37. [14] M. Epstein, J.-F. Manzoni, Implementing corporate strategy: from tableaux De Bord to balanced scorecards, European Management Journal 16 (2), 1998, pp. 190–203. [15] C. Fornell, D.F. Larcker, Evaluating structural equation models with unobservable variables and measurement error, Journal of Marketing Research 18 (1), 1981, pp. 39–50.

[16] G.A. Fowler, B. Worthen, The internet industry is on a cloud – whatever that may mean, Wall Street Journal – Eastern Edition 253 (70), 2009, pp. A1–A10. [17] M.L. Frigo, K.R. Krumwiede, Automating the Balanced Scorecard – Maximizing Corporate Performance Through Successful Enterprise-Wide Deployment, Gentia Software plc, 1998. [18] P. Garengo, S. Biazzo, Unveiling strategy in SME through balanced scorecard implementation: a circular methodology, Total Quality Management & Business Excellence 23 (1), 2012, pp. 79–102. [19] D. Gefen, C.M. Ridings, IT Acceptance: Managing User-IT Group, 2003. [20] D. Gefen, D.W. Straub, M. Boudreau, Structural equation modeling and regression: guidelines for research practice, Communications of the Association for Information Systems 4 (7), 2000. [21] T. Gonzalez-Padron, B. Chabowski, G.T.M. Hult, D.J. Ketchen, Knowledge management and balanced scorecard outcomes: exploring the importance of interpretation, learning and internationality, British Journal of Management 21 (4), 2010, pp. 967–982. [22] M.G. Harvey, F.R. Lusch, Balancing the intellectual capital books: intangible liabilities, European Management Journal 17 (1), 1999, pp. 85–92. [23] J. Hoffecker, C. Goldenberg, Using the balanced scorecard to develop company wide performance measures, Journal of Cost Management fall, 1994, pp. 5–17. [24] C. Homburg, M. Artz, J. Wieseke, Marketing performance measurement systems: does comprehensiveness really improve performance? Journal of Marketing 76 (3), 2012, pp. 56–77. [25] H.-C. Huang, Designing a knowledge-based system for strategic planning: A balanced scorecard perspective, Expert Systems with Applications 36 (January (1)), 2009, pp. 209–218. [26] C.D. Ittner, D.F. Larcker, Are non-financial measures leading indicators of financial performance? An analysis of customer satisfaction Journal of Accounting Research Supplement 36, 1998, pp. 1–35. [27] B.-K. Jeong, A.C. Stylianou, Market reaction to application service provider (ASP) adoption: an empirical investigation, Information & Management 47 (3), 2010, pp. 176–187. [28] S.D. Johnson, Identification and selection of environmental performance indicators: application of the balanced scorecard approach, Corporate Environmental Strategy 5 (4), 1998, pp. 35–41. [29] R.S. Kaplan, R.D. Norton, Using the balanced scorecard as a strategic management system, Harvard Business Review 1996, pp. 75–85. [30] R.S. Kaplan, D.P. Norton, The Strategy-Focused Organization: How Balanced Scorecard Company thrive in the New Business Environment, Harvard Business School Press, 2001. [31] T. Kern, J. Krijger, L. Willcocks, Exploring ASP as sourcing strategy: theoretical perspecitves, proposions for practice, Journal of Strategic Information Systems 11, 2002, pp. 153–177. [32] G. Kim, E. Kim, An exploratory study of factors influencing ASP (application service provider) success, Journal of Computer Information Systems 48 (3), 2008, p. 118. [33] J.W. Kim, H.S. Han, Y.J. Choi, J.A. Woo, S.C. Park, Evaluation of SME’s ASP Adoption .Effects on Economic Performance, National Computerization Agency, Korea, 2006 [35] M. Kumru, A balanced scorecard-based composite measuring approach to assessing the performance of a media outlet, Service Industries Journal 32 (5), 2012, pp. 821–843. [36] H. Kunz, T. Schaaf, General and specific formalization approach for a balanced scorecard: an expert system with application in health care, Expert Systems with Applications 38 (March (3)), 2011, pp. 1947–1955. [37] C. Leam, H. Lee, Development of certification and audit processes of application service provider for IT outsourcing, Technovation 24 (1), 2004, pp. 63–71. [38] H. Lee, J. Kim, J. Kim, Determinants of success for application service provider: An empirical test in small businesses, International Journal of Human–Computer Studies 65 (9), 2007, pp. 796–815. [39] A.H.I. Lee, W.-C. Chen, C.-J. Chang, A fuzzy AHP and BSC for evaluating performance of IT department in the manufacturing industry in Taiwan, Expert Systems with Applications 34 (January (1)), 2008, pp. 96–107. [40] M.G. Lipe, S.E. Salterio, The balanced scorecard: judgmental effects of common and unique performance measures, Accounting Review 75 (3), 2000, pp. 283– 298. [41] Y. Malhotra, Organizational controls as enablers and constraints in successful knowledge management systems implementation, in: Y. Malhotra (Ed.), Knowledge Management and Business Model Innovation, Idea Group Publishing, Hershey, PA, 2001 , pp. 326–336. [42] M. Martinsons, R. Davison, D. Tse, The balanced scorecard: a foundation for the strategic management of information systems, Decision Support Systems 25 (1), 1999, pp. 25–36. [43] J. Mason, From e-learning to e-knowledge, in: M. Rao (Ed.), KM Tools and Techniques: Practitioners and Experts Evaluate KM Solutions, ButterworthHeinemann, Oxford, UK, 2004 , pp. 320–328. [44] A. Mayo, The role of employee development in the growth of intellectual capital, Personnel Review 29 (4), 2000, p. 2000. [45] C. Mendoza, R. Zrihen, Measuring Up, Financial Up, 2001 pp. 26–29. [46] A. Moreo, F.J. DeMicco, L. Xiong, Towards a model to measure the quality of environmental sustainability: the hospitality environmental scorecard, Journal of Quality Assurance in Hospitality & Tourism 10 (1), 2009, pp. 44–58. [47] S. Nielsen, E.H. Nielsen, Discussing feedback system thinking in relation to scenario evaluation in a balanced scorecard setup, Production Planning & Control 23 (6), 2012, pp. 436–451. [48] D. Northcott, T.M. Taulapapa, Using the balanced scorecard to manage performance in public sector organizations: issues and challenges, International Journal of Public Sector Management 25 (3), 2012, pp. 166–191.

S. Lee et al. / Information & Management 50 (2013) 553–561 [49] N.-G. Olve, J. Roy, M. Wetter, Performance Drivers: A Practical Guide to Using the Balanced Scorecard, Wiley, 1999. [50] R. Petty, J. Guthrie, Intellectual capital literature review: measurement, reporting, and management, Journal of Intellectual Capital 1 (July (2)), 2000, pp. 155–176. [51] J. Pfeffer, Changing mental models: HR’s most important task, Human Resource Management 44 (2), 2005, pp. 123–128. [52] G. Roos, J. Roos, Measuring your company’s intellectual performance, Long Range Planning 30 (3), 1997, pp. 413–426. [53] A. Schwarz, B. Jayatilaka, R. Hirschheim, T. Goles, A conjoint approach to understanding IT application services outsourcing, Journal of the Association for Information Systems 10 (10), 2009, pp. 748–781. [54] K.L. Sim, H.C. Koh, Balanced scorecard: a rising trend in strategic performance measurement, Measuring Business Excellence 5, 2001, pp. 18–27. [55] B. Smandek, A. Barthel, J. Winkler, P. Ulbig, Balanced score card implementation for IP rights management in a public research institution, Measuring Business Excellence 14 (4), 2010, pp. 65–75. [56] M. Sohn, T. You, S. Lee, H. Lee, Corporate strategies, environmental forces, and performance measures: a weighting decision support system using the k-nearest neighbor technique, Expert Systems with Applications 25 (3), 2003, pp. 279–292. [57] W.E. Stewart, Balanced scorecard for projects, Project Management Journal 32 (1), 2001, pp. 38–53. [58] A. Susarla, A. Barua, A.B. Whinston, Understanding the service component of application service provision: An empirical analysis of satisfaction with ASP services, MIS Quarterly 27 (1), 2003, pp. 91–123. [59] A. Susarla, A. Barua, A.B. Whinston, Transaction cost perspective of the ‘‘software as a service’’ business model, Journal of Management Information Systems 26 (2), 2009, pp. 205–240. [60] J. Taylor, C. Baines, Performance management in UK universities: implementing the balanced scorecard, Journal of Higher Education Policy & Management 34 (2), 2012, pp. 111–124. [61] R.D. Tobia, An Introduction to Least Squares Regression, SAS Institution Inc., 1999. [63] O. Velcu, Strategic alignment of ERP implementation stages: an empirical investigation, Information & Management 47 (April (3)), 2010, pp. 158–166. [64] C.E. Werts, R.L. Lin, K.G. Joreskog, Intraclass reliability estimates: testing structural assumptions, Educational & Psychological Measurement 34 (1), 1974, pp. 25–33. [65] N. Withanachchi, Y. Handa, K.K.W. Karandagoda, P.P. Pathirage, N.C.K. Tennakoon, D.S.P. Pullaperuma, TQM emphasizing 5-S principles/A breakthrough for chronic managerial constraints at public hospitals in developing countries, International Journal of Public Sector Management 20 (3), 2007, pp. 168–177. [66] H. Wold, Partial least squares, in: S. Kotz, N. Johnson (Eds.), Encyclopedia of Statistical Sciences, Wiley, New York, NY, 1985, pp. 58–591. [68] H.-Y. Wu, Constructing a strategy map for banking institutions with key performance indicators of the balanced scorecard, Evaluation & Program Planning 35 (3), 2012, pp. 303–320. [69] H.-Y. Wu, G.-H. Tzeng, Y.-H. Chen, A fuzzy MCDM approach for evaluating banking performance based on balanced scorecard, Expert Systems with Applications 36 (August (6)), 2009, pp. 10135–10147.

561

[70] Y. Yao, E. Watson, B. Kahn, Application service providers: market and adoption decisions, Communications of the ACM 53 (7), 2010, pp. 113–117. [71] R. Yawson, W. Amoa-Awua, A. Sutherland, D. Smith, S. Noamesi, Developing a performance measurement framework to enhance the impact orientation of the Food Research Institute, Ghana, R&D Management 36 (2), 2006, pp. 161–172. [72] M. Zrivan, N. Pliskin, R. Levin, Measuring user satisfaction and perceived usefulness in the ERP context, Journal of Computer Information Systems 45 (3), 2005, pp. 43–52. Sangjae Lee is a professor at the college of business administration in Sejong University. He received PhD in Management Information Systems from the Graduate School of management, KAIST. His research interests include decision support systems, electronic commerce, information systems controls, data mining, and IS planning. He has published in such journals as Decision Support Systems, Information & Management.

Sung Bum Park received a master degree in Management Information and a Ph. D. degree in Management Science from Korea Advanced Institute of Science and Technology. He has been working as a senior researcher for National Information Society Agency since then. His current research interests include performance evaluation of information system and digital content distribution.

Gyoo Gun Lim is a professor of MIS at School of Business, Hanyang University, Seoul, Korea. He received PhD in Management Engineering from Korea Advanced Institute of Science and Technology (KAIST) of Korea in 2001. His current research interests include Innovative business model, e-business, IT service, and intelligent systems.