Researching the investment of information technology in construction: An examination of evaluation practices

Automation in Construction 14 (2005) 569 – 582 www.elsevier.com/locate/autcon

Researching the investment of information technology in construction: An examination of evaluation practices Peter E.D. Lovea,*, Zahir Iranib, David J. Edwardsc a e-Business Research Centre, School of Management Information Systems, Edith Cowan University, Joondalup, WA 6027, Australia Information Systems Evaluation and Integration Network Group, Department of Information Systems and Computing, Brunel University, Uxbridge Middlesex, UB8 3PH, UK c Off-highway Plant and Equipment Research Centre (OPERC), Department of Civil and Building Engineering, Loughborough University, Loughborough, Leicestershire, LE11 3TU, UK b

Accepted 17 December 2004

Abstract The inherent worth and value of information technology (IT) employed within a business setting have been meticulously defined and subsequently examined throughout the general business community. However, within the construction industry issues surrounding the IT investment process remain largely unexplored as exhibited by recent survey work in Australia, which concludes that many construction organizations have not measured the impact of their IT investments. To address this aforementioned deficiency, a questionnaire survey was undertaken to examine the approaches used by construction organizations to evaluate and justify their IT investments. The analysis of 126 responses revealed the following key findings: different types of organization significantly ( pb0.05) differ in the amount they investment in IT; investment levels in IT were not influenced by organizational size; and the scope of purpose of ex-ante IT evaluation was considered broader than a financial control mechanism. Instead, the organizations sampled used ex-post evaluation as an opportunity for learning and thus regenerated knowledge. Based on these findings a pragmatic ex-ante IT evaluation framework is proposed which can be used by construction organizations to ameliorate their investment decision-making process. D 2005 Published by Elsevier B.V. Keywords: Appraisal techniques; Evaluation; Information technology; Investment

1. Introduction The evaluation of information technology (IT) has received considerable attention in the informa* Corresponding author. E-mail address: [email protected] (P.E.D. Love). 0926-5805/$ - see front matter D 2005 Published by Elsevier B.V. doi:10.1016/j.autcon.2004.12.005

tion systems (IS) and business management literature [1] and it is widely recognised that evaluation is critical for the successful adoption and implementation of IT (e.g., [2]). Many construction organizations, however, have reported that they are unaware of a generic methodology that can be used to justify their IT investments [3]. Indeed,

570

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

this critical activity has been generally neglected by the construction community, particularly the dInternational Council for Building Research Studies and Documentation (CIB) W-78 for Information Technology in ConstructionT. One reason proffered for this is that disparate IT evaluation activities create a myriad of alternative devaluationT perceptions between respective groups of stakeholders due to differences in goals, motivations and contexts [4]. In addition, the emergences of e-business, enterprise wide applications and mobile commerce technologies have added to the complexity associated with their evaluation, and as a result, have made it difficult for managers to articulate their benefits and prove an immediate return on investment. Despite such difficulties, the need to undertake evaluations of IT exists (i.e. ex-ante, ex-post, formative and summative) so that benefits management process and the technology’s effectiveness can be translated into favourable outcomes for the organization and its respective stakeholders. Moreover, such evaluations provide a control loop of invaluable feedback which in-turn allows the performance of technology employed to be readily judged over its life-cycle. There is an implicit assumption amongst some construction management researchers that the adoption of IT is a panacea to the problems of sustaining a construction organization’s competitive advantage as well as the industry’s overall performance. This presupposition is erroneous [5] because the evaluation of an IT project’s lifecycle is necessary to determine whether the technology is an investment sinkhole (specifically, should the project be concluded to minimize any further additional capital outlay) or an appropriate investment for the organization [6–9]. As IT applications are becoming an obligatory resource required to sustain the well-being of construction organizations, their justification and evaluation require serious consideration. In this paper, IT investment evaluation practices adopted within Australian construction organizations are examined. Findings emanating from the research work are then used to develop a framework to assist construction organizations (i.e., architects, quantity surveyors, contractors, etc.) with the process of IT evaluation.

2. Defining information technology evaluation Definitions of IT evaluation abound within the literature [10]. For example, Remenyi et al. [11] focused on the determination of worth and value per se and thus offer the following definition: ba series of activities incorporating understanding, measurement and assessment. It is either a conscious or tacit process which aims to establish the value of or the contribution made by a particular situation. It can also relate to the determination of the worth of an objectQ (p. 46). Farbey et al. [12], however, provided a broader definition of IT evaluation as: ba process that takes place at different points in time or continuously, for searching for and making explicit, quantitatively or qualitatively, all impacts of an IT projectQ (p. 205). Other definitions of evaluation take a far narrower perspective and emphasis is placed upon the activities of improved learning and understanding and their role supporting addressing organizational learning. For example, Forss et al. [13] and Psonios [14] suggest that evaluation is, by definition, synonymous with feedback and that feedback is the link between performance and knowledge structures. Symons and Walsham [15] consider evaluation as a process to be understood and argue that such an understanding should include an awareness of: why it is undertaken, the nature of the process which is undertaken and also its inherent complexity (i.e. limitations and problems). In a similar vein, Serafeimidis and Psonios [14], Farbey et al. [12] and Irani and Love [16] do not consider the process of IT evaluation to be solely rational, but interwoven with an array of social activities that are influenced by the human bias of behaviour. Before the commencement of any IT evaluation exercise well defined goals must be established [17], which are parallel to the organization’s business strategy [8]. This would involve articulating the strategic, tactical and operational expectations of the technology to be considered, to the organization’s immediate stakeholders.

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

3. Investment in IT Information technology investments differ in nature from other capital investments as there is a substantial human and organizational interface [9]. Such investments can be high risk as they often manifest themselves as erratic timing of cash flows and significant intangible costs [18]. Despite these potential financial problems IT capital investments are frequently evaluated using traditional appraisal techniques as adopted within the property sector. These traditional methods are suitable for capital investments where the fixed asset has a long term resale value that often appreciates with inflation. For IT investments however, the converse is true since the exponential rate of technological development renders IT resources obsolete in a far shorter duration [19]. There are a myriad of reasons why organizations should appraise their IT investments, these include [16]: ! comparison between different IT projects so as to determine those that are going to add value to the organization’s strategic positioning; ! ranking of projects in terms of organizational priorities; ! justify investment requests by management; ! control expenditure, benefits, risk, development and implementation of projects; and ! provide a framework that facilitates organizational learning. Primrose [19] noted that many managers negatively viewed project appraisal as a financial hurdle vis-a`-vis a useful technique for evaluating a project’s worth.

571

Information technology project investments often involve multiple stakeholders and increasingly impact upon the organization’s supply chain. According to Milis Mercken [18], there are typically five core parties involved with the IT investment process; each having their own set of objectives and expectations (Table 1). For financial based evaluations using traditional appraisal techniques, the process only serves the objective of management and thus neglects all other parties’ objectives (such as project team members and users). In turn, critical factors that might affect the willingness of these parties to cooperate in realising the objectives of the investment are not incorporated into the investment decision-making process. Table 1 summarizes the different parties (stakeholders) and their expectations from the IT investment. Table 1 also indicates that any form of evaluation must be multi-faceted and driven by the intention to include the individual stakeholder agendas. Choosing an evaluation approach that reaches beyond the traditional boundaries of financial evaluation (for example, direct cost analysis and cash flow projections) is increasingly important, and many factors associated with developing a robust IS requires a business, user and technology context. The fact that financial transparency is paramount so as to ensure a suitable return on investment, other strategically softer, political and social factors need to be considered during the evaluation process. A comprehensive list of investment appraisal approaches can be found in Renkema and Berghout [20], Irani and Love [16], and Mills and Mercken [18]. Several predominant generic ex-ante evaluation methods (and techniques within each method) used to

Table 1 IT investment stakeholders [18] Parties involved in IT investments

Objectives and expectations

Organization (management)

Interested in the gains (financial/and other) generated by the investment Seeks to ensure that the project is implemented on time, within budget and to user requirements Technology should meet their requirements while integrating flexibility to adapt to the changing requirements of users/customers Focus on short-term criteria set by sponsors (used to judge their performance) Focus on short-term criteria Consists of many groups, each with its own goals and objectives Might support or oppose the investment—possible covert resistance

Users Project team (implementers) Supporters (sub-contractors) Stakeholders (do not benefit from or influence the investment)

572

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

justify capital investments in IT are identified in Table 2; these methods are traditional evaluation, adjusted traditional evaluation, new evaluation techniques and mixed evaluation methods. A further perspective is a move away from generic appraisal techniques to sympathetic techniques where IT evaluations are based on bespoke project characteristics. For example, Hochstrasser [21] classified IT evaluations based on their objectives: ! Infrastructure projects: Hardware or software systems installed to enable the subsequent development of front–end systems; ! Cost replacement projects: IT systems introduced to automate manual activities; ! Economy of scale projects: Systems introduced to allow a company to handle an increased volume of data; ! Economy of scope projects: IT systems introduced to allow a company to perform an extended range of tasks; ! Customer support projects: IT systems introduced to offer better services to customers; ! Quality support projects: IT introduced to increase the quality of the finished product; ! Information sharing and manipulation projects: IT systems introduced to offer better information sharing and information manipulation; and ! New technology projects: IT systems introduced to exploit strategically the business potential of the new technology, to do things that were not possible before.

Table 2 Milis and Mercken [18] ex-ante evaluation techniques Appraisal method

Techniques

Traditional evaluation methods

! ! ! ! ! ! ! ! ! ! ! !

Adjusted traditional evaluation methods New evaluation techniques

Mixed evaluation methods

Payback period Return on investment (ROI) Internal rate of return (IRR) Net present value (NPV) Adjusted cost/benefit analysis Discount rate sensitivity Adjusted interpretation process Strategic fit Information economics The options model Multi-layer evaluation process Balanced scorecard

Hochstrasser [21] suggests that a particular appraisal technique should be matched to one of the above project characteristics. For example, if the project were a cost replacement project (where costs are direct in nature and perceived benefits largely efficiency gains), then the investment could be evaluated using traditional economic appraisal approaches such as, ratio based or discounted approaches. Alternatively, for new technology project investment which include a strategic dimension and substantial intangible benefits and indirect costs, then approaches like the balanced score card might well be more appropriate (e.g., [22,23]).

4. Research methodology A thorough literature review revealed a notable dearth of construction industry IT evaluation studies [1,3,23–30]. To address this deficiency, the IT evaluation practices of various organization types (such as, architects, consulting engineers, consulting project managers, quantity surveyors (QS) and contractors) were examined using a questionnaire survey. The fundamental purpose of the questionnaire was to determine the IT investment practices that are implemented by organizations in the construction industry. The research methodology used in paper has also been reported in [3]. 4.1. Questionnaire survey The population of participants was extracted from the telephone directory dYellow PagesT and a stratified random selected. A pilot study was first undertaken to measure the potential response rate and inherent suitability of the questionnaire developed using a sub-sample of 25 selected organizations. Various professional occupations were represented including architects, consulting engineers, consulting project managers, contractors and QSs. Speculative telephone interviews were conducted to obtain each organization’s commitment to the research work prior to posting a package of information which contained the questionnaire and stamp addressed dreturnT envelope. Such an approach ensured that a 100% response rate was obtained. Importantly, respondents were invited to actively

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

participate in the work by providing constructive comments on the design and structure of the survey. At the end of this consultation process, all comments received were positive and therefore the decision was taken to post an unadulterated copy of the original questionnaire to the main sample participants. Having successfully concluded the pilot work, the main survey work was then commenced and consisted of 50 questionnaires mailed to each of the five aforementioned organization types throughout Australia. Of the 250 questionnaires posted, 101 valid responses were received from the main survey, which when added to the 25 dunchangedT pilot questionnaires gave a total of 126 valid responses; representing a total consolidated response rate of 45%.

5. Findings 5.1. Sample characteristics Figs. 1 and 2 provide a breakdown of the valid respondent responses by organization type and geographical state. The sample is considered to be a reasonable representation of the Australian construction industry’s IT investment practices, considering its low up-take of IT. Figs. 3 and 4 provide details

573

about the sample’s distribution in terms of the organization’s number of employees and the turnover. Of the 126 organizations, 75% employed less than 30 employees and 79% had a turnover less than $A10 million. Thus, most of the sample was comprised of small and micro-organizations. Tables 3 and 4 provide a summary of organizational investment in IT as a percentage of company turnover and that 93% of construction contractors invested less than 1% of their turnover on IT, whereas over 50% of the other organizations sampled invested between 1% and 5% of their turnover. Table 4 indicates that 90% of the organizations surveyed invested less than 5% of their turnover on IT, with 44% investing less than 1%. Only 10% of organizations sampled invested more than 5% of their turnover in IT and notably, most of these were architects and QSs. The ANOVA revealed that investments in IT did not significantly vary with firm size (turnover and number of employees) ( pb0.05). However, differences in IT investments were found between organization types, F(4, 126)=10.48, ( pb0.05). A Tukey’s honestly significant difference (HSD) post hoc test was undertaken but did not identify differences between organizations ( pb0.05). Thus, investments in IT have not increased despite the widespread use of e-business and ecommerce applications throughout the Australian economy.

Fig. 1. Respondents by organization type.

574

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Fig. 2. Respondents by state.

5.2. Evaluation approaches Table 5 provides a summary of the extent to which methods for ex-ante evaluation were used. Interestingly, only 40% of organizations who invested in IT used such methods identified in Table 5. Anecdotal evidence would suggest that this

finding is not due to a lack of knowledge of the available techniques; instead, it is apparent that IT does not form an integral part of their business strategy for competitive advantage. At a tactical and operational level, however, IT is being increasingly used throughout the construction industry The emergence of e-business applications,

100

Number of Employees

80

81

60

40

20 14

17 8

6

51-100

101-250

0 <10

11-30

31-50

Number Employed Fig. 3. Firm size by number of employees.

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

575

60 57

Number of Organizations

50

40

43

30

20

14 10 9 3

0 <$1M

$1-10M

$11-50M

$51-100M

$101-250M

Turnover ($A) Fig. 4. Turnover of organizations sampled.

uating IT investments in small-to-medium than larger organizations. Table 6 identifies the formative evaluation processes adopted within the sample. Over 50% prepare an IT benefits delivery plan prior to, and during, system design and implementation. There is considerable divergence in the use of formative evaluation processes within the sample. A Kruskal–Wallis test was undertaken to determine if any significant differences between the size of the organization, the organization type, and the evaluation processes employed exist. In the case of turnover, there were significant differences between turnover and all evaluation

however, for the procurement of materials [31] and sharing information between project participants [32] are beginning to be embraced by some construction organizations. The adoption and implementation of the technology requires significant capital outlay at the outset, and on a continual basis as technology develops. As a result, ex-ante evaluation will have to form an integral part of their IT management strategy if organizations are too remain competitive. From the findings presented, it would appear that the construction organizations sampled are less likely to adopt a formal ex-ante evaluation process. According to Ballantine et al. [33] traditional financial techniques are more appropriate for eval-

Table 3 Organization types in relation to IT investment as a percentage of turnover Organization type

IT investment as a % of turnover b1%

1–5%

6–10%

11–20%

Contractor Quantity surveyors Engineering consultants Project management consultants Architects Total

27 8 6 8 6 55

2 16 15 18 7 58

– 6 (20%) 2 (9%) 2 (7%) 2 (13%) 12 (9%)

– 1 (3%) – – – 1 (1%)

(93%) (26%) (26%) (29%) (40%) (44%)

(7%) (52%) (65%) (65%) (47%) (46%)

Total

29 31 23 28 15 126

(100%) (100%) (100%) (100%) (100%) (100%)

576

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Table 4 Number of employees in relation to IT investment as a percentage of turnover Number IT investment as a % turnover employed b1% 1–5% 6–10% b10 11–30 31–50 51–100 101–250 Total

39 7 3 2 4 55

Total

Significant differences between turnover and the justification inhibitors were identified for the following variables:

11–20%

(48%) 34 (42%) 7 (9%) 1 (1%) (50%) 6 (43%) 1 (7%) – (18%) 10 (595) 4 (24%) – (25%) 6 (75%) – – (67%) 2 (33%) – – (44%) 58 (46%) 12 (9%) 1 (1%)

81 14 17 8 6 126

! dlimited managerial and technological knowledgeT (v 2=20.77, pb0.00); ! dlack of strategic visionT (v 2=29.49, pb0.00); and ! dreluctance of employees to adapt to new technologyT (v 2=18.15, pb0.01).

(100%) (100%) (100%) (100%) (100%) (100%)

processes employed with the exception of the duse of IT to develop future processesT (v 2=7.06, pb0.13). This implies that when construction organizations do implement IT, they aim to utilize its value adding potential. There were also significant differences between the number of people employed and all of the evaluation process adopted ( pb0.05). However, no significant differences between organization types where evaluation processes adopted were identified ( pb0.5). The size of the organization (in terms of employees employed) therefore influences the extent of evaluation processes implemented. Ballantine et al. [33] proposed that small-to-medium sized firms were more likely to focus on control of their investment rather than learning from its implementation. However, it would appear that predominantly, the evaluation process is used by construction organizations as both a control and learning mechanism, even though it may not form an integral part of their business strategy. Factors inhibiting the evaluation process are identified in Table 7. A significant proportion of the organizations sampled indicated that some difficulty was encountered when determining their IT investment, particularly the need to demonstrate quick financial returns.

There was also significant difference between the number of people employed and justification inhibitors for the following variables: ! dinability to select an appropriate IT appraisal techniqueT (v 2=11.06, pb0.02) ! dlack of strategic visionT (v 2=14.79, pb0.05); and ! dan ability to account for the full business benefitsT (v 2=10.84, pb0.02). Several differences with respect to the organizational size and organization types were identified: Those identified were: ! dlimited managerial and technological knowledgeT (v 2=18.99, pb0.01); ! dlack of strategic visionT (v 2=11.69, pb0.02); ! dunable to identify financial benefitsT (v 2=11.69, pb0.01); ! dan ability to account for the full business benefitsT (v 2=17.83, pb0.01); ! dreluctance of employees to adapt to new technologyT (v 2=22.15, pb0.00); and ! dinability to select an appropriate IT appraisal techniqueT (v 2=11.63, pb0.02). Considering the evidence provided, a lack of strategic vision is a key factor inhibiting the justifi-

Table 5 Ex-ante methods used for evaluating IT investments Evaluation method

Mean (n=126)

S.D.

Not at all

Some extent

Moderate extent

A large extent

Very large extent

Return on investment Discounted cash flow and IRR Net Present Value (NPV) Profitability index Payback period Present worth

2.11 1.72 1.69 1.78 2.19 1.98

1.17 1.03 .09 1.08 1.23 1.08

56 73 74 73 55 57

20 27 25 23 18 28

32 18 18 15 30 30

15 4 9 14 19 8

3 4 _ 1 4 3

(44%) (58%) (59%) (58%) (44%) (45%)

(16%) (21%) (20%) (18%) (14%) (22%)

(25%) (14%) (14%) (12%) (24%) (24%)

(12%) (3%) (7%) (11%) (15%) (6%)

(2%) (3%) (1%) (3%) (2%)

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

577

Table 6 Evaluation processes implemented Evaluation process

Mean S.D. Not at all Some (n=126) extent

Moderate A large extent extent

Prepare a benefits delivery plan Prepare a benefits delivery plan during system design Prepare a benefits delivery plan during the implementation of IT Prepare a benefits delivery plan once the technology is implemented Plan organizational changes associated with the implementation of IT before approval Plan organizational changes associated with the implementation of IT during system design Plan organizational changes associated with the implementation of IT during implementation Plan organizational changes associated with the implementation of IT once the technology is implemented Conduct reviews during the implementation of IT Conduct post implementation reviews Use IT to develop future processes

2.40 2.00 1.92 1.81

1.51 1.20 1.17 1.06

26 19 19 16

2.47

1.26 37 (29%) 32 (25%) 25 (24%) 24 (19%) 8 (6%)

2.23

1.18 47 (37%) 25 (20%) 37 (30%) 11 (9%)

2.45

1.28 42 (33%) 23 (18%) 31 (25%) 22 (18%) 8 (6%)

2.61

1.28 37 (29%) 18 (14%) 34 (27%) 30 (24%) 7 (6%)

2.50 2.60 2.80

1.23 38 (30%) 25 (20%) 28 (22%) 32 (25%) 3 (2%) 1.15 30 (24%) 26 (21%) 37 (30%) 30 (24%) 3 (2%) 1.23 29 (23%) 16 (13%) 40 (32%) 33 (26%) 8 (6%)

cation process for organizations. As construction organizations now need to embrace IT to gain a competitive advantage, it is expected that they will begin to evaluate their investments in a more systematic and structured manner. 5.3. Motivations for IT adoption Table 8 identifies the motivations for organizations adopting IT. It would appear that organizational drivers for IT implementation are anticipated improvements to productivity (cost efficiency) and business processes. To gain a competitive advantage, the improvement of service quality and firm profitability were also identified as primary motivations for IT

49 61 63 67

(39%) (48%) (50%) (53%)

21 27 30 30

(17%) (22%) (24%) (24%)

Very large extent

(21%) 22 (18%) 8 (6%) (15%) 14 (11%) 5 (4%) (15%) 7 (6%) 7 (6%) (13%) 11 (9%) 2 (2%)

6 (5%)

adoption. Unexpectedly, over 70% of the organizations suggested that a motivation for adopting IT was to support the strategic direction of the organization. Differences were found between the number of people employed and motivation factors, with the exception of dto gain a competitive advantageT ( pb0.05). With respect to turnover, the only significant differences were with dsupport the strategic direction of the organizationT (v 2=18.07, pb0.01) and dimprove service qualityT (v 2=17.75, pb0.01). Noteworthy, the only significant difference between organization types for motivation factors was dto improve service qualityT (v 2=13.83, pb0.00). Information technology applications can be used for an array of services provided by construction

Table 7 Justification inhibitors Justification inhibitors

Mean (n=126)

S.D.

Not at all

Some extent

Moderate extent

A large extent

Very large extent

Limited managerial and technological knowledge Lack of strategic vision Unable to identify financial benefits Limited organizational resources and resistance to technology related change The need to show quick financial returns with minimal risk A multiplicity of justification and implementation paths An ability to account for the full business benefits Unable to identify and manage the scope of IT/IS related costs Reluctance of employees to adapt to new technology Inability to select an appropriate IT appraisal technique

2.19 2.07 2.28 2.23

2.19 2.07 2.28 2.23

43 52 45 44

(34%) (41%) (36%) (35%)

37 33 25 34

(29%) (26%) (20%) (27%)

28 27 36 25

(22%) (21%) (29%) (20%)

15 8 15 21

(12%) (6%) (12%) (17%)

3 6 5 2

(2%) (5%) (4%) (2%)

2.52 2.34 2.36 1.93 1.69 2.27

2.52 2.34 2.36 1.93 1.69 2.27

27 34 36 53 69 52

(21%) (27%) (29%) (42%) (55%) (41%)

40 37 35 42 39 33

(32%) (29%) (28%) (33%) (31%) (26%)

35 35 34 20 9 27

(28%) (27%) (27%) (16%) (7%) (21%)

14 18 15 8 6 8

(11%) (14%) (12%) (6%) (5%) (6%)

10 2 6 3 3 6

(8%) (2%) (5%) (2%) (2%) (5%)

578

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Table 8 Motivation for adopting IT Motivation for adopting IT

Mean (n=126)

S.D.

Not at all

Improve productivity (i.e. cost efficiency) of business processes Improve performance of business (effectiveness) processes Seemed like a good idea at the time To gain a competitive advantage Improve profitability Pressure from rivals who are implementing IT Support the strategic direction of the organization Improve service quality Improve market share

4.02

1.03

6 (5%)

4.13

0.85

1 (1%)

2.00 3.73 3.66 2.32 3.27 3.92 3.1

1.11 0.98 1.01 1.10 1.17 1.01 1.29

organizations. For example, at an operational level, Computer Aided Design (CAD) often improves the quality of contract documentation, especially when design professionals integrate and coordinate their outputs. This appears to be a relatively straightforward process, but cultural and behavioural barriers, juxtaposed with problems associated with interoperability have hindered the production of effective contract documentation. In turn, an adverse affect on the service quality of organizations is experienced [34].

6. An IT evaluation ex-ante framework The evaluation methods that are used by the construction organizations sampled neglect to address the complexity associated with IT decision-making process. In addition, they do very little in the way of providing management and stakeholders a true reflection of the investments total costs because only tangible costs are taken into account, which only account for a fraction of total costs experienced in an investment’s lifecycle (e.g. [35]). Discounted cash flow techniques such as net present value (NPV) and internal rate of return (IRR) are used to: ! essentially estimate and evaluate cash flows; ! recognise and discount for the time value of money; ! establish whether incremental benefits exceed incremental costs; and ! provide a figure that indicates project viability.

56 1 2 32 9 17 15

(44%) (1%) (2%) (25%) (7%) (14%) (12%)

Some extent

Moderate extent

A large extent

Very large extent

4 (3%)

17 (14.0%)

53 (42%)

46 (37%)

6 (5%)

14 (11%)

59 (47%)

46 (37%)

26 31 42 26 29 48 30

9 49 37 16 42 43 32

4 30 32 5 19 _ 24

31 15 13 47 27 18 25

(25%) (12%) (10%) (37%) (21%) (14%) (20%)

(21%) (25%) (33%) (20%) (23%) (38%) (24%)

(7%) (39%) (29%) (13%) (33%) (34%) (25%)

(3%) (24%) (25%) (4%) (15%) (19%)

While a plethora of methods for justifying IT investments have been developed [16], no single technique can cope with the wide range of circumstances in which IT may be required [2,35]. Rather than using a technique that focuses on tangible benefits and costs it is important that an evaluation method also takes into account both the intangible benefits and costs associated with IT adoption. Rather than deviating from current practice, and developing a new technique that requires a significant change to organizational mindset, a pragmatic framework that can readily used by construction organization is proposed (Fig. 5). Baldwin et al. [28] have proposed a similar framework but with the significant difference that the quantification of indirect benefits and costs is not taken into account. Furthermore, potential benefits are examined from an efficiency, effectiveness and performance perspective. These are relatively meaningless taxonomies because they are in fact interrelated and difficult to operationalize in a process. Thus, it is suggested that business process such as those proposed by Baldwin et al. [28] (see Fig. 5) be analysed in terms of the technologies contribution to an organization’s strategic, tactical and operational levels of a process. This can enable an organization to isolate specific IT contributions in terms of benefit and cost, specifically those of an intangible nature. In fact, the most significant intangible benefits/costs that occur in organizations are those at the strategic level [16]. Instead of the decision to invest in IT being taken by an accountant or management, it is suggested that key stakeholders be actively involved in the justifica-

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Stakeholders

579

Knowledge

Determine business benefits Step 1

1. Tangible

2. Intangible Benefit dimensions Strategic Tactical Operational • • • • • • •

Business planning Marketing Information management Procurement Design management Construction production Human resources

Determine cost of technologies Step 2

Step 3

1. Tangible

2. Intangible

• • • • • •

• • • • • • •

Hardware/software System design and programming Networking and telecommunication Education and training Maintenance supplies and services External consultant services

Process/role re-design Time for training Resistance Loss of productivity Displacement and disruption Salary changes Staff turnover

Conduct financial appraisal and risk assessment • • • •

Return on investment (ROI) Net present value (NPV) Internal rate of return (IRR) Risk analysis and assessment

Feedback on investment viability

, , As noted a real options or balanced scored approach can be , adopted though this would depend on the IT project s characteristics ,

Fig. 5. An IT investment framework for construction organizations.

tion process, especially since intangible benefits of an IT investment are difficult to determine. The expected value of benefits can be determined using probabilities for different scenarios and computing expected values for each of the processes identified, which in turn can be used to establish the probability of savings to be made [2,28]. Alternatively, the opportunity cost could be determined by examining the amount of time that

can be saved from implementing the technology and/or application. For example, during the construction production process a contractor could use mobile applications to conduct real-time health and safety reviews and thus reduce the time to rectify hazards and eliminate the possibility of accidents. The hypothetical intangible monetary benefits that could materialise from utilising such technology are identified in Table 9.

580

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Table 9 Example of expected value of intangible benefits Possible reduction in administrative time over a month (h)

Probability of occurrence

Mid-point of anticipated increase

Incremental savings (Col 3$200 average saving per hour)

Probability of savings (Col 24)

0 1–5 6–10 11–20 21–30 31–40 41–50

0 3 5 10 5 4 3

– 2.5 7.5 15.5 25.5 35.5 45.5

0 $500 $1 500 $3 100 $5 100 $7 100 $9 100

0 $1 500 $7 500 $31 000 $25 500 $28 400 $27 300 $121 200

Note: This is a hypothetical example and the figures reported are only used to demonstrate the determination of an intangible benefit. Also, note the savings associated with the site not being closed/or no access to a work area have not been included here. Only the time of the persons responsible for conducting safety audits, etc. is considered.

If a contractor was to implement an enterprise wide application such as enterprise resource planning (ERP), then benefits would need to be determined for all the business functions and process levels identified in Table 9. This represents a time-consuming but worthy process considering the significant capital outlay that would be required to implement this type of application. The determination of direct costs is a relatively straightforward process. The taxonomy of IT costs proposed in Irani et al. [36] can be used as a reference point for identifying those costs that are likely to be incurred during an IT project’s lifecycle. Once the NPV, IRR and payback are determined for the direct and indirect costs that have been identified, sensitivity analysis should be undertaken to take account for any errors that could be contained in the forecasts. As a result, the factors that affect the cost adopting IT can be identified. The risks associated with the adoption of IT and its mitigation should be identified. Factors that might lead to the failure of an IT project include, lack of alignment to organizational strategy, lack of organizational adaptation to complement technological change, too much faith in a technical fix and poor management of change [8].

7. Conclusion Evidence provided in this paper suggests that the evaluation of IT is an area that Australian construc-

tion organizations have neglected, even though significant a proportion of those sampled suggested that they had implemented IT for strategic reasons. Discounted cash flow methods were the primary methods being used by the sampled construction organizations to evaluate their IT investments. Yet, the research revealed that a considerable proportion of the sample did not undertake any form of financial evaluation of their IT investments. Based on the findings presented a pragmatic framework that can be used to guide construction organizations through the ex-ante evaluation process for an IT investment was proposed. The framework takes into account the importance of stakeholders in the evaluation exercise and proposes that the initial business benefits to be expected from implementing IT are determined from a strategic, tactical and operational perspective. Once this has been undertaken tangible and intangible costs should be identified and then financial appraisal and risk assessments undertaken. Implementation of the framework does not require any significant change to business practices, though it does require organizations to provide greater attention to the justification process by placing emphasis on tangible and intangible benefits/costs of their potential investment. The problems associated with IT investment evaluation are complex, but it is hoped that the framework that has been propagated provides construction organizations with the foundation to better understand the implications of their investments in IT.

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

Acknowledgements The authors would like to acknowledge the financial support provided by the Australian Research Council (DP0344682) and Engineering and Physical Sciences Research Council (GR/R08025).

[15]

[16]

[17]

References [1] Z. Irani, P.E.D. Love, Information systems evaluation: past, present and future, European Journal of Information Systems 10 (4) (2001) 183 – 189. [2] A. Anandarajan, H.J. Wen, Evaluation of information technology investment, Management Decision 37 (4) (1999) 329 – 337. [3] P.E.D. Love, Z. Irani, D.J. Edwards, Industry-centric benchmarking of information technology benefits, costs, and risks for small to medium sized enterprises, Automation in Construction 13 (4) (2004) 507 – 524. [4] G. Walsham, Evaluation, in: G. Walsham (Ed.), Interpreting Information Systems in Organizations, John Wiley & Sons, Chichester, England, 1993. [5] H. Li, P.E.D. Love, Z. Irani, The relationship between the use of IT/IS and the productivity of consulting firms in construction, International Journal of Construction Information Technology 8 (1) (2000) 15 – 27. [6] J. Ballantine, R. Galliers, S. Stray, Information systems/ technology practice: evidence from UK organizations, Journal of Information Technology 11 (2) (1996) 129 – 142. [7] L. Willcocks, S. Lester, Information technology: transformer or sinkhole, in: L. Willcocks, S. Lester (Eds.), Beyond the Productivity Paradox, John Wiley & Sons, 1997, pp. 1 – 38. [8] L. Willcocks, V. Graeser, Delivering IT and e-business Value, Butterworth-Heinemann, Oxford, UK, 2001. [9] Z. Irani, M. Themistocleous, P.E.D. Love, The impact of enterprise application integration on information system lifecycles, Information and Management 41 (2) (2003) 177 – 187. [10] M. Ramage, Developing a methodology for the evaluation of cooperative systems, Proceedings of the 20th Information Systems Research Conference in Scandinavia, Department of Informatics, University of Oslo, Norway, 1997, pp. 769 – 789. [11] D. Remenyi, M. Sherwood-Smith, T. White, Achieving Maximum Value From Information Systems: A Process Approach, John Wiley & Sons, Chichester, UK, 1997. [12] B. Farbey, F. Land, D. Targett, IS evaluation: a process of bringing together benefits, costs and risks, in: W. Currie, R. Gallier (Eds.), Rethinking Management Information Systems, Oxford University Press, New York, 1999, pp. 204 – 228. [13] K. Forss, B. Cracknell, K. Samset, Can evaluation help an organization learn? Evaluation Practice 18 (3) (1994) 253 – 266. [14] V. Serafeimidis, A. Psoinos, Emerging issues and a holistic approach for information technology investments evaluation

[18]

[19] [20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

581

into the 1990s, Proceedings of the 5th Hellenic Conference on Informatics, 7th–9th December, Athens, Greece, 1995. V.J. Symons, G. Walsham, The evaluation of information systems: a critique, Journal of Applied Systems Analysis 15 (1988) 119 – 132. Z. Irani, P.E.D. Love, Developing a frame of reference for exante IT/IS investment evaluation, European Journal of Information Systems 11 (1) (2002) 74 – 82. A.W. Brown, K.C. Wallnau, Framework for evaluating software technology, IEEE Software 13 (5) (1996) 39 – 49. K. Milis, R. Mercken, The use of the balanced scorecard for the evaluation of information and communication technology projects, International Journal of Project Management 22 (2004) 87 – 97. P.L. Primrose, Investment in Manufacturing Technology, Chapman and Hall, UK, 1991. J.W. Renkema, E.W. Berghout, Methodologies for information systems investment evaluation at the proposal stage: a comparative review, Information and Software Technology 39 (1997) 1 – 13. B. Hochstrasser, Evaluating IT investments—matching techniques to projects, Journal of Intelligent Systems 5 (1990) 215 – 221. R.A. Stewart, S. Mohamed, IT/IS projects selection using multi-criteria utility theory, Logistics Information Management 15 (4) (2002) 254 – 270. R.A. Stewart, S. Mohamed, Evaluating the value IT adds to the process of project information management in construction, Automation in Construction 12 (4) (2003) 407 – 417. H. Li, Towards quantitatively measuring the performance of construction IT systems, Building Research and Information 24 (1) (1996) 379 – 382. CIRIA, IT in construction—quantifying the benefits. Construction Industry Research and Information Association, report 160, CIRIA, London, 1996. Construct IT, Measuring the benefits of IT innovation. Construct IT Centre for Excellence, University of Salford, Salford, UK, 1998. B. Atkin, A. Clark, J. Gravett, D. Smith, S. Walker, Benchmarking IT use in construction, in: M. Betts (Ed.), Strategic management of IT in construction, Blackwell Science, Oxford, UK, 1999, pp. 235 – 256. A. Baldwin, M. Betts, D. Blundell, K.L. Hansen, T. Thorpe, Measuring the benefits of IT innovation, in: M. Betts (Ed.), Strategic Management of IT in Construction, Blackwell Science, 1999, pp. 288 – 310. J. Andersen, A. Baldwin, B. Betts, C. Carter, A. Hamilton, E. Stokes, A. Thorpe, A framework for measuring IT innovation benefits, Electronic Journal of Information Technology in Construction 5 (2000) 57 – 72 (http://www.ITcon.org/). P.E.D. Love, Z. Irani, H. Li, R.Y.C. Tse, E.W.L. Cheng, An empirical analysis of IT/IS evaluation in construction, International Journal of Construction Information Technology 8 (2) (2000) 21 – 38. C.K. Kong, H. Li, P.E.D. Love, An e-commerce system for construction materials procurement, Construction Innovation 1 (1) (2001) 31 – 43.

582

P.E.D. Love et al. / Automation in Construction 14 (2005) 569–582

[32] T. Elliman, G. Orange, Electronic commerce to support construction design and supply chain management, International Journal of Physical Distribution and Logistics Management 30 (3/4) (2000) 345 – 360. [33] J. Ballantine, M. Levy, P. Powell, Firm size and the evaluation of information systems: issues and evidence, Proceedings of the European Conference on Information Systems, 2nd–4th July, Lisbon, Portugal, 1996, pp. 797 – 809.

[34] P.E.D. Love, Z. Irani, D.J. Edwards, A rework reduction model for construction projects, IEEE Transactions on Engineering Management 51 (4) (2004) 426 – 440. [35] B. Farbey, F. Land, D. Target, How to Assess Your IT Investment, Butterworth Heinemann, UK, 1993. [36] Z. Irani, A. Ghoneim, P.E.D. Love, in press. Cost taxonomies for information systems management. European Journal of Operations Research.