An empirical examination of CobiT as an internal control framework for information technology

International Journal of Accounting Information Systems 8 (2007) 240 – 263

An empirical examination of CobiT as an internal control framework for information technology Brad Tuttle ⁎, Scott D. Vandervelde 1 University of South Carolina, Moore School of Business, 1705 College Street, Columbia, SC 29208, USA Received 26 September 2006; received in revised form 25 September 2007; accepted 30 September 2007

Abstract One commonly used framework for developing and evaluating technology intensive information systems is CobiT. This framework was originally a benchmark of best control practices developed and maintained by the Information Technology Governance Institute, the umbrella organization to the Information Systems Audit and Control Association. We empirically examine the conceptual model that underlies the CobiT internal control framework as it applies to an audit setting (including operational, compliance, and financial audit settings). We find that superimposing CobiT's conceptual model onto audit relevant assessments made by a panel of highly experienced IT auditors confirms the internal consistency between the underlying constructs of CobiT. Furthermore, we find that CobiT's conceptual model predicts auditor behavior in the field related to their seeking help and giving help as evidenced by their postings to a general IT audit listserv. Given the results of this study, we propose future research aimed at developing a general theory of internal control applicable to information technology based on CobiT. © 2007 Elsevier Inc. All rights reserved. Keywords: Internal controls; IT controls; Internal control frameworks; CobiT

1. Introduction Organizations and their auditors use frameworks to guide their design and evaluation of internal controls. This use of internal control frameworks has dramatically increased in importance since the passage of the Sarbanes–Oxley Act of 2002 and since the release of the Public Company Accounting Oversight Board's (PCAOB) Auditing Standard No. 2 (AS2) in 2004. Presumably, the ⁎ Corresponding author. Tel.: +1 803 777 6639; fax: +1 803 777 0712. E-mail addresses: [email protected] (B. Tuttle), [email protected] (S.D. Vandervelde). 1 Tel.: +1 803 777 6075; fax: +1 803 777 0712. 1467-0895/$ - see front matter © 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.accinf.2007.09.001

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

241

use of a framework to guide the assessment of internal controls results in more comprehensive, reliable, and complete assessments. To achieve these goals in today's information technology (IT) intensive environment, a control framework must conceptualize the important aspects of internal control within an IT context in a complete and logically consistent manner. In the absence of a comprehensive and conceptually sound framework, the complexity of modern systems can overwhelm an auditor. This suggests that the quality of the internal control audit assessment depends on the conceptual model upon which a framework rests. This paper looks at Control Objectives for Information Related Technology (CobiT) by examining its conceptual consistency in an audit setting. Within IT intensive environments, CobiT is a widely recognized control framework that is emerging as the supplemental framework of choice to the Treadway Commission's Committee of Sponsoring Organizations (COSO) evaluation framework (IT Governance Institute, 2005; see also Colbert and Bowen, 1996; Netegrity, 2004; Ramos, 2004). Fedorowicz and Gelinas (1998) state that CobiT complements the COSO framework for assessing the internal controls and overall corporate governance of an organization. Likewise, Lainhart (2001, 19–20) states that CobiT is a tool that “helps enterprises balance IT risk and investment in controls.” These sentiments are echoed by Dennis Reynolds, KPMG partner and head of the Financial Services Risk Governance in London (KPMG, 2003 13), The Committee of Sponsoring Organizations of the Treadway Committee (COSO) evaluation framework is recommended by the Commission as an appropriate basis for management's assessments. Most international organizations are adopting the COSO framework for their evaluation, but are supplementing its control criteria with those recommended by the Control Objectives for Information and related Technology (CobiT)…2 CobiT was originally intended for use by an organization's management as a benchmarking tool consisting of the best practices related to IT controls. Since then and because of its strong control focus, both internal and external auditors have applied CobiT to financial statement audits as well as to operational and compliance audits. In regards to financial statement audits, AS2 mandates that management use a control framework in order to assess the effectiveness of internal controls over financial reporting. The use of the term “framework” in this paper is in the same vein as used in AS2. While CobiT is apparently useful for financial statement audit purposes, this study takes a broader view to include internal controls related to operational and compliance audits. Despite the importance of using a sound conceptual model, no practitioner developed internal control framework, that we are aware of, has undergone rigorous academic examination in the same manner that researchers routinely examine the conceptual models developed by other academics. The objective of the present study is to examine the internal consistency of CobiT's conceptual model within an audit setting by investigating whether auditor perceptions of audit risk related to complexity, client importance, client attention, and process risk combine to represent IT process risk in the manner asserted by CobiT. (See Appendix A for definitions and Section 3.1 for a discussion of these measures of risk.) The present study provides further

2 Perhaps one measure that CobiT is generally regarded as an appropriate supplement to COSO is found in the following entry to the Wikipedia dictionary (http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act): “The PCAOB suggests considering the COSO framework in management/auditor assessment of controls. Auditors have also looked to the IT Governance Institute's “CobiT": Control Objectives of Information and Related Technology for more appropriate standards of measure. This framework focuses on IT processes while keeping in mind the big picture of COSO's control activities and information and communication.” The authors did not submit this entry to the dictionary just so we could include it in our paper.

242

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

collaborating evidence as to whether the extent to which auditors seek and give IT audit assistance reflects the conceptual model underlying the CobiT framework. An examination of the consistency of CobiT's conceptual model is important for three reasons. First, users of CobiT as a framework in order to maintain effective IT control would benefit from knowing if the underlying conceptual model holds together under scrutiny. In this manner, internal audit functions can use CobiT with increased confidence as a framework for any type of IT audit they perform, whether it is an operational audit, compliance audit, or financial audit. Second, based on discussions with practicing auditors, the major public accounting firms use either CobiT, or something very similar, when working on clients with significant IT controls. Policy-makers would benefit from evidence that either supports or calls into question the conceptual foundations of current audit practice. Having this data would greatly aid policymakers who must set auditing standards related to internal controls for publicly traded companies. Third, it is possible that examining CobiT's conceptual model is a first step toward the development of a more general theory of internal control. Although no academic theory of internal control exists, the profession is essentially proposing CobiT as a process oriented theory of internal control based on IT processes, IT domains, information criteria and the IT resources employed to generate information. Further development of a formal theory of internal control, especially as it relates to IT, should lead to more effective compliance and operational audits. This study uses audit related assessments of CobiT constructs provided by a panel of experts together with postings from the Information Systems Audit and Control Association's (ISACA) general listserv to provide empirical support for CobiT as an IT internal control framework to support the audit function (including operational, compliance, and financial audits). First, a panel of 12 highly experienced IT auditors evaluated key aspects of CobiT's conceptual model using the following measures of audit risk with regards to IT: complexity, client importance, client attention, and process risk. Analysis shows that responses from the expert panel combine in a manner that is generally consistent with CobiT's underlying conceptual model. We discuss departures in our findings from the CobiT model in the final section of the paper. Second, the 12 IT auditors along with 17 non-IT auditors evaluated each of the CobiT processes based on the level of risk to a typical organization. These ratings correlate with data obtained from the ISACA listserv. The analysis suggests that the conceptual model upon which the CobiT framework is based is associated with (1) the number of threads or topics discussed on the ISACA listserv, and (2) the total number of messages posted. To further support the CobiT conceptual model, all relevant listserv messages were successfully coded to specific CobiT control objectives based on their content. That is, CobiT appears to be sufficiently comprehensive to encompass the audit specific questions posted in our sample from the listserv. The remainder of the paper proceeds as follows. Section 2 provides the background, theoretical framework, and research propositions. Section 3 provides the first phase of the CobiT validation based on experienced auditors. Section 4 provides the second phase of the CobiT validation using archival information. Finally, Section 5 provides a discussion of our results. 2. Background and theory The well-established COSO framework relies on the idea that the achievement of the following objectives is important for strong internal controls: effectiveness and efficiency of operations, reliability of financial reporting, and compliance with applicable laws and regulations (COSO, 2004). Its underlying conceptual model suggests that internal control objectives are achieved by paying attention to five components of control: (1) the control environment, (2) risk assessment,

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

243

(3) control activities, (4) information and communication, and (5) monitoring. By conceptualizing internal controls in this manner, control frameworks are meant to accomplish three objectives: (1) ensure completeness in coverage, (2) aid in identifying high risk areas, and (3) help to accurately assess the impact of controls (COSO, 1992). As a practical matter, however, COSO is a highly abstract conceptual framework and does not identify control objectives at a level of specificity sufficient to design detailed audit tests. Furthermore, the general nature of COSO does not address the complexity and special risks inherent in IT (Colbert and Bowen, 1996). Given the reliance on technology within most organizations, organizations need a framework to address technology to be functional in today's audit environment. Furthermore, because COSO expresses its components at a very high level of abstraction, it may not be possible to design an empirical test of its internal conceptual consistency. For these reasons, organizations and auditors in computerized environments are adopting specialized frameworks, such as CobiT, to supplement COSO. Every major international accounting firm has adopted CobiT or at a minimum its major constructs in connection with their review of internal control. This trend extends beyond the U.S. as evidenced by the European Union's recent adoption of CobiT as an Auditing Standard (Summerfield, 2005). Unlike COSO's five components, which are structured by semantic category, the CobiT framework relies on a process model that is organized around a system life cycle approach containing four primary domains (see Fig. 1). These domains are labeled: Plan and Organise; Acquire and Implement; Deliver and Support; and Monitor and Evaluate. Within each domain there are specific processes that an organization should address to achieve detailed and specific IT related control objectives. For instance, within the Deliver and Support domain is the process, “DS4 Ensure Continuous Service.” This process is associated with 10 detailed control objectives (not listed in the figure) that IT best practices suggest should be met in order to achieve a high level of control. An example control objective from this process is objective DS4.6, “IT Continuity Plan Training” which states, “Ensure that all concerned parties receive regular training sessions regarding the procedures and their roles and responsibilities in case of an incident or disaster. Verify and enhance training according to the results of the contingency tests” (IT Governance Institute, 2005, 116). These detailed control objectives are further supplemented by audit guidelines for each CobiT process. It is important to note that the control objectives in CobiT are specific enough to be easily implementable; yet general enough to be applicable to various types of audits (e.g., operational, compliance, and financial). CobiT's underlying conceptual model asserts that to satisfy business requirements, information must meet seven criteria: (1) Effectiveness, (2) Efficiency, (3) Confidentiality, (4) Integrity, (5) Availability, (6) Compliance, (7) and Reliability (Appendix B provides detailed descriptions for each criterion as presented in CobiT 4.0). The conceptual model relates each CobiT process to the information criteria that the process affects, and therefore, should provide an auditor with a means of directly assessing specific controls for their effect on the quality of information, whether the audit is operational, compliance, or financial in nature (see Fig. 1). Furthermore, there are clear linkages between the CobiT information criteria and COSO's objectives related to effectiveness and efficiency of operations, compliance with laws and regulations, and reliability of information.3 Achieving the CobiT information criteria, therefore, has important implications for financial statement assertions as well as broader implications for the efficiency and effectiveness of operations. See CobiT 4.0 Appendix II, “Mapping IT Processes To IT Governance Focus Areas, COSO, CobiT IT Resources And CobiT Information Criteria” (IT Governance Institute 2005). 3

244

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Fig. 1. CobiT Version 4.0 conceptual model.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

245

For each CobiT process, the IT resources (i.e., assets) that the process affects are also identified. These resources consist of (1) People, (2) Information or data, (3) Applications, and (4) Infrastructure.4 Melville et al. (2004) provide a related business valuation model based on existing research that shows the important components of a good IT structure. The primary components of Melville et al.'s model consist of physical capital, human capital, and organizational capital. We note that each component of Melville et al.'s model maps directly into at least one of the IT resources within the CobiT framework. CobiT expands this model by adding Information as a critical IT resource. As is identified by Melville et al. (2004), research has shown that strong IT improves organizational performance. Given the importance of IT to organizational performance and the direct link between the primary components of the business value model and the IT resources of CobiT, one can expect risk assessments for a particular process to correlate with the IT resources it affects. For example, if CobiT's conceptual model is internally consistent and the IT resource, people, is considered a relatively high risk factor, then the assessed risk of not satisfying a particular CobiT process that is closely associated with the people resource should reflect this heightened risk. Likewise, a test of internal consistency within an audit context involves examining whether risk assessments for the various CobiT processes correlate with those information criteria which the conceptual model purports the process to affect. That is, suppose the information criterion of integrity is more complex, difficult to audit, or otherwise risky in comparison to the other information criteria. In this case, the risk of not satisfying a CobiT process associated with the integrity criterion should reflect its underlying complexity, etc. These arguments suggest that if CobiT is internally consistent, then the information criteria and IT resource constructs should exhibit construct validity. This leads to our first proposition: Proposition 1. Risk assessments for CobiT processes will be correlated with the underlying audit related characteristics for the information criteria and IT resources CobiT purports that the process affects. An additional proposition is set forth in relation to the propensity of auditors to seek additional information and assistance via postings to the Information Systems Audit and Control Association (ISACA) IT audit listserv.5 This particular listserv consists of audit related questions posed by IT auditors to other IT auditors and is distinct from a separate listserv maintained by ISACA that addresses specific CobiT issues.6 Hence, message threads on the listserv provide an unobtrusive measure of the need to obtain assistance and information on specific topics within a broad IT audit setting. Furthermore, the total number of postings (i.e., questions and responses) provides a measure of what it takes to provide a satisfactory answer. The number of postings related to a specific topic is a proxy for the complexity or the importance of the topic to auditors. The more complex the issue, the less likely that one response to a posted question will be sufficient to provide a complete solution, and will therefore lead to more posted responses and follow-up questions. The more important the topic, the more likely multiple people will respond to the posted message with a response. Therefore, a strong test of the internal consistency of CobiT's underlying conceptual model is to examine its association with the frequency with which auditors seek assistance via the listserv. 4 CobiT 3.0 identified five IT resources. CobiT 4.0, which was released in November of 2005, combined the Technology and Facilities IT resources into Infrastructure, thus reducing to four IT resources. 5 See www.isaca.org. 6 ISACA maintains several specialized listservs. Using the general audit listserv provides a stronger test of CobiT's applicability to an audit setting than using the listserv dedicated solely to CobiT issues.

246

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Data from the listserv provides a stronger test of the CobiT model's internal consistency than would data about purposeful auditor behavior, such as data about audit plans and tests. This is because auditors might adopt and use an internally inconsistent conceptual model based on their training and culture without being aware of the model's shortcomings. Seeking and giving help, on the other hand, is a spontaneous behavior that is likely to reflect audit needs associated with the risk associated with the particular issue. For instance, if CobiT constitutes an internally consistent conceptual model in an audit setting, one expects ITauditors to serendipitously seek help with CobiT processes when the information criteria associated with that process embody important audit characteristics related to various aspects of risk, such as its complexity, importance. That is, a process, which is complex, and that impacts information criteria should create a greater need for audit assistance. It should elicit, therefore, greater assistance from other auditors. This logic suggests that the help auditors seek in relation to a CobiT process will be influenced by the information criteria related to that process. The same expectation exists in regards to IT resources. This leads to our second proposition: Proposition 2. CobiT's conceptual model (i.e., processes, information criteria, and IT resources) predicts the extent to which an auditor will seek and give assistance related to an IT audit topic. Proposition 2 is a strong test of the theoretical consistency of CobiT's conceptual model because in order to support its premise the CobiT processes must be valid categories of IT activities within an audit context.7 Otherwise, if CobiT processes are not audit relevant then no association between CobiT and the need for information and assistance in an audit context (i.e., listserv postings) will result. Furthermore, CobiT's assertions regarding which information criteria and which IT resources are affected by each process are subject to the same logic. If CobiT does not extend to an audit context then no association will result between the amount of information an auditor seeks and the need for information related to the listserv topic as implied by the information criteria and IT resources asserted by the CobiT framework. In addition to the information criteria and IT resource taxonomy, beginning with Version 4.0, CobiT identifies the importance level it places on each CobiT process as shown in Fig. 1. If the CobiT framework is applicable to an audit setting, then the importance level (i.e., H = high, M = medium, or L = low) should be useful in audit planning and should correlate with the propensity of auditors to seek and give advice on the listserv. We, therefore, present our third and final proposition: Proposition 3. There is a positive association between the perceived importance level of an IT audit topic and the extent to which auditors seek assistance related to that particular ITaudit topic. 3. Test of CobiT using responses from experienced auditors 3.1. IT and non-IT auditor assessments As a first step to test our propositions regarding CobiT's conceptual model, audit relevant assessments of CobiT constructs were obtained from an expert panel consisting of 12 highly experienced IT auditors and 17 non-IT auditors.8 The IT auditors have a mean full-time work 7

Regardless of the results of this study, CobiT may still be a valid framework for IT related purposes other than as an internal control framework in an audit setting. 8 Of the 17 non-IT auditors one did not complete the demographic questions of the questionnaire, so the information for only 16 is reported here.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

247

experience of approximately 11.9 years, whereas the non-IT auditors have a mean work experience of approximately 2.1 years. Seven IT auditors held four-year college degrees and six held graduate degrees. Sixteen non-IT auditors held graduate degrees. Eight of the IT auditors work for a “professional assurance/consulting firm,” while four work in “insurance, real estate, finance, banking, and accounting.” Fifteen of the non-IT auditors work for a “professional assurance/ consulting firm,” while one works in “insurance, real estate, finance, banking and accounting.” The professional designations possessed by our IT auditors include three CPAs, eight CISAs, three CIAs, one CCNA, one CFE, one CFSA, and one CISSP. Two panel members did not indicate a professional designation. The non-IT auditors include eight CPAs. The mean self-assessed knowledge of the IT concepts in the questionnaire for the IT auditor and non-IT auditor groups respectively is 6.6 and 6.0 on a nine-point scale with one being “low knowledge of the concepts” and nine being “high knowledge of the concepts.” The mean self-assessed familiarity with CobiT is 7.0 and 4.6 on a nine-point scale with one being “very unfamiliar with CobiT” and nine being “very familiar with CobiT,” for the IT auditor and non-IT auditor groups, respectively. The non-IT auditors responded (mean = 3.1) to the question, “Please indicate the extent to which you personally apply CobiT in your day-to-day work activities” on a nine-point scale with one being “Very infrequently” to nine being “Very frequently.” These results suggest that the IT auditors have significant familiarity with CobiT and that even the non-IT auditors have some familiarity. The 12 highly experienced IT auditors performed three tasks in which they evaluated the following: (1) each of the information criteria, (2) each of the IT resources, and (3) assessed the risk associated with each of the 34 CobiT processes.9 The evaluation tasks took approximately 40 min to complete. The 17 non-IT auditors assessed only the risk associated with each of the 34 CobiT processes (i.e., task 3). The evaluation of each of the information criteria and each of the IT resources was based on providing evaluations for four measures of audit risk: complexity, client importance, client attention, and process risk.10 Consistent with the desirability of obtaining multiple measures of a single construct, each of these evaluations is expected to capture slightly different aspects of audit risk in an IT setting so that when combined, a better overall measure of IT audit risk emerges. Our expectation is that an auditor will seek more information about a CobiT process associated with an information criterion or IT resource that is complex, important, requires more attention or high risk. The motivation for each measure appears below. 3.1.1. Complexity Complexity in a process or transaction increases the risk of material misstatement, and therefore, increases audit risk. As an example, in reference to revenue recognition, in Messier, Glover, and Prawitt's 5th edition auditing textbook they state “recognition of revenue may involve complex calculations….In such circumstances, the auditor should assess the risk of material misstatement to be high” (2008, p. 386). As pointed out by Ridley et al. (2004, p. 7) the more complex the IT governance of an organization, “it is likely that there will be more interest in IT control from these organizations” (Ridley et al., 2004, p. 7). “The challenges include 9 At the time of the expert panel participation, CobiT 3.0 was being used. Therefore, the panel made judgments on five IT resources. As indicated in note two, CobiT 4.0 combines Technology and Facilities. During the analysis of the expert panel responses, these two IT resources are combined to reflect CobiT 4.0. 10 Audit difficulty was also measured but is not correlated with the other risk measures and is therefore not included in any analyses. Initially audit difficulty was included as it seemed reasonable that it would impact auditor risk judgments. As it turns out, it appears that our participants viewed audit difficulty more as being related to detection risk in the audit risk model, while the other factors relate to inherent risk and control risk associated with the client's system.

248

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

identifying a hornet's nest of controls and interfaces among decentralized business units and trying to manage the efforts with scarce resources” (Hoffman, 2004, p. 2). 3.1.2. Importance As a whole, “the importance of IT governance can be appreciated in light of Gartner Group's finding that large organizations spend over 50% of their capital investment on IT” (Ridley et al., 2004, p. 1). More specifically, we expect management to place more importance on IT areas that are high risk. For instance, self-assessment programs recognize the relationship between risk and, “how important the process is for their business objectives” (Lainhart, 2001, p. 21). The more important the IT process, the more likely auditors are to seek assistance when a question arises. 3.1.3. Attention Similar to importance, we argue that organizations focus their attention with respect to IT governance on areas of higher risk. This notion is consistent with self-assessment programs and the risk-based audit approach applied by major accounting firms. “Management can concentrate on the areas of high risk identified through auditors' assessments and then use CoBiT's high level and detailed control objectives to determine cost-effective means for mitigating these risks” (Lainhart, 2001, pp. 21–22). 3.1.4. Process risk The risk associated with a process failing, regardless of the “type” of risk (i.e., inherent risk, control risk, business risk, or fraud risk) increases audit risk. “COBIT is a breakthrough tool that helps enterprises balance IT risk and investment in controls” (Lainhart, 2001, pp. 19–20). Each evaluation was elicited on a scale from one to nine with one being “very low [insert one of the four measures]” and nine being “very high [insert one of the four measures]” (i.e., “very low complexity” and “very high complexity”) as it relates to a “typical organization.” Table 1 shows mean ratings from the IT auditors for each information criteria and IT resource for each measure. Although the IT auditors expressed a high level of familiarity with the concepts in the survey and with CobiT, to ensure reliability the IT auditors first read the CobiT definitions of the information criteria and IT resources before completing their assessments (IT Governance Institute, 2000).11 We perform an exploratory factor analysis to determine whether the above measures load on a single risk factor. This analysis suggests that complexity, client importance, client attention, and process risk load on the same factor. Table 1 also shows what we label a “Combined Assessment,” for each information criterion and each IT resource. The combined assessment represents an “overall” risk measure score that is derived by weighting the assessments for each of the four different risk measures (complexity, client importance, client attention, and process risk) by their factor scores and then summing the products (this process is described in more detail below). The combined assessments are used to

11

The manner in which we obtain the ratings makes it highly likely that the responses result from knowledge gained from experience rather than from training in CobiT. For instance, Fig. 1 of the paper shows that there are 226 instances in which an Information Criteria or IT Resource apply to one of the 34 CobiT processes. In addition, three importance levels are associated with 34 separate IT processes. Both facets of the framework far exceed a person's ability to recall from memory based on training alone. More importantly, it is unlikely that our auditors previously encountered ratings of risk, complexity, client attention, etc. as they apply to the specific Information Criteria or IT Resources or CobiT processes. In contrast, the IT auditors do encounter circumstances during their audits in which they learn to associate these factors together.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

249

Table 1 Expert panel assessments of CobiT information criteria and IT resources Mean audit related assessments Framework construct

Combined assessment (Std Dev)

Factor scores (Weight) a

Complexity

Client importance

Client attention

Process risk

0.79560

0.63194

0.90126

0.94105

Panel A: Information criteria Effectiveness Efficiency Confidentiality Integrity Availability Compliance Reliability

5.34 4.51 5.57 5.70 5.23 5.98 5.72

6.42 6.50 6.25 6.67 5.25 6.33 5.58

7.25 6.25 6.67 7.33 7.50 7.50 7.92

6.17 4.58 6.25 6.25 6.25 7.17 6.83

6.50 5.08 7.92 7.67 6.75 8.17 7.75

Panel B: IT resources People Information Applications Infrastructure a

5.38 6.21 6.27 5.24

5.67 6.75 7.00 6.04

7.33 8.33 7.92 6.96

6.50 7.50 7.92 6.21

6.92 7.92 7.83 6.54

CobiT 4.0 released in November 2005 combined the IT Resources of technology and facilities into one category, which it labeled “Infrastructure.” The mean audit related assessments in the table for Infrastructure consists of the expert ratings for technology and facilities added together, divided by two. Combined assessments equal the sum of the audit related assessments weighted by the factor scores. a The prior communality estimate for each variable was set at its squared multiple correlation with all other variables.

create a “framework” score according to the relationships inherent in CobiT. The framework scores are then used for predicting auditor listserv behavior as described later. Regarding the third task, all auditors (the 12 IT auditors and 17 non-IT auditors) were asked to “consider the risk to the typical organization associated with an unsatisfactory outcome in each of the following CobiT processes.” They indicated their CobiT process assessments of risk on a scale from one to nine with one being “very low risk” and nine being “very high risk.” Mean risk assessments for each CobiT process are shown in Table 2 along with framework scores (explained below). Framework scores are computed for each CobiT process in the following manner: (1) the information criteria and IT resources associated with the process are coded as a one if indicated in Fig. 1 and as a zero otherwise, and (2) the result is multiplied by the combined assessments shown in the first numeric column of Table 1. These numbers are then summed and averaged into a single framework score for each CobiT process.12 The framework scores permit a test of whether CobiT's conceptual model, from an audit relevant standpoint as reflected in the expert panel's assessments of the information criteria and IT resources, predicts IT auditor behavior in seeking and providing assistance on IT audit related topics. The use of the framework score allows the analysis to reflect audit risk as measured serendipitously by the information criteria and IT resources associated with each CobiT process. To see how the framework scores are computed, recall that the CobiT conceptual model is categorized into four domains, which in turn are associated with 34 IT processes. Conceptually, 12

Framework scores consisting of a simple count (i.e., unweighted by the combined assessments of the experts) produce essentially the same results as analysis using weighted framework scores.

250

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 2 Expert panel: mean risk assessments and framework scores CobiT process

Description

N = 29 risk assessment

N = 12 framework scores a

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 AI1 AI2 AI3 AI4 AI6 b AI7 DS1 DS2 DS3 DS4 DS5 DS6 DS7 DS8 DS9 DS10 DS11 DS12 DS13 ME3 c

Define a strategic IT plan Define the information architecture Determine technological direction Define the IT processes, organization and relationships Manage the IT investment Communicate management aims and direction Manage IT human resources Manage quality (PO 11 from CobiT 3rd ed.) Assess and manage IT risks Manage projects Identify automated solutions Acquire and maintain application software Acquire and maintain technology infrastructure Enable operations and use Manage changes Install and accredit solutions and changes (AI 5 from CobiT 3rd ed.) Define and manage service levels Manage third-party services Manage performance and capacity Ensure continuous service Ensure systems security Identify and allocate costs Educate and train users Manage service desk and incidents Manage the configuration Manage problems Manage information Manage the physical environment Manage operations Ensure compliance with external requirements

7.24 6.24 6.14 6.86 6.28 6.59 5.83 7.21 7.72 6.62 6.83 6.86 6.59 7.14 8.14 7.41 6.21 6.59 6.62 6.86 8.00 5.69 6.97 5.45 6.90 7.34 7.62 6.28 6.86 7.90

3.00 3.05 1.94 1.38 2.95 2.08 1.38 4.03 5.56 2.43 1.94 2.50 2.36 4.49 4.51 4.51 5.56 5.56 2.42 3.47 4.66 3.03 1.38 1.95 3.50 3.47 1.60 1.47 3.99 3.16

a Framework Score represents the combined assessments from Table 1 for each information criteria and IT resource identified to affect the given CobiT process. After the respective combined assessments are added together, the total is divided by 11 to arrive at the framework score. b The assessments made by the expert panel were based on the CobiT 3.0 framework. Cobit 4.0 released in November 2005 has added a new AI5. The previous AI5 under CobiT 3.0 is now AI7 and is shown as such in this table. c The CobiT process was PO8 in CobiT 3.0. The remaining Monitor and Evaluate processes in CobiT 3.0 do not map to CobiT 4.0, and therefore, are not used in the analysis.

CobiT asserts that each process affects two or more information criteria and one or more IT resource as shown in Fig. 1. If the information criterion has a dot in the row for a given process, the process is said to affect the information criterion. Likewise, if the IT resource has a dot in the row for a given process, the process is said to affect the IT resource. For example, Fig. 1 shows that the process “PO3 Determine Technological Direction” affects the effectiveness and efficiency information criteria and the applications and infrastructure IT resources. The combined assessments (from Table 1) for the effectiveness (5.34) and efficiency (4.51) information criteria and the applications (6.27) and infrastructure (5.24) IT resources sum to 21.36. Dividing by 11 (seven information criteria + four IT resources) produces a framework score of 1.94 as shown in Table 2. This score represents the IT auditors' assessment of the complexity, client importance,

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

251

client attention and process risk associated with CobiT process PO3 as reflected by the information criteria and IT resources that are identified as applying to this particular CobiT process. From a practical standpoint, one can think of a higher score as representing what CobiT's conceptual model asserts should be an increased audit risk for that particular CobiT process. 3.2. Preliminary analysis As a first step to testing the internal consistency and internal validity of CobiT, we perform confirmatory factor analysis to verify that the combined assessments from Table 1 for the seven information criteria and the four IT resources fall into these two categories as asserted by CobiT. The resulting model fails to fit the data and so we next conduct an exploratory factor analysis (with varimax rotation) to see what factors might underlie the IT auditors' assessments. As shown in Table 3, the rotated factor pattern produces three factors. Together, these factors account for 84.5% of the variance. The Reliability, Confidentiality, Integrity, and Efficiency information criteria load on the first factor. The Compliance and Effectiveness information criteria and the People and Data IT resources load on the second factor. The Availability information criteria and the Applications and Infrastructure IT Resources load on the third factor. This analysis suggests that information criteria and IT resources represent three separate constructs that do not conform strictly to the information criteria and the IT resources categories presented in CobiT. We discuss the implications of this finding in the discussion section of the paper. Further analysis is not dependent on these measures loading on their respective CobiT categories. 3.3. Proposition 1—CobiT internal consistency and risk assessments Proposition 1 provides an initial test of CobiT's internal consistency by examining whether expert panel assessments related to the information criteria and the IT resources combine, as asserted by CobiT's conceptual model, in a manner that they correlate with separate risk assessments for the CobiT process. To test this proposition the risk assessments for the CobiT processes from Table 2 are regressed on the framework scores from the same table using a generalized linear model while controlling for individual differences among the expert IT auditors. The results shown in Table 4 indicates that the framework scores are highly associated Table 3 Rotated factor pattern based on expert panel combined assessments

IC = Information Criteria, R = IT Resources.

252

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 4 GLM analysis of expert risk assessments for CobiT processes regressed on framework scores from Table 2 Variable

df

Type III sum of squares

F-Value

p-value

Auditor Framework Model Error R2 = 0.2138

11 1 12 311

88.0043 59.5932 168.3678 618.9994

4.02 29.94 7.05

b0.0001 b0.0001 b0.0001

Each expert is associated with a vector containing one row for each CobiT process and two variables in each row: a framework score and the dependent measure, i.e., the expert's risk assessment for the process (see Table 2 for the framework score and risk assessment). Framework is computed by first coding each Information Criterion and IT Resource as a zero or one variable per Fig. 1. These are then multiplied by the combined assessments from Table 1 to arrive at weights for each Information Criterion and IT resource based on the dimensions of (1) complexity, (2) client importance, (3) client attention, and (4) process risk. The resulting numbers are then summed for each CobiT process to arrive at the framework scores found in Table 2 and used in this analysis.

with the IT auditor's risk assessments (F = 29.94; p b 0.0001).13 Note that the risk assessment used in the framework score is based on the IT experts' assessment of risk associated with each information criterion and IT resource, and is combined with the other audit relevant assessments per the CobiT conceptual model. In contrast, the risk assessment used here as the dependent variable is provided directly from each of the IT auditors' for each of the 34 CobiT processes. Hence, using two distinct measures, one that focuses solely on the CobiT processes (risk assessments) and the other that focuses solely on the information criteria and IT resources (framework scores), the CobiT conceptual model demonstrates significant (internal) convergence and consistency, thus supporting Proposition 1. The significant effect of auditor simply controls for individual differences between auditors in overall risk assessments. 4. CobiT examination using archival information The primary examination of the CobiT conceptual model involves analyzing postings to a general IT audit listserv maintained by ISACA in which auditors post and respond to IT audit related questions. The ISACA listserv provides a surrogate for how IT processes may affect audit risk in that audit risk should influence auditor behavior with respect to seeking and giving audit help.14 Auditors are more likely to seek help for something they audit that is of higher risk, and they are more likely to audit IT processes that they associate with more risk. Furthermore, auditors are more likely to question their own abilities and knowledge and thusly be more likely to ask for help on issues of importance and high complexity (i.e., high risk). Auditors are also more likely to take their time to respond to questions posted on a listserv when they believe the question is important, complex, or related to a highly risky issue. For these reasons, we argue that a listserv in which IT auditors seek and give audit related advice to each other is an unobtrusive measure of audit risk that can be categorized as to the topic being discussed. 13

Only the IT auditors are used in this analysis, because they are the only participants who provided assessments of the information criteria and the IT resources that make up the framework scores. The significant result on the auditor variable (coded as a class variable) suggests that the assessment of audit risk is significantly impacted by characteristics specific to the individual auditor. 14 At this time, no publicly available empirical data exists that relates specific IT processes to audit risk. Such data are unlikely to exist for sometime.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

253

Because the study seeks to examine IT processes in an audit context, any listserv not used by auditors to discuss IT issues is irrelevant to the current study. At the same time, it is important that the listserv not bias the findings by specifically addressing only CobiT questions. While we expect most participants on the general ISACA listserv to be familiar with CobiT, if they had a question that they believed was specific to CobiT, we expect them to use the CobiT listserv. A word search on the listserv postings confirms that only six threads contain “Cobit” (not case sensitive) in the subject line. Hence, the general ISACA listserv appears to address IT audit related issues by IT audit specialists but does not address issues explicitly arising from the use of CobiT. Additionally, we gathered risk ratings for the 34 IT processes from 17 non-IT auditors. The correlation between the 12 IT auditors and the 17 non-IT auditors is 0.718. In general, the IT and non-IT auditors are similar in their risk assessments on the 34 IT processes. As shown in Table 5, 616 messages that were posted from January 2002 through December 2003, and 601 messages that were posted from September 2004 through December 2005 are analyzed.15 These postings related to 342 and 297 separately identified message threads, respectively, in which a query along with its follow-up responses constitutes a single thread.16 Two Ph.D. students whose area of study is information systems and who were blind to the purpose of the study separately coded the message threads, classifying each thread to the CobiT process to which they felt it primarily related. Their initial agreement rate is 62% (Kappa = 0.59) for the period ending in 2003 and 64% (Kappa = 61) for the period ending in 2005 which is substantially greater than chance (i.e., 1 out of 34 processes = 3%).17 All disagreements were resolved between the two coders by discussion. As shown in Table 6, 205 auditors posted to the listserv during the 2002–2003 period, while 152 auditors posted to the listserv during the 2004–2005 period for a total of 357 different IT auditors. Slightly fewer than half of the auditors posted multiple messages. Three auditors posted more than 30 messages. E-mail addresses were examined for the type of domain and country of origin shown in Table 7. Although ISACA is an international organization, as can be seen in Table 7, 280 contributors are domiciled in the U.S. (79%), while 74 (21%) are from domains outside the U.S. CobiT is asserted to be a comprehensive, open standard based on industry best practices, and on this basis, meets the requirements to be a useful framework. Some indication of the comprehensiveness of CobiT is the number of audit specific issues that were posted on ISACA's general listserv that could not be coded as pertaining to a specific CobiT process by our coders. As can be seen from Table 5, 21 of the 342 message threads (6.1%) in the 2002–2003 period were not coded to a specific CobiT process, while 51 of the 296 message threads (17.2%) in the 2004–2005 period were not coded to a specific CobiT process. Initially, the proportion not coded in the 2004–2005 15 We begin the second time period (September 2004) six months after the issuance of AS2 by the PCAOB (2004) so that the auditors had time to implement this standard in their work. 16 Although internal controls have been an important component of the audit process, given that the messages from the first period were immediately surrounding the period when SOX was implemented (and prior to the ratification of AS2), it is possible that the issues related to internal controls might not be reflected in the postings on the listserv. The second period is analyzed in order to test to see if there is a delay in the impact of the new regulations on the postings on the listserv. 17 While both of the PhD students are accounting information systems minors in their doctoral program, one of the coders had significant audit experience and limited IT experience, while the other coder had significant IT experience and limited audit experience. Hence, a somewhat higher than normal initial disagreement is expected, but the result of their combined experience should be a higher quality coding after discussion. However, because of the subjectivity of the coding process (i.e., a thread may relate to two processes but the coders may disagree on which is the primary process), we analyze instances in which the coders initially indicated a secondary process to determine if substituting the rejected classification alters our conclusions. Out of the 639 threads and 1217 messages analyzed in the study, 55 threads and 86 messages are associated with a secondary classification. This re-analysis does not change the interpretation of our results.

254

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 5 Listserv postings 2002–2003

2004–2005

a

CobiT process

Thread count

Message count

Thread count

Message count

DS5 ME2 DS11 ME3 DS4 ME4 AI3 AI6 DS10 DS7 PO2 AI7 DS12 ME1 AI2 DS8 DS9 PO6 PO7 PO3 PO4 PO8 PO9 PO10 DS13 PO1 DS3 AI4 PO5 AI1 DS1 DS2 DS6 Not coded Total

114 113 25 13 9 6 4 4 4 4 4 3 3 3 2 2 2 2 2 1 1 0 0 0 0 0 0 0 0 0 0 0 0 21 342

242 189 25 18 24 11 13 15 6 4 4 3 3 6 2 2 3 18 3 1 1 0 0 0 0 0 0 0 0 0 0 0 0 23 616

99 62 24 12 7 6 0 1 1 0 0 0 3 1 2 1 1 1 0 0 8 4 7 0 0 0 0 1 4 0 0 1 0 51 297

196 138 67 16 18 10 0 3 1 0 0 0 3 2 4 8 1 2 0 0 23 12 7 0 0 0 0 3 12 0 0 1 0 74 601

a

A thread represents a stream of messages that relate to a particular topic.

Table 6 Number of unique contributors by number of postings Posting frequency

1 2–5 6–10 11–30 31+ Total

Number of unique contributors 2002–2003 Listserv

2004–2005 Listserv

110 77 12 5 1 205

83 49 14 4 2 152

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

255

Table 7 E-mail domiciles Country and e-mail domain USA .com .edu a .gov .mil .net .org .us Total USA Non-USA .ae .ar .au .br .ca .cl .co .de .ed .es .fr .id .il .in .jo .ke .mil .my .nl .nz .pe .pk .qa .se .tn .tr .uk .za Total non-USA Total

Number of contributors 2002–2003 Listserv 99 22 5 2 12 14 7 161

2 1 7 1 9 1

2004–2005 Listserv 83 11 3 9 9 4 119

1 1 3 1 6 1

1 1

1 1 3 1 2 2 1

1 1 1 2 1 1 2 1 1 1

1 1 2 2 2 42 203

1 1 3 2 32 151

a Postings from the.edu domain are IT auditors and not professors. Two 2003 and one 2005 postings do not have e-mail addresses.

period appears to be relatively high. Analysis of all 72 threads, however, suggests a few reasons why postings were not coded to specific CobiT processes. A number of threads dealt with issues that were so general that they span the entire CobiT spectrum or otherwise did not meet the objectives of the study. By including these in the analysis, it would muddy the results, as they would indicate all of the CobiT categories apply. Examples include announcements of conferences, meetings, and general

256

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

CPE training. One typical thread in this category dealt with how to use the “COSO framework to meet Section 404 requirements.” Such a thread regarding the use of COSO could be coded as applicable to many CobiT processes. A few threads dealt with issues that while not spanning the entire framework, related to multiple CobiT processes and so could not easily be coded for purposes of the present study. Discussions related to fraud dominated these threads. A significant number of threads dealt with current news events rather than specific audit issues. Examples include discussions about changes to state and national laws, discussions about Internet voting and messages dealing with U.S. Governmental Accounting Office (GAO) activities. Given that our study examines whether CobiT covers IT audit issues, non-audit issues from the listserv are not included or applicable to the analysis. Another set of messages dealt with listserv administration and with job seeking advice. It seems logical, given the timing of the adoption of AS2, that the 2004–2005 time period would have a significant number of messages related to seminars and training. Of the 72 threads that were not coded to a specific CobiT process, not one thread related to what could be considered an IT process that is missing from the CobiT framework. Nevertheless, based on the nature of our data, these findings must be considered preliminary. After eliminating the threads that clearly did not represent specific audit questions, the set of potential threads that could not be coded to CobiT processes is empty. Our subjective analysis, therefore, leads us to conclude that the CobiT encompasses every audit specific question posted to the listserv. 4.1. Proposition 2—thread and message counts reflect the CobiT conceptual model Proposition 2 suggests that the underlying complexity, client attention, importance, and process risk embodied by the information criteria and IT resources associated with each CobiT process should influence the amount of aid and assistance ITauditors seek and give on the ISACA listserv. In order to examine Proposition 2, we regress counts of messages and threads coded to each CobiT process as shown in Table 5 on the CobiT framework scores from Table 2, while controlling for the time period (2002–2003 versus 2004–2005).18 The framework score reflects both audit related assessments across the five dimensions and the role that the information criteria and IT resources play in CobiT. The expectation is that the number of listserv postings should be positively correlated with framework scores. A positive coefficient indicates that as the number of information criteria and IT resources increases, and the higher the combined assessment across the four audit risk measures, there will be more listserv postings. We include time period to test whether implementation experience regarding SOX compliance is affecting IT auditors' seeking and giving assistance via the listserv. Several issues must be considered in this analysis. First, because the dependent variable is a count, it is unlikely to follow a normal distribution so that Poisson or negative binomial regression is recommended (Cameron and Trivedi, 1998). Poisson regression requires that the sample mean and variance be equal; which assumption our data violate—a condition known as over dispersion. We therefore conduct a negative binomial regression. As can be seen in Table 8, the framework score is highly significant whether the dependent measure is the number of threads in Panel A ( p = 0.0023) or the total number of posted messages in Panel B ( p = 0.0043). Time period is not significant in either the analysis using the number of threads 18

Another way to consider analyzing the relation between the CobiT conceptual model and the message postings is to look at characteristics of the messages such as quality, clarity, completeness and accuracy of the postings. These measures, however, lack clear predictions. For example, an important but complex topic may result in inaccurate or unclear postings because of its complexity. Conversely, the meaning behind a single thread is clear: i.e., a single issue or idea that has evoked sufficient motivation to cause an auditor to seek or give help. For this reason, we only analyze counts. Future research might explore the relation between audit risk and characteristics of message quality.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

257

Table 8 Negative binomial regression of thread and message counts on framework scores from Table 2 Information criteria

df

Estimate

Chi-square

p-value

Panel A: number of threads per CobiT process a Intercept 1 Framework score 1 Time period 1 Framework a time period 1

0.3383 0.4640 0.0130 − 0.0330

0.35 9.30 0.00 0.05

0.5516 0.0023 0.9817 0.8284

Panel B: number of messages per CobiT process b Intercept 1 Framework score 1 Time period 1 Framework a time period 1

0.9780 0.4754 0.4929 − 0.1303

2.52 8.14 0.64 0.61

0.1124 0.0043 0.4237 0.4342

Main effects

Note: A thread represents a stream of messages that relate to a particular topic. a Goodness-of-fit criteria: deviance Chi-square value of 67.0964 divided by 62 df = 1.0822. b Goodness-of-fit criteria: deviance Chi-square value of 68.2458 divided by 62 df = 1.1007.

or the number of messages. Recall that the framework score aggregates the expert panels' assessments for complexity, client attention, client importance, and process risk according to each information criterion and IT resource based on the indicators from Fig. 1. Hence, we view the correlation between the framework score and practicing auditors' listserv behavior to provide remarkable evidence consistent with Proposition 2 and consistent with our assertion that CobiT is a sound framework when used in an IT audit setting. Essentially, Proposition 2 proposes a link between CobiT and listserv behavior. Additional evidence in support of such a link is obtained by regressing message counts (using negative binomial regression) on the risk assessments associated with each CobiT process as supplied by the expert panel and as shown in Table 2. As can be seen in Table 9, the risk assessments from the expert panel predict the number of messages posted by the auditors ( p b 0.0001). The significant relation between the risk assessment and the number of messages is not affected by the time period in which the posting was made. 4.2. Proposition 3—CobiT importance predicts listserv behavior Proposition 3 suggests that if the CobiT importance levels as shown in Fig. 1 are meaningful in an audit context, then one expects that more important processes should result in more postings to Table 9 Negative binomial regression number of listserv messages regressed on expert panel risk assessments

Intercept Time period Risk assessments Time period a risk assessment

df

Estimate

Chi-square

p-value

1 1 1 1

− 8.1294 0.9756 1.4630 − 0.1334

15.36 0.22 24.02 0.20

b0.0001 0.6379 b0.0001 0.6547

Note: Unit of Analysis = CobiT Process by Time Period. AI5, ME1, ME2, and ME4 added to CobiT after expert panel data collected and therefore excluded from this analysis. a Goodness-of-fit criteria: deviance Chi-square value of 60.2062 divided by 56 df = 1.0751.

258

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

Table 10 Negative binomial regression number of listserv postings regressed on CobiT importance rating df

Estimate

Chi-square

p-value

Panel A: message counts Intercept CobiT importance Time period Time period a importance

1 1 1 1

− 0.0155 1.2861 0.1228 − 0.0854

0.00 12.62 0.03 0.06

0.9833 0.0004 0.8683 0.8136

Panel B: thread counts Intercept CobiT importance Time period Time period a importance

1 1 1 1

− 1.7164 1.2867 − 0.2922 0.0539

1.09 15.00 0.18 0.03

0.2973 b0.0001 0.6707 0.8710

Panel C: least square mean number of postings per CobiT process by CobiT importance rating CobiT importance rating

Message count

Thread count

Low Medium High

1.29 9.33 17.17

1.42 9.08 17.50

a

Goodness-of-fit criteria: deviance Chi-square value of 68.0897 divided by 62 df = 1.0982. CobiT importance ratings are provided for each CobiT process within the CobiT framework as shown in Fig. 1.

the listserv. To test this proposition we regress the number of listserv postings (message and thread counts) on the importance level associated with each CobiT process. Importance is coded as a discrete variable with 1 = low, 2 = medium, and 3 = high.19 The time period in which the data were collected is included as a control variable. As can be seen from Panels A and B of Table 10, CobiT importance is highly significant for message counts ( p = 0.0004) and for thread counts (p b 0.0001) and is unaffected by time period. This provides strong evidence to support Proposition 3. 5. Discussion of the findings Prior to discussing the findings, it is important to discuss some of the limitations and strengths of the study. The study employed two separate sources of data to examine CobiT as a theoretical framework pertinent to auditing (including operational, compliance, and financial auditing). Each of these sources has limitations that are well recognized. For instance, the survey data is limited because the participants are in a hypothetical context and might respond differently than they would in practice. Additionally, the archival data is limited to only the data that are available. However, an important strength of the study is that by using a combination of data sources we mitigate the inherent limitations of survey and archival research methods when used in isolation. Using the survey data we obtain various audit related assessments from a panel of highly experienced IT auditors yielding consistent results. Using the archival data, we obtain evidence from the ISACA listserv that is unobtrusive and reflects actual auditor behavior in the field. By combining these two sources of data, we observe some remarkable correlations when superimposed onto CobiT's conceptual model. When considered together rather than 19

Analysis with importance coded as a categorical variable produces similar results but its presentation is more difficult.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

259

separately, the two sources of data employed in this study produce consistent and compelling evidence that CobiT's conceptual model is internally consistent and useful when applied to auditing IT controls. From a practical standpoint, the results of this study suggest that it is very important and potentially very useful for the audit profession to seek academic examination of its practices. Such examination provides highly needed evidence to policy-makers that either supports or calls into question the conceptual foundations of current audit practice. In this case, the findings suggest that the CobiT framework is significantly related to overall risk assessments of the CobiT processes for which they are associated. Furthermore, the results indicate that the CobiT framework can be used to predict the auditors' behavior in terms of seeking and giving IT audit related help as reflected by the listserv postings. Together, these results should give auditors and policy-makers assurance that CobiT is an appropriate supplement to COSO and in an IT setting. At the same time that our results generally support CobiT, the results also uncover some areas that warrant additional scrutiny. In particular, seven of the thirty-four CobiT processes, as shown in Table 5, have no associated listserv posting. This requires further study into how these seven particular IT processes relate to audit settings. Furthermore, we find that the seven information criteria and the five IT resources do not load cleanly on their respective categories. Rather, the IT auditors in our study appear to think about these 12 items along three distinct dimensions. This inconsistency is hardly surprising in that CobiT's original audience was primarily management whereas its application today has changed towards its use as an audit framework for IT control. This change in focus very likely changes the way that auditors think about information criteria and IT resources. We note that, at the present time, the accounting and information systems domains lack an empirically validated theory of internal control in the sense of identifying the variables that determine good control. Recognizing that the factor analysis reported in the paper is exploratory and based on a relatively small sample, we conjecture on what the three factors may mean as a start in building a preliminary theory of IT control. The first and strongest factor consists of four of the seven information criteria and no IT resources. We interpret this factor to be an information quality dimension as reflected by information reliability, confidentiality, and integrity obtained in an efficient manner. We interpret the second factor to consist of IT processing considerations that are strongly related to controls. That is, effective compliance with laws, regulations, and contracts that is affected by people and by having the necessary data. The third factor we interpret as audit considerations relating to IT design (i.e., applications and infrastructure), thus ensuring the availability of information to the business. Extending this interpretation to a preliminary model of internal control in an IT environment suggests that processes that affect information quality, information processing, and system design directly impact the effectiveness of internal controls. These dimensions are not too dissimilar from the CobiT framework, which suggests that business requirements for information embody (1) quality requirements, (2) fiduciary requirements, and (3) security requirements (IT Governance Institute, 2005). A third aspect of our study, which suggests that CobiT needs additional work in an audit setting, is that the analysis reported in Table 4 shows only a modest R-square (0.2138) when relating the CobiT framework to risk assessments for IT processes. The low proportion of explained variance may be the result of large individual differences in opinions about risk. Alternatively, some significant variables that impact audit risk for IT processes may be missing from the CobiT framework. One possibility may be that CobiT does not adequately consider the environment outside the organization and how variables associated with the

260

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

competitive, legal, and economic environment might interact with IT processes. Along these lines, CobiT has been criticized for being weak on security issues and companies typically augment both the COSO and CobiT frameworks with specific security frameworks such as ISO 17799. We collected the data from the listserv over two different periods of time. The first period covered the time immediately after Congress enacted SOX, whereas the second period covered a period of years after the PCAOB released AS2. Although both periods are post SOX, it is possible that auditors' response to internal control is still evolving. For this reason, we include a variable in each analysis to control for the possible effects that the evolving audit environment might have on the auditors' behavior. Throughout, we find no effect for the time period in which the data were collected. One possible explanation for the consistency of behavior over this time frame is that IT auditors were already highly engaged in IT control issues prior to or at the outset of the post SOX era. IT auditors have been applying principles of internal control consistent with the ideas of CobiT as it relates to operational, compliance, and financial auditing, because internal controls are critical to the processing of information within and between organizations. The fact that we find strong support for our propositions suggests that IT auditors were already up to speed in terms of IT control issues (or at least became up to speed in a short amount of time). Some might argue, however, that financial statement auditors without IT experience are now wrestling with IT control issues. Hence, further research is needed to extend this study to the time period after auditors have gained greater experience performing Section 404 audits and to different groups of auditors with a specific focus exclusively on financial statement audits (or the audit of internal controls as it relates to the financial statement audit) as opposed to auditing in general. In personal conversations between the authors and IT auditors about Section 404 work, the auditors tell us that IT departments typically resist outside frameworks although auditing standards clearly impose a requirement that the framework be publicly available. Efficiencies should result by having IT departments and auditors sharing the same framework. Any framework that is acceptable from not only a financial audit perspective but also from an operational perspective is preferred to one that is only useful in the financial audit. COSO appears to serve this need at a relatively general level. At an operational level, we note that CobiT was initially developed as an IT benchmark consisting of best practices. From that context, we investigate the appropriateness of CobiT to an audit setting. To the extent that CobiT is applicable as a dual use framework; organizations can achieve efficiencies in either operations and/or IT audits through its use. Issues related to efficiency and multiple use provide distinct possibilities for future research. Testing the internal consistency of the CobiT processes, information criteria, and IT resources as they apply to an audit setting may be an important first step in developing a theory of internal control. The CobiT framework conceptualizes and describes current practice with the aim of helping us understand the domain. As defined by Merriam-Webster, a framework is “a basic conceptual structure (as of ideas)” and as such closely resembles a theory. CobiT is a taxonomy of IT processes and related controls that are asserted to affect certain information criteria and IT resources. If the boundaries between processes are not valid in an audit setting, or if these processes do not affect the information criteria or the IT resources that they are purported to affect, we would have observed no significant results. Conversely, we tested and found support for these assertions. By subjecting CobiT to the analysis in this paper, the present study demonstrates that the relations and constructs within CobiT are relevant to an audit in a manner that one would expect a theoretical model to behave.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

261

Considering our test of CobiT as a first step to developing a theory of IT control opens the possibility for future research in several areas. These include studies into the types of control deficiencies and their characteristics that give rise to variations in audit programs and audit opinions. CobiT provides a means of classifying such control deficiencies and the results of this study demonstrate that these classifications relate to various aspects of audit risk. Furthermore, we only tested the core, basic structure of CobiT's conceptual model. CobiT contains other testable constructs including a comprehensive and well-articulated maturity model for IT control. The maturity model enables management of a company to evaluate and determine where on the internal control quality spectrum their controls are currently located. We hope that the positive results of this study will encourage others to begin a collaborative effort with the aim of developing a comprehensive, validated, practical, and generally accepted theory of internal control as it relates to information technology. Acknowledgements The authors contributed equally to the study and are listed in alphabetical order. This paper benefited from helpful comments from Wendy Bailey, Mark Cecchini, Michael Cipriano, Uday Murthy, Yi-Jing Wu and workshop participants at Brigham Young University, especially Scott Summers, Mark Zimbelman and Doug Prawitt. We also thank Kelvin Liu and Yi-Jing Wu for assistance with the data. Appendix A. Concepts evaluated by expert panel The following definitions were provided to the expert panel to ensure common understanding of the concepts against which a “typical organization” was to be judged. Complexity refers to the inherent complexity of the CobiT Information Criteria (IT Resources) considered separately from the audit and in relation to a typical organization. Client Importance refers to how critical each of the CobiT Information Criteria (IT Resources) are to the mission of a typical organization. Attention refers to the amount of time and resources the client devotes to each of the CobiT Information Criteria (IT Resources) in a typical organization. Process Risk refers to the typical organization associated with not achieving each of the CobiT Information Criteria (IT Resources). Appendix B. Information criteria and IT resources definitions from CobiT 4.0 Information criteria To satisfy business objectives, information needs to conform to certain control criteria, which CobiT refers to as business requirements for information. Based on the broader quality, fiduciary and security requirements, seven distinct (and certainly overlapping) information criteria are defined as follows: • Effectiveness deals with information being relevant and pertinent to the business process as well as being delivered in a timely, correct, consistent and usable manner. • Efficiency concerns the provision of information through the optimal (most productive and economical) use of resources.

262

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

• Confidentiality concerns the protection of sensitive information from unauthorized disclosure. • Integrity relates to the accuracy and completeness of information as well as to its validity in accordance with business values and expectations. • Availability relates to information being available when required by the business process now and in the future. It also concerns the safeguarding of necessary resources and associated capabilities. • Compliance deals with complying with those laws, regulations and contractual arrangements to which the business process is a subject, i.e., externally imposed business criteria, as well as internal policies. • Reliability relates to the provision of appropriate information for management to operate the entity and exercise its fiduciary and governance responsibilities. IT resources The IT resources identified in CobiT can be defined as follows: • Applications are the automated user systems and manual procedures that process the information. • Information is the data in all their forms input, processed and output by the information systems, in whatever form is used by the business. • Infrastructure is the technology and facilities (hardware, operating systems, database management systems, networking, multimedia, etc., and the environment that houses and supports them) that enable the processing of the applications. • People are the personnel required to plan, organize, acquire, implement, deliver, support, monitor and evaluate the information systems and services. They may be internal, outsourced or contracted as required. References Cameron AC, Trivedi PK. Regression analysis of count data, econometric society monograph no. 30. Cambridge University Press; 1998. Colbert JL, Bowen PL. A comparison of Internal Controls: COBIT, SAC, COSO, and SAS 55/78. IS Audit Control J 1996;4:26–35. COSO. Internal control—an integrated framework. The Committee of Sponsoring Organizations of the Treadway Commission; 1992. COSO. Enterprise risk management—integrated framework. The Committee of Sponsoring Organizations of the Treadway Commission; 2004. http://www.coso.org. Fedorowicz J, Gelinas Jr UJ. Adoption and usage patterns of CobiT: results from a survey of CobiT purchasers. IS Audit Control J 1998;VI:45–51. Hoffman T. IT auditors seek Sarb-Ox guidance. Computerworld 2004;38(15) April 12. IT Governance Institute. Governance, control and audit for information technology. CobiT 3rd edition. Rolling Meadows, IL: IT Governance Institute; 2000. IT Governance Institute. “Control objectives, management guidelines, maturity models” in CobiT 4.0. Rolling Meadows, IL: IT Governance Institute; 2005. KPMG. S-O rules finalized. Frontiers in finance for decision makers in financial services; 2003. 10–15. November. Lainhart IV JW. An IT assurance framework for the future. Ohio CPA J 2001 19–23. January–March. Melville N, Kraemer K, Gurbaxani V. Review: information technology and organizational performance: an integrative model of IT business value. MIS Quarterly 2004;28:283–322 June. Merriam-Webster Online Dictionary; 2005. http://www.m-w.com/cgi-bin/dictionary?book=Dictionary&va=framework. Messier Jr WF, Glover SM, Prawitt DF. Auditing & assurance services: a systematic approach. 5th edition. New York, NY: McGraw-Hill Irwin; 2008.

B. Tuttle, S.D. Vandervelde / International Journal of Accounting Information Systems 8 (2007) 240–263

263

Netegrity. Sarbanes–Oxley. Regulatory compliance handbook; 2004. http://www.netegrity.com/PDFS/REGULATORY/ SOA%20Handbook%20Sheet.PDF. Public Company Accounting Oversight Board. An audit of internal control over financial reporting performed in conjunction with an audit of financial statements; 2004. http://www.pcaobus.org/documents/rules_of_the_board/ Standards%20-%20AS2.pdf. Ramos M. Evaluate the control environment. J Account 2004;197:75–8 May. Ridley G, Young J, Carroll P. COBIT and its utilization: a framework from the literature. Proceedings of the 37th Hawaii International Conference on System Sciences; 2004. Summerfield B. EU selects CobiT as an auditing standard; 2005. http://www.certmag.com.