Supporting a complex audit judgment task: An expert network approach

EUROPEAN JOURNAL OF OPERATIONAL RESEARCH ELSEVIER

European Journal of Operational Research 103 (1997) 350-372

Supporting a complex audit judgment task: An expert network approach J e f f e r s o n T . D a v i s a, A n n e P . M a s s e y

b,,, Ronald E.R. Lovell II a

a Department of Accounting, Management Information Systems and Marketing. Clarkson University, Potsdam, NY 13699-5795, USA b Department of Accounting and lnfi)rmation Systems, School of Business, Indiana University, 1309E. lOth Street, Bloomington, IN 47405, USA

Abstract

An auditor considers a tremendous amount of data when assessing the risk that the internal control (IC) structure of an entity will fail to prevent or detect significant misstatements in financial statements. The myriad of relationships between 1C variables that must be identified, selected, and analyzed often makes assessing control risk a difficult task. While some general procedures and guidelines are provided, audit standards dictate no specifically set procedures and rules for making a preliminary control risk asscssmcnt (CRA). Rather, the proccdures and rules are left mostly to auditor judgment. This paper considers the appropriateness of applying artificial intelligence (A1) techniques to support this audit judgment task. It details the construction of a prototype expert network; an integration of an expert system (ES) and a neural network (NN). The rules contained in the ES model basic CRA heuristics, thus allowing for efficient use of well-known control variable relationships. The NN provides a way to recognize patterns in the large number of control variable inter-relationships that even experienced auditors cannot express as a logical set of specific rules. The NN was trained using actual case decisions of practicing auditors. © 1997 Elsevier Science B.V. Keywords: Expert network; Neural network; Expert system; Control risk assessment; Auditing

1. Introduction

Many decision-making tasks do not lend themselves to formulation through the sole use of quantitative models, nor simple intuitive problem solving (Rosenhead, 1992). Building and incorporating qualitative and quantitative reasoning and modeling into a decision-aiding system is a challenge for practitioners and researchers (Gupta, 1994; Silverman, 1995; I.iberatore and Stylianou, 1993, 1995). As the num-

"Corresponding author. E-mail: [email protected]. edu.

ber of variables in a problem increase and the specificity and measurability of their efficacy relationships diminishes, difficulties are exacerbated. The literature suggests that builders of systems to support complex decision-making should draw from and attempt to combine a multitude of paradigms (for example, decision support, neural networks, casebased reasoning, and models with rules or objects) (Silverman, 1995; Beulens and Van Nunen, 1988; Turban and Watkins, 1986; Yoon et al., 1993). This paper analyses the process of building a prototype intelligence-based system for application to a complex problem within the field of auditing. The prototype system is constructed as a decision-support tool for auditors analyzing the internal control (IC) struc-

0377-2217/97/$17.00 © 1997 Elsevier Science B.V. All rights reserved. PII S0377-221 7(97)001 25-2

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

ture of a business entity in order to derive a preliminary control risk assessment (CRA). It is designed as an expert network that combines two AI paradigms - expert systems (ESs) and neural networks (NNs). A review of the literature suggests that AI technologies such as ESs and NNs have found only limited use as decision aids in audit practice. Moreover, developers of such systems have generally relied on singular means of knowledge representation and reasoning. Intelligence-based systems in auditing have been primarily employed as traditional knowledge-based systems (O'Leary and Watkins, 1991; Gray et al., 1991; Eining et al., 1994; Smith, 1996), traditional statistical systems (Bell et al., 1990; Eining et al., 1994; Scott and Wallace, 1994), and as stand-alone neural network systems (Bell et al., 1990; Coakley and Brown, 1993). This paper forms a part of a wider study into the increasing use of AI technologies in audit practice. As for any system, effectiveness and efficiency gains are not automatic or achievable without users accepting the system as a decision aiding tool. Thus, system design, development and implementation are key factors in promoting acceptance (Gillett et al., 1995). Although design and development include, for example, traditional system aspects such as the design of the user interface, etc., it may be that the characteristics of the underlying system technologies - here ESs and NNs - also have a major impact on user acceptance (Davis, 1994; Eining et al., 1994). More specifically, users such as auditors may be leery of technology that appears to take the decision out of their hands. However, acceptance and use may be enhanced as users gain a better understanding of the characteristics, benefits and limitations of these intelligence-based systems, and the role of the technology as a decision aid as opposed to a decision maker. Thus, a motivation for the development of the prototype system is to enable future research designed to explore practitioners' acceptance (or rejcction) of such systems. However, the objective of the research presented in this paper is narrower in scope. More specifically, this paper focuses on: (1)demonstrating the appropriateness and applicability of an expert network within audit practice; and (2) describing the methodology employed to design and develop the prototype system.

351

This paper, therefore, contributes to the literature on the practical application of AI-based systems to audit judgment tasks in at least two ways: (1) consideration of the application of hybrid intelligent computing techniques to an area of auditing that is at present making almost no use of them, but where there is developing interest and a potential for benefits to be accrued from their use; and (2) development of an integrated ES and NN prototype system, the CRA Expert Network, constructed for a PC-based environment using tools that lead to a PC-based product appropriate for field experimentation. The remainder of the paper is organized as follows. Section 2 describes the nature of the audit task, providing the background to the selection of an expert network approach. Section 2 also describes the principles behind the integration of ESs and NNs and introduces the prototype CRA Expert Network. Section 3 details the design and construction of the prototype system, including a description of an experiment conducted with practicing auditors used to obtain judgment data for training the NN. Section 4 contains brief concluding comments.

2. O v e r v i e w

of domain and system architecture

When a public accounting firm audits a business entity, the potential exists that the auditors may not discover material misstatements in the entity's financial statements. The likelihood of not discovering a significant misstatement is called audit risk. During the audit process, the business entity's internal control (IC) structure is evaluated to determine the nature, timing and extent of audit tests to be performed that will be most effective in reducing audit risk. The purpose of the IC structure is to prevent a n d / o r detect erroneous, fraudulent, or missing accounting transactions. Consequently, a complete and proper assessment of this structure is critical to a successful audit. The attempt to build intelligence-based systems to support audit judgment tasks, such as control risk assessment (CRA), is dependent on the acquisition and representation of knowledge and heuristics gained through audit experience (Deng, 1994). General audit theory and heuristics may be captured and represented using logical constructs. But, obtaining

352

J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372

knowledge from auditors for making small yet important distinctions between situations in the form of specific rules may be nearly impossible. That is, assessing the IC structure potentially requires evaluating hundreds of variables with thousands of possible inter-relationships. It is not likely done by a serial, step-by-step reasoning process, but rather by recognizing patterns in a given situation and reacting appropriately based on experience (Dreyfus and Dreyfus, 1986; Anderson, 1983). Evaluating these many complex relationships is a difficult task even for the most experienced auditors - articulating them is essentially impossible. However, neural network systems can be used to automate judgment tasks that require this pattern recognition. Thus, the innate complexities of the audit judgment task suggest that an appropriate approach to system design is to integrate sub-systems of AI techniques each of which address distinct aspects of domain knowledge and reasoning (Lymer and Richards, 1995). Given the characteristics of the CRA task, the prototype system is constructed as an expert network reflecting the integration of an expert system (ES) and neural network (NN). Briefly, the ES incorporates general audit theory and well-known control relationships using a logical set of explicit rules. The NN in the CRA Expert Network is used to recognize and establish patterns among the large number of inter-related variables inherent in the task. When exercised, the ES provides the user interface and evaluates the complexity and basic structure of an entity's internal controls. If the ES determines that the IC structure is sufficiently complex to warrant further analysis, the data collected during interaction with the ES is fed as input into the NN. The NN is stimulated by the input data and provides a preliminary CRA. The NN evaluates approximately two hundred variables with thousands of potential inter-relationships used to make a preliminary CRA. The following section examines the principles behind expert networks such as the CRA Expert Network.

2.1. Expert networks: An integrated approach An expert network is one form of integrated systems designed to address limitations of using a

single representation and reasoning approach (Lymer and Richards, 1995; Frisch and Cohn, 1991). ESs based on symbolic computation - are best at modeling structured problem domains (or aspects thereof) that conform well to logical constructs. Conversely, NNs - based on numerical computation - can successfully model problems that do not conform well to explicit logical constructs. It is when these situations cross, such as in the CRA application, that the combined capabilities of a blended solution should be considered (Caudill, 1991; Medsker and Bailey, 1992; Medsker, 1994). By combining the deductive reasoning approach of an ES with the inductive approach of a NN, difficult and somewhat unstructured tasks may be performed. Integration takes advantage of the strengths of each type of system, while mitigating the inherent weaknesses of each when used alone. Let us briefly examine the subsystems of an expert network.

2.1.1. Components of expert networks ,sognitive models, focused on representations of knowledge, suggest that human problem solvers organize knowledge through the use of propositions. A proposition, in the form of an if-then rule, is the atomic building block of rule-based expert systems. Domain knowledge - generally elicited from a single or multiple subject matter experts - is codified into a series of linearly executed discrete rules (Solso, 1988; Greeno, 1973; Johnson-Laird, 1983; Minsky, 1986). However, significant development barriers arise because experts often cannot articulate knowledge and complex relationships as discrete rules. Only when asked do they produce a justification for the judgments made. Even then, the justification is a generalization that more than likely has a fair number of exceptions (Deng, ! 994). Propositional logic is rarely a sufficient means to represent complex reasoning (Kunz et al., 1987; Jackson, 1990). Rather, ESs are a particularly good approach for closed-system applications that have literal and precise inputs that lead to logical outputs (MacLennan, 1993). However, they are relatively inflexible since performance degrades sharply when they are applied to problems outside their original scope (Jackson, 1990; Kunz et al., 1987; Edwards and Connell, 1989, p. 25). Moreover, changes in domain knowledge structure and content often re-

353

J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372

A N N is a statistical modeling technique. However, NNs can make less stringent assumptions conceming independence of variables and the shapes of underlying data distributions than other statistical techniques, e.g., regression or multiple discriminant analysis (Rumelhart and McClelland, 1986; Lippmann, 1987; Lacher et al., 1995). Rich discussions of the statistical aspects of NNs and their relation to more traditional statistical models may be found in Ripley (1993, 1994), Cheng and Titterington (1994) and Sarle (1994). While NN systems are flexible in terms of fault tolerance to missing or noisy data, they do not have some basic characteristics for flexible precise commonsense reasoning, e.g., symbolic processing or interpretation of internally stored knowledge (Sun, 1994, p. 247). Unlike ESs, NNs have no inherent explanatory function 'module'. This has hindered acceptance in practice as it is not clear to a non-technical user how the network derived a given conclusion. However, research is being conducted to extract comprehensible symbolic representations from trained NNs (e.g., Craven and Shavlik, 1996).

quire substantial system modifications to continue or enhance system viability. Conversely, NNs can analyze large numbers of inter-related variables to establish patterns and characteristics in situations where precise, discrete rules are not known or discernible (MacLennan, 1993). Rather than depending on explicit knowledge as expressed by a domain expert, NNs model the implicit relationships in exemplar domain data. The continuous nature of the stimulus/response approach allows for efficient modeling of complex tasks. Simply put, a neural network discovers its own numeric, continuous rules as it learns through the examples it is provided. NN systems may be able to perform certain types or parts of audit judgment tasks that are difficult and perhaps inappropriate for the capabilities of other types of intelligent systems. An artificial NN consists of processing elements linked via weighted uni-directional signal channels called connections to form a distributed, parallel processing architecture (Rumelhart and McClelland, 1986; Hecht-Nielsen, 1990). Each processing element can possess local memory and carry out localized information processing operations. The processing element or neuron is the atomic building block of the NN (Fig. 1). NN paradigms differ in how the processing elements are connected and the manner in which the weights are updated (Markham and Ragsdale, 1995; Rumelhart and McClelland, 1986).

2.1.2. Relationship o f the CRA expert network to existing systems

A number of hybrid ES and NN systems have been described in the literature. Kandel and Langholz (1992), Gallant (1993), Medsker (1995), and Sun

Xo= 1

(Bias)

Neuron or Processing Element Inputs

XI

XL

(Outputs from other neurons or processing nodes)

Summation Function

] !

Transfer Function e.g. Sigmoid

Fig. I. Neural network processingelement structure.

1

Neuron or Processing Element Output

354

J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372

and Bookman (1995) have compiled research concerning the integration of symbolic and numeric systems. These compilations include descriptions of expert networks applied to application domains such as natural language, signal processing, biology and medicine, management, and engineering (see particularly Medsker, 1995, pp. 3 9 - 5 6 ) . The relationship between the ES and the NN in the CRA Expert Network is structured similarly to systems reported by Lin and Hendler (1995), Maclntyre et al. (1995), and Bataineh et al. (1996). Lin and Hendler (1995) use a NN to classify ballistic signals and the output is passed on to the ES for further processing and interpretation. Using the same neural network software employed by the CRA Expert Network, Maclntyre et al. (1995) and Bataineh et al. (1996) present expert networks for application within the electric utilities industry. The ESs in these applications also provide processing inputs to the NNs. Thus, while expert networks (and other forms o f integrated systems) are not an explicitly new technique, we have found no evidence o f the technique being applied to audit judgment tasks such as CRA. However, given the characteristics of the task, there is significant scope for consideration of their applicability to this domain. Furthermore, the ability to construct such systems using PC-based tools should facilitate experimentation with them in the field. The following sections detail the design and construction of the CRA Expert Network.

the user interface and conducts preprocessing of data that is fed into the NN. After the NN performs its pattern-matching activities, output is passed back to the ES for display. Advantages of loosely coupling the components of the prototype system are twofold: (1) system development is amenable to commercially available software; t and (2) maintenance is reduced due to the simplicity o f the data file interface approach. The disadvantage of a great deal of redundancy that usually accompanies loosely coupled system has been largely avoided in the C R A Expert Network, perhaps due the nature of the problem more than any other reason. Before proceeding to an in-depth discussion of the structure of the C R A Expert Network, the following two sections describe the knowledge - and sources of that knowledge reflected in the ES and NN components, respectively.

3.1. Expert system development

As introduced earlier, the prototype CRA Expert Network was constructed by integrating an ES and

The rules for the ES were primarily derived from structured logic and questions found in Grant Thomton's internal control documentation and evaluation software - Information and Control Understanding System (lnfocus) (Grant Thornton, 1992), which is used in audit and consulting practice. Infocus does not make any risk assessment itself and is not an AI decision tool. However, it provides general relationships among IC variables to assist the auditor in structuring the preliminary C R A process. The knowledge and logic of Infocus were formalized in a logical rule structure. The interface provided by the ES allows the auditor to document the IC data of a given client. It should be noted, however, that the user interface screens that are provided to an auditor

NN. The ES and NN components respectively address two types of knowledge and logic relevant to auditors: (1) well-known audit relationships - approximately 20% of the distinct internal control structure variables - encoded as logical rules in the knowledge-base; and (2) more complex control t;ariable relationships determined by the 'trained' NN. The CRA Expert Network uses a loosely coupled model of an integrated system in which the CRA process is decomposed into separate ES and NN components that communicate via data files (Medsker and Bailey, 1992; Medsker, 1994). The ES provides

The ES component of the CRA Expert Network was developed using Microsoft Visual Basic 3.0. This tool provided the means to: (a) develop an object-oriented GUI user interface; (b) encode the well-known audit relationships in a rule structure that logically drives data collection; and (c) directly link to a Microsoft Access 2.0 database. The data collected as an auditor interfaces with the ES component is stored in an Access 2.0 database file. If warranted, an ASCII text file of this collected data is fed to the NN as input. The NN, stimulated by these input variables, derives a CRA. The NN was created in NeuralWorks Professional I1/Plus (NeuralWorks, 1992).

3. C o n t r o l r i s k a s s e s s m e n t (CRA) e x p e r t n e t w o r k

J.T. Daois et aL/ EuropeanJournal of OperationalResearch 103 (1997)350-372 during a given session are controlled by the logic structure encoded in the ES - that is, data is neither requested of a user or screens provided that are not relevant to a given client. Use of Infocus' logic and question was necessary because lnfocus was an integral part of the knowledge acquisition process conducted with audit seniors - whose knowledge was ultimately used to train the NN. In addition, lnfocus also provides a real world, accepted and tested basis for the majority of the ES logic. Added to the knowledge-base - that is not from Infocus' logic and questions - are CRA threshold rules and rules translating the preliminary CRA output of the NN to a CRA category. The CRA threshold rules are used to control the current assessment of control risk while a session is in progress. For example, depending on the earliest data collected by the ES component and the current threshold level setting, the ES may determine that further processing is not warranted, i.e., the data collection by the ES is halted and the NN is bypassed. This process is detailed in a following section.

3.2. Neural network development 3.2.1. Data source for training and testing The data used to train the NN was from an experiment conducted with 64 senior auditors from Grant Thornton (see Davis, 1996). These subjects averaged 4.5 years of audit experience and bad completed a CRA an average of 37 times. Each auditor was given one of three client cases. The first represented an actual small sized client of the firm. The other two were based on example cases found in the SAS No. 55 Audit Guide (AICPA, 1989). All three cases involved merchandising entities. The preliminary CRA was restricted to the sales stream within the revenue cycle of each entity. The financial statement assertion group consisted of completeness, existence/occurrence, and valuation core assertions related to the IC system goal of preventing and detecting errors. The three cases were chosen because the entities had computerized accounting systems and reflected situations in which the auditor could potentially rely on all three types of internal controls - manual controls, programmed controls, and segregation of

355

duties. 2 However, the cases represented different situations in terms of firm size, complexity of control system, and strength of controls. The first case represented an actual small-sized client, with low control system complexity, yet a fairly strong IC structure. This case was also compatible with a purely substantive audit approach. 3 The two cases from the audit guide represented larger entities than the first case. The first, a closely held company, did not have as strong of an IC structure as the second, a larger public corporation that had a very strong IC structure. While both had computerized accounting systems, the public corporation had a more complex and well-controlled computer system. Very broadly, while conducting a preliminary CRA, an auditor raises and reviews questions regarding the ICs (e.g., manual, computerized) of an entity that are designed to ensure that significant misstatements on financial statements will be prevented or detected. However, the questions that an auditor chooses to examine are left primarily to the auditor's judgment. Thus, the purpose of working with the 64 auditors was to determine not only what their CRA was for a specific case, but also what set of questions each auditor was using during the CRA process.

3.2.1.1. Collection of CRA process data. Using lnfocus, each auditor began their CRA by selecting variables a n d / o r addressing specific questions from a set of potential judgment variables and questions related to the revenue cycle IC environment, EDP environment, and accounting controls. Values assigned to selected variables and the responses to chosen questions constituted each auditor's 'cue set'

2 Segregation of duties refers to the principle that an individual responsible for the conduct of a process cannot also be responsible for the controls associated with that process. For example, if an individual is responsible for receiving checks from customers, they should not be also responsiblefor recording those payments in the accountingsystem. 3 Essentiallythere are two audit approaches. In the substantive approach, the auditor elects to ignore the controls in place and focuses their analysis directly on the numbers represented in the accounting statements. Conversely, in the control approach the auditor focuses first on an analysis of the IC structure of the client in an effort to reduce the amount of substantive testing that will follow.

356

J.T. Davis et al./European Journal of Operational Research 103 (1997) 350-372

issues deemed as relevant to the specific case and, ultimately, assessment of control risk. The IC potential variables/questions, and the possible values that each may be assigned, are presented in Appendix A. The appendix reflects the complexities and relationships of the questions and variables that are potentially considered by an auditor (including the 64 auditors that participated in the experiment) during the CRA task. Appendix A indicates the percentage of the auditors, by case, that deemed each variable/question as relevant. IC environmental features, EDP environmental features, accounting controls, and segregation of duties operate as inter-dependent parts of the IC structure. While these variables were included in this experiment and addressed by the auditors in various fashions, the complex inter-dependencies between the variables makes it virtually impossible for an auditor to explain the structure of their cue set, i.e. the relationships and strengths of relationships between items in the cue set, and the impact of this structure on the CRA. These inter-dependencies serve to illustrate the complexity of the domain and the task facing an auditor when conducting a CRA. This level of complexity also highlights the role that the NN will play in structuring these relationships. In addition to identifying their respective cue set, each auditor recorded: (1) whether s / h e planned to rely on manual controls, programmed controls, a n d / o r segregation of duties - indicating whether more tests of controls as opposed to more substantive tests would be used to reach an acceptable level of audit risk; (2) the existing controls selected as key controls, i.e., controls the auditor intended to rely on and test to determine if the control is operating as it should; and (3) any controls that s / h e felt were missing. -

based on the numeric scale. The point estimate corresponds to the categorical judgment responses as follows:

Risk Category

Point Estimate Interval

Limited (LTD) Moderate (MOD) Slightly below maximum (SBM)

0-25 2 6 - 50 51-75

Maximum (MAX)

76-100

Each auditor's selected cue set and corresponding derived CRA point estimate 4 make up one observation. Although there were only three cases, the observations were distinct as each auditor indicated a different cue set for making their preliminary CRA. For example, it is possible for an auditor to rely on programmed controls and choose a CRA of MOD. Conversely, another auditor (examining the same case) could choose to rely on manual controls and make the same preliminary CRA. The 64 observations were used to develop and test the NN model. The network training (within sample) and testing (out-of-sample or hold-out) data sets each contained 32 observations. Once trained, using the 32 within sample observations, the NN model serves as a proxy for the typical knowledge structure used by the experienced senior auditors in the training set. The 32 observations in the hold-out sample were used to test the resulting NN model.

3.2.2. Training and validation of NN The CRA Expert Network employs a feedforward classification NN that was trained using the backpropagation learning algorithm (Rumelhart and Mc-

3.2.1.2. Collection of CRA output data. Following analysis of the case, each auditor recorded their preliminary CRA in Infocus. lnfocus provides a choice of four risk categories: maximum; slightly below maximum; moderate; and, limited. These risk categories are consistent with SAS No. 55 Audit Guide. In addition to the categorical response, each auditor recorded their CRA using a point estimate (0 to 100 scale) which provides an indication of how close to the category border their judgment would be

4 While it m a y seem tvdd to use a point estimate here, the motivation is rather simple. It allows for the use of a t-test to c o m p a r e the N N ' s C R A (a value between 0 and 1) to that of the auditors, rather than relying solely on classification a c c u r a c y . A classification NN does not yield a c c u r a c y in relation to a t-test just classification a c c u r a c y . Furthermore, this a p p r o a c h allows for the NN response to be m a p p e d to the C R A risk categories that are inherent in thc experimental task.

J.T. Davis et a l . / European Journal o f Operational Research 103 (1997) 350-372

Clelland, 1986; Tam and Kiang, 1992). A feedforward NN using sigmoidal activation functions is mathematically capable of any continuous function and thus is applicable to a large variety of knowledge intensive tasks by distributing knowledge encoded into link weights that is learned from data examples (Hornick et al., 1989). In the network the processing element (refer to Fig. 1) takes inputs from other neurons and sums these inputs, X 0 . . . X,, using a summation function, L i = EN_oWI,jXj. The transfer function 'normalizes' the input summation by vectoring it into a predetermined range. The activation level of the processing element is determined by an activation function: X i = 1/(1 + e-~'). The backpropagation learning algorithm was used to find the functional relationship between the inputs (judgment cues/variables) and the target outputs (the auditors preliminary CRAs). A variety of network configurations were tested during the design phase. Networks with a larger number of middle nodes learned the training data sample quite well, but performed poorly on the test data sample. Networks

357

with a smaller number of middle nodes were too general and performed poorly overall. The final NN architecture has 210 input nodes (control cues/variables), 30 hidden (middle) layer nodes, and one output node for the preliminary CRA using the auditors point estimates within their chosen CRA category. Delta rule summations were employed with sigmoid transfer functions on the middle and output nodes, while the standard root mean square error (RMSE) function was used for "all network layers. Training stopped at 1600 iterations using the early stopping technique to avoid overfitting of the training sample - the RMSE of the sample began to increase instead of decrease at that point. In addition, the NN was analyzed after training at 4800 iterations. While accuracy - both RMSE and category accuracy - on the training sample was better than at 1600 iterations, accuracy on the hold-out sample and overall accuracy on the total sample declined. The final trained network (at 1600 iterations) had a Pearson correlation coefficient of 0.869, and a CRA category accuracy rate of 72% for the training

Network Accuracy - Training Data "E

0.9

0.7 0.5 '~

0.3 0.1

o

-0.1

O

-0.3

o

iii,iiiiiil

¢

Desired Output I

.._~.... Network Output I

! ,t

Difference

J

Observations

Network Accuracy - Testing Data 1 0.8

:~:~i~i~ii~

~

r_.4k_. Desired Output ,¢

[ ~ 0.2 ;~!~:i~i~!~:.~?~:~i~:N:~! ~: i~.:~4i~: %::~ ! ~ ~i:!-~i!~!i~::~~-:: ~ ~......... 0

O

o

Obs e rvations Fig. 2. Network validation.

!~:~:;

,¢

Netw ork Output Difference

358

J.T. Davis et al./European Journal o f Operational Research 103 (1997) 350-372

~.

[

Environment ~ Questions

General

.... 1

.

.

.

.

Compu,er Processing Overview (CPO)

L ~ ~ r////~,

I General Control _1 Questions ]Computer Controls~ | (Gce)

Phase

t ounting A Controls

1

Phase 2

~.

211 Vmmbles

1 2 3 .... Neural ,

210

i 1 2 .... -A ....

30

Input Layer Phase Middle Layer

3

LJ I

1

Output

Fig. 3. Expert network: Knowledge base structure with embedded neural network.

data set. Testing using the hold-out sample resulted in a Pearson correlation coefficient of 0.695, with a category accuracy rate of 78%. 5 The risk category prediction errors for the test sample included: four observations that were one category higher than the assessment by the auditor: one observation each that were one category lower than the auditor, two categories higher than the auditor, and two categories

The correlation coefficient is a measure of relationship between paired observations in two data sets - here, the relationship between the auditor's point estimate and the model's point estimate for each observation. Category accuracy rates were derived by comparing the NN point estimate output - based on an auditor's input cue set - to the auditor's chosen risk category. For comparison, error rates for NN models built for financial distress applications (Bell et al., 1990; Tam and Kiang, 1992) were in the range of 10-23%. However, it is important to note that the models in those studies were much simpler than the NN developed in this study. In those studies there were less than 10 input variables with two response levels for output. Predicting a four category response with 210 input nodes is a significantly more difficult problem.

lower than the auditor. Fig. 2 presents the point estimate network accuracy in relation to both the training and testing samples. A paired t-test (see Koopmans, 1987, p. 325) was also • run on the predicted and actual network output. No statistical difference was found for the training sample (mean difference = - 0 . 0 0 0 3 , one-tail p value = 0.48) or the testing sample (mean difference = 0.0239, one-tail p-value = 0.24). In addition, two classification networks were developed - the first employed backpropagation and the second a radial basis function - using only the C R A classification categories and not the auditors' point estimates. The hold-out sample classification accuracy for both these models was approximately 49%. Conversely, as described above, the point estimates for the trained network in the CRA Expert Network were within the classification ranges chosen by the auditors 78% of the time. Clearly, for this data, using a point estimate within C R A category provided superior model precision.

J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372

(a) ! !

Ask Control Environment Questions

PHASE1

Yes

No

Yes

Set CRT = MAX

Set CRT >= Slightly Below Maximum

1 --No--

I SetManu,,Cootro,---) Neural Network Inputs = 0

I I

yes I

Set CRT >= Limited

CRA = Current Threshold

I

Yes Computer Processing Overview (CPO)

I Set CPO & GCQ---~ Neural Network Inputs = 0

)

(

Fig. 4. (a) Expert network processes flowchart; (b) Expert network flowchart.

359

360

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

(b)

General Controls Questionnaire (GCQ)

PHASE1 No Set GCQ Neural Network Inputs = 0

No

l Rely on Manual Controls (MC)

Rely on Manual Controls

-'--!

t Y~--

(MC) . S

-- Cu

i l

ent

I Yes I

. ~

~Accounting Controls _bJ Questionnaire: -I Set PC Questions = 0 Ask MC Questions

Accounting Controls Questionnaire: Set MC Questions = 0 Ask PC Questions

PHASE2

Accounting Controls Questionnaire: Ask MC & PC Questions

_ _ _ T . . _ _ _ _

PHASE3

General Environment Variable Values

Fig. 4 (continued).

3.3. CRA Expert Network Design

The ES and trained NN constitute the CRA Expert Network. The system employs a three phased ap-

• •

cessing overview, and general control questions (computer-related); Phase 2. Accounting controls phase; and, Phase 3. The Neural Network phase.

proach to determine a preliminary CRA as follows: •

Phase I. Environment phase that encompasses

the general environment questions, computer pro-

Fig. 3 illustrates the relationships among these three system phases. Fig. 4 provides a flowchart

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

overview of the main logic contained within the CRA Expert Network and its three phases. Phases 1 and 2 address the logical data collection aspect of the CRA Expert Network by incorporating and presenting to the user the judgment variables/ questions found in Appendix A. As noted previously, the data collected during these phases is stored in a database file, which is fed to the NN. The NN (during Phase 3) accepts and utilizes values for the following (as detailed in Appendix A): (1) the Revenue Cycle Environment variables; (2) the Planned Reliance variables; (3) responses to the Computer Processing Overview (CPO) questions; (4) responses to the General Control Questions (GCO, and; (5) all the Accounting Controls variables. Values accepted by the NN are: a ' 1 ' for a Yes response to a question or the indication that an Accounting Control variable is a key control; a ' - 1 ' for a No response to a question or that a specific Accounting Control variable is missing, and; a ' 0 ' indicating that the question/variable is not applicable to the CRA. As Fig. 3 indicates, there exist 210 values passed to the NN each constituting a separate node in the input layer of the NN. However, it should be emphasized here that not all questions/variables are applicable to all cases. Data is collected in Phases 1 and 2 via user interface screens - the presentation of which is controlled by the logic structure encoded in the ES. This logic structure is illustrated in Fig. 4. The ES does not continue along a data collection path that is not relevant. For example, if the answer by the auditor to a particular CPO question is ' N o ' , then the related GCQ questions are not applicable to that case. The ES then bypasses the related GCQ questions - the user never sees those screens - and assigns '0s' (which are passed to the NN), representing not applicable, to those GCQ questions (see Fig. 4(b)). The following sections present specific details concerning each phase.

3.3.1. Phase 1: Environment phase During the environment phase, information necessary for the ES to apply broad internal control audit heuristics is collected. These heuristics should be well known to most auditors. However, the possibility exists that one or more of these heuristics could be omitted or misapplied. Thus, including these

361

heuristics as rules in the ES should improve an auditor's control variable selection consistency as well as inter-auditor consistency. The questions are designed to collect information concerning the control environment and computer environment (if any) in which the accounting controls, addressed in Phase 2, must operate. This information is used by the ES, to make (if possible) a preliminary CRA without proceeding to Phases 2 and 3. In addition, the information is used to set a control risk threshold (CRT). The purpose of the CRT is to prevent a final CRA that is below a predetermined level, given that certain conditions exist in the client environment. Broadly, during Phase 1 (see the flowchart in Fig. 4 and the specific questions found in Appendix A), the auditor is asked if s / h e plans to rely on the control environment. If not, i.e., the auditor believes that the control environment is not reliable or testing the controls is inefficient, the ES sets the CRA to maximum (MAX) and the task is completed. This indicates that the auditor will conduct a purely substantive audit approach. Conversely, a yes response indicates that a control audit will be conducted and the auditor is then asked if s / h e plans to rely on segregation of duties. If so, reducing the control risk threshold (CRT) to slightly below maximum (SBM) is justified; if not, the CRT is set to maximum (MAX). In either case, the auditor is asked if they plan to rely on manual controls (MC). If s / h e does, the CRT is reduced to limited (LTD), and the system is triggered to obtain information concerning MC during Phase 2. If no reliance is planned, the CRT remains unchanged and the MC questions are omitted from Phase 2. The control variables related to the omitted questions will be input to the NN as '0s' (i.e., not applicable). The next set of questions concerns whether any part of the accounting system is automated. If there is no planned reliance on MC and none of the accounting system is automated, the CRA is set to the current CRT and the task is complete. If there is automation, the Computer Processing Overview (CPO) will be completed, thus gathering information about the general computer environment and application complexity. The responses to these questions determine thrce things: (1) the level of computer environment risk; (2) the level of computer applica-

362

J.T. Davis et aL / European Journal of Operational Research 103 (1997) 350-372

tion complexity, and; (3) the relevant questions to be included in the General Controls Questionnaire (GCQ). At this point the auditor may decide not to rely on p r o g r a m m e d controls. If this is the case, no GCQ questions are asked and all the GCQ inputs to the NN will be set to 0. In the event that manual and programmed controls are not relied on, the system sets the C R A to the current value of the CRT and exits. Conversely, if the auditor is relying solely on manual controls, the system sets the program control NN input variables to 0, and proceeds to Phase 2 where it asks the manual controls questions. If the auditor elects to rely on program controls, the GCQ questions are presented for completion. Finally, after completing the General Controls Questionnaire (GCQ), the auditor (usually consulting with the computer audit specialist) will be given another chance to reject reliance on programmed controls. If the auditor decides that computer controis are too weak to support reliance on programmed accounting controls, all the GCQ inputs to the NN will be set to 0. The system then moves on to Phase 2 for the manual accounting controls information, if the auditor has previously indicated reliance on manual accounting controls. Once again, in the event that manual and programmed controls are not relied on, the system uses the current CRT to complete the CRA task. 3.3.2. Phase 2: Accounting controls" phase The accounting controls phase includes the completion of the manual accounting control questions, the programmed control questions, or both depending on the decision path taken during Phase 1. If either the manual controls or the programmed were considered not applicable in Phase 1, the ES sets the corresponding NN input values to 0 and does not present the user with any questions related to these controls. If either or both are considered relevant, the appropriate Accounting Controls Questionnaire is presented to the user. The questions for Phase 2 relate to potential accounting controls (see Appendix A). The auditor responds to each question by labeling each individual potential control as a key control (coded as a ' 1 ' for the neural network); missing control ( - l); or not applicable (0). Upon completion, the CRA Expert Network proceeds to Phase 3.

3.3.3. Phase 3: Neural network Assuming the auditor chooses to rely on manual accounting controls, programmed accounting controis, or both, the responses collected from Phases 1 and 2 are fed into the NN (Fig. 4). Each question from the ES is mapped to one of the 210 NN input nodes. Given this input, the NN derives a preliminary CRA - an evaluation among 0 and 1 - which is presented to the user by the CRA Expert Network. In addition to presenting the numerical evaluation, the ES transforms the response to the CRA risk category. For example, if the response was 0.52, the risk category would be slightly below maximum (SBM), although it is close to the S B M / M O D category border. The point estimate from the NN does provide information to the auditor as to how close the NN response is to a risk category border. Whether this information ultimately contributes to an auditor's Judgment a n d / o r leads to better system acceptance are empirical questions. 3.4. Directions f o r future development and research

The prototype system offers the opportunity for experimentation with practicing auditors, and provides a basis from which standards may be established for a task in which no specifically set procedures and rules are currently delineated. Future developments will center on addressing current limitations in the prototype CRA Expert Network. The first limitation relates the knowledge acquired and represented in the system. More specifically, the system is based on a limited number of cases and the fact that the auditor subjects were from the same firm. However, a sizable set of cases and additional auditors could be obtained from audits already completed a n d / o r by providing auditors with additional case scenarios. A related issue is whether the NN model using point estimates within categories provides a more precise decision-aid than a pure classification model. For this data, the extra information as to how close to a category border the auditors' judgments were, seemed to give the point estimate model an advantage over the pure classification models. Of course, using a point estimate within category assumes more judgment precision on the part of the auditors. There may be a trade-off between judgment precision and

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

generalizability between the point estimate within the categories approach and the purely classification approach. This empirical question should be tested in future research particularly with regard to acceptability by auditors. Second, most auditors are not computer audit specialists. When the computer processing overview (CPO) indicates a complex system, a computer audit specialist should be consulted. The computer audit specialist assists the auditor in deciding whether the computer controls are adequate to support reliance on accounting programmed controls. Thus, efforts are underway to capture and represent in the system this specialized knowledge. Third, the system is currently narrow in scope. In addressing the risk assessment task, only the sales stream within the revenue cycle is included. However, the sales stream is general to nearly all businesses and important in most all audit situations. In order to increase the usefulness of the system, a set of integrated systems each designed to deal with particular parts of all the accounting cycles should be developed. Additionally, the system does not include assignment of accounting control variables to particular accounting processes or control objectives. The NN only considers whether a particular control is a key control and should be relied on, is a missing control, or is not applicable. However, the same control may apply to more than one accounting process. The system does not take into consideration these compensating controls - controls that are in operation later in the accounting process stream that may strengthen the overall effectiveness of the IC system. Further design efforts will involve including this assignment of controls to processes and accounting objectives. Efforts in this vein may take the form of additional rules in the knowledge-base, more neural network variables (and, perhaps even a different NN architecture), and more than three levels or values that a particular control variable could be given. Currently, research efforts with regard to the CRA Expert Network involve two main areas: (1) establishment of experiments with auditors in the field, and (2) design and development efforts focused on enhancements to the user interface and the construction of an explanation module. While the NN provides the capability to model complex relationships

363

and derive a conclusion, it does not provide any justification for the conclusion. A major concem for the auditor is being able to document audit judgments in order to defend his/her judgment, if necessary. For example, if the auditor agrees or disagrees with the suggested CRA, they must be able to document their reasoning. This dilemma is similar to the dilemma faced by the medical profession for using intelligent systems as decision aids. However, interpretation of how a NN produced a particular conclusion is not a trivial task, given that the very purpose of using a NN is to model complex relationships that are not well understood. Much research in this area is left to be done.

4. Conclusion The creation of the CRA Expert Network provides a good example of the potential applicability of integrate d AI technologies to audit practice and demonstrates the appropriateness of the integrated technique to the specific audit judgment task of assessing control risk. Potentially, intelligent systems such as this can provide several benefits to audit firms, including: preservation, replication, and distribution of expertise; new insights into the decision process; decision-aiding support: consistency of decisions; and, increased productivity (Borthick and West, 1987). In conclusion, the CRA Expert Network takes advantage of both the deductive approach of an ES and the inductive approach of a NN to provide a unique decision aid designed to support and facilitate the process of conducting a CRA. The system is tailored to auditors requirements in terms of data collection and analysis and allows an auditor to take a more considered and structured approach. The system offers audit firms the opportunity to improve upon the accuracy of the CRA by capturing the expertise of multiple auditors and multiple client experiences. Finally, the nature of the system encourages efficiency in the analysis process as it follows a logic-driven data collection path, requesting the user to only address questions that are logically required to make a CRA. The validation results to date indicate that an expert network shows promise

J.T. Davis et al. / European Journal o f Operational Research 103 (1997) 350-372

364

for addressing the large complex judgment processes inherent in control risk assessments. While intelligent systems are domain and task specific, this approach is conceivably transferable to other large complex judgment tasks.

Acknowledgements The authors wish to thank Dr. George W. Krull, Jr., National Director of Professional Development at Grant Thornton, whom was the driving force behind the experiment presented as part of this research. The experiment took place in conjunction with the firm's national professional education program. Dr. Kmll also provided valuable comments with regard to the experiment. In addition, we appreciate the direction provided by Stephen Yates, Partner and National Director of Advanced Audit Techniques, who provided the cases, instruction and opportunity to utilize Infocus for the study.

Appendix A. questions 6

List of judgment

variables/

A.1. General environment questions (Phase 1) Revenue cycle environment variables. Volume of transactions: High, Medium, Low (59%, 100%, 100%);

6 Each of the following represent an input node to the Neural Network. There are 210 input nodes, including three each for volume and dollar value (high, medium, low). These inputs can take on values of 1 (Yes or Key Accounting Control), " - I' (No or Missing Accounting Control), and "0' (Not Applicable to the CRA). A 'Yes' response to a particular CPO question will trigger a corresponding GCQ analysis to determine the status of a GCQ objective. The percentages in the parentheses, following each variable/question, represent the percentage of the auditors in the experiment that had included and addressed this item in their "cue set'. The order respectively reflects values for the three experimental cases - small client, nonpublic client, and public client. An n / a indicates that the item was not applicable to the case.

Dollar value of transactions: High, Medium, Low (63%, 93%, 100%); Whether part of the cycle is automated (100%, 93%, 91%); Whether management override is a concern (81%, 60%, 61%); Whether client has an internal audit staff that works to prevent or detect material misstatements in the financial statements (89%, 33%, 87%).

Planned reliance on type of controls. Whether reliance is planned on manual controls (required) Whether reliance is planned on programmed controis (required) Whether reliance is planned on segregation of duties (required) A.2. Computer processing overview (CPO) and general controls questionnaire (GCQ) company wide (Phase I) CPO question: Is a third party service organization used to process all transactions involving electronic data processing? (n/a, n/a, n/a) Has the service auditor reported on the processing of transactions? (n/a, n / a , n / a ) Does the report only cover policies and procedures in place? (n/a, n / a , n / a ) Does the report test the policies and procedures? (n/a, n / a , n / a ) Do we have a copy of the report? (n/a, n / a , n / a ) CPO question: Are the client's computers in a dedicated physical area or facility? (n/a, 80%, 87%) GCQ objective: Computer system physical security is adequate. Is entry to the computer and supervisor terminal areas restricted to those staff required for computer operations? (n/a, 80% 87%) Are lists of authorized personnel maintained and kept up-to-data? (n/a, 80%, 78%) Is a log of all visitors maintained and reviewed? (n/a, 80%, 78%)

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

Do procedures exist to control access by non-DP personnel (e.g. engineers, janitors)? (n/a, 80%, 87%) Are DP management and security personnel notified when DP staff /eave the client's employment? (n/a, 80%, 87%) Are employees who work outside normal operation hours properly authorized and adequately supervised? (n/a, 80%, 87%) Is the computer room physical security adequate to restrict unauthorized access to program and data files? (n/a, 80% 87%)

CPO question: Is there a separate EDP depart ment? (n/a, 93%, 96%) GCQ objective: Separation of EDP duties is adequate. Systems Programming? Are systems programmers prohibited from operating the computer system when production files or application programs are resident? (n/a, n/a, 91%) Are systems programmers prohibited from changing production files and programs? (n/a, n/a, 96%) Are responsibilities for various processors or software products periodically rotated among members of the systems programming staff?. (n/a, n/a, 70%) Are systems programmers' activities logged? (n/a, n/a, 87%) Are systems programmers' activity logs reviewed by management? (n/a, n/a, 96%) Are system utilities (ZAP, Super-ZAP) in use and appropriately controlled? (n/a, n/a, 87%) Is there a written computer development strategy? (n/a, n/a, 52%) Are there adequate reporting procedures to monitor the progress of computer development? (n/a, n/a, 39%) Application Programming? Is development work carried out by using separate source libraries, separate object libraries, and separate development machines? (n/a, n/a, 74%) Are application programmers restricted from accessing program libraries or data files which are

365

used for production runs? (n/a, n/a, 96%) Are application programmers prohibited from setting up and operating the computer, even during program testing? (n/a, n / a 96%), Does management approve development at key stages: feasibility systems proposals, outline systems proposals, detailed system design including systems specifications, parallel running or system acceptance testing? (n/a, n/a, 65%) Computer Operations? Is access to operators' consoles restricted to computer operators and other authorized staff?. (n/a, 93%, 96%) Are computer operators restricted from performing programming functions or running unauthorized jobs? (n/a, 93%, 96%) Are computer operators excluded from access to cash and accounting source documents? (n/a, 93%, 96%) Is access to production source libraries by computer operations' personnel restricted? (n/a, n/a, 96%) Are logs or records of computer system activity (jobs and program runs, reruns, abnormal terminations of jobs and programs, system console operator commands) maintained and reviewed? (n/a, 93%, 96%) Are system log exceptions investigated and results of the investigation documented? (n/a, 93%, 96%) Are system utilities (ZAP, Super-ZAP, etc.) in use and appropriately controlled? (n/a, 93%, 83%) Input/Output Scheduling? Is the preissuance of files for night and weekend shifts in accordance with approved job schedules? (n/a, 93%, 87%) Is authorization required for unscheduled job requests? (n/a, 93%, 87%) Are any scheduling overrides automatically logged and subject to review? (n/a, 93%, 91%) Are all terminals closed down at the end of operations? (n/a, 93%, 96%) Is there a secure location for sensitive output (e.g. check printing)'? (n/a, 93%, 91%) Library maintenance? Is formal authorization required to transfer pro-

366

J.T. Davis et aL / European Journal of Operational Research 103 (1997) 350-372

grams to the production library? (n/a, n / a , 78%) Is the transfer of files to the production library carried out by computer operations staff?. (n/a, n / a , 78%) Are all additions to the utility library authorized? (n/a, n / a , 83%) Do header labels hold reference and control data (i.e. file name, generation numbers, volume ID, time/date stamps on creation, expiration dates, control totals)? (n/a, n / a , 78%) Does program library software, if any, monitor and record program changes? (n/a, n / a , 91%) Are the activities of the EDP department (including management) appropriately segregated? Is there an organizational plan that defines and allocates responsibilities and identifies lines of communication? (n/a, 93%, 70%) Are the responsibilities of the EDP department, accounting department and other DP users clearly defined? (n/a, 93%,96%) Are there appropriate procedures for: rotation of duties/shifts, holiday arrangements, and termination of employment? (n/a, 93%, 91%) Are source documents handled within data processing only by data preparation and data control staff?. (n/a, 93%, 87%) Is a lack of segregation compensated for by increased management supervision? Is there independent management review and approval of various data processing functions? (n/a, n/a, n/a) Are there controls over certain privileged functions, such as requiring that all program library updates are assigned to designated personnel? (n/a, n / a , n / a ) CPO question: Can separate u ~ r s access the system concurrently? (19%, 93%, 91%) GCQ objective: Access control over programs and data files is adequate.

Is access control software used to restrict access to the production programs? Are master lists of authorized personnel maintained, indicating restrictions on access? (n/a, n / a , 91%)

Are access and security features within the operating system utilized to restrict access to and amendment of program files? (n/a, n / a , 91%) Is terminal activity automatically monitored and rejected access attempts reported? (n/a, n / a , 91%) Does the access control software identify each terminal via the logical address? (n/a, n / a , 91%) Does the access control software identify each terminal via the physical address? (n/a, n / a , 91%) Is access control software used to restrict access to the data files? Are master lists of authorized personnel maintained, indicating restrictions on access? (19%, 93%, 91%) Are access and security features within the operating system utilized to restrict access to and amendment of program files? (19%, 93%, 91%) Is terminal activity automatically monitored and rejected access attempts reported? (19%, 93%, 91%) Is there encryption of all sensitive data files? (15%, 93%, 91%) Does the access control software identify each terminal via the logical address? (11%, 93%, 87%) Does the access control software identify each terminal via the physical address? (11%, 93%, 87%) For data base applications, is access to the data possible only through the DBMS? (15%, 93%, 91%) Are reports generated by such software reviewed by management? Is the record of jobs actual/y run, and console logs, reviewed by supervisory personnel? (15%, 93%, 87%) Is the computer usage summary produced and reviewed? (15%, 93%. 87%)

A.3. Computer processing overview (CPO) and general controls questionnaire (GCQ) accounting cycle specific (Phase 1) Note: The next two questions determine whether the environment for the particular accounting cycle is/is not low risk. The next ten questions determine

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

whether the application complexity for this accounting cycle is simple or advanced.

CPO question: Are transactions for this cycle processed by a third party service organization? (n/a, n/a, n/a) Has the service auditor reported on the processing of transactions? Does the report only cover policies and procedures in place? Does the report test the policies and procedures? Do we have a copy of the report? CPO question: Is this application PC-based? (n/a, n/a, n/a) Is this application running on a standalone PC? CPO question: Does the client have the source code for the computer programs used in this accounting cycle? (n/a, n/a, 100%) GCQ objective: Access control over source code is adequate. Is access to source code controlled? ls the copying or renaming of sensitive programs or utilities prevented/detected? ( n / a , n / a , 91%) Are utilities, which have been made widely available (e.g. for reporting), restricted to readonly access? (n/a, n / a , 96%) Does the operating system provide operation logging which records: all use of a compiler and the targeted files, access and amendments to the program libraries, attempts to copy program files to/from the production library? ( n / a , n / a , 91%) If source code is stored off-line on tape or cartridge, is the location physically secure? (n/a, n / a , 96%) If source code is stored off-line on tape or cartridge, is the individual responsible an individual other than an applications programmer, systems programmer or computer operator? ( n / a , n / a , 96%) Could or does the client make modifications to the source code? Is there a computer program that reports all changes to each program and program library? (n/a, n / a , 91%)

367

Are logs of all modifications to the source code maintained? ( n / a , n / a , 91%) Is access to and use of the application software restricted? ( n / a , n / a , 87%) Are source modifications authorized, tested, and documented? Does testing follow the predefined objectives of the test? ( n / a , n / a , 74%) Does testing use test data files instead of production files for its tests? ( n / a , n / a , 91%) Is a chronological record of all program amendments maintained? ( n / a , n / a , 91%) Is the system documentation updated to reflect amendments? ( n / a , n / a , 87%) Are modifications made by third-party vendors to existing systems adequately recorded and controlled to ensure that all procedures are appropriately updated? (n/a, n / a , 91%) Is the vendor software tested in the same way as in-house developments? ( n / a , n / a , 83%) Are emergency fixes to production programs fully reported and independently reviewed by management? (n/a, n / a , 96%) Are the tests of the modified source code properly supervised by management? ( n / a , n / a , 91%) Do the tests of the modified source code use complete computer systems to test the interaction of different programs? ( n / a , n / a , 91%)

CPO question: Do multiple applications share the same data bases (files)? (n/a, n/a, n/a) GCQ objective: Data base application control is adequate. Is there an integrated dictionary system? Is the data base administrator responsible for the development of the data dictionary? (n/a, n/a, n/a) Are the data base changes requested by the data base administrator approved by some other authority? (n/a, n / a , n / a ) Is the distribution of the data dictionary restricted? (n/a, n / a , n / a ) Is there a data base administrator? Is the data base administrator responsible for controlling, developing and maintaining the data dictionary? (n/a, n / a , n / a )

368

J.T. Davis et al./ European Journal of Operational Research 103 (1997) 350-372

Does the data base administrator monitor data base usage? ( n / a , n / a , n / a ) Are procedures established which allow access to the data base only through the DBMS software and prevent unauthorized access while the data base is not under DBMS software control? ( n / a , n/a, n/a) Does the data base administrator approved and log all changes to the DBMS library? (n/a, n / a , n / a ) Is access to the computer operation area by the data base administrator restricted or supervised? (n/a, n/a, n/a) Are utility programs under control of the data base administrator? ( n / a , n / a , Does the data base administrator maintain and review a log of programs run? ( n / a , n / a , n / a ) CPO question: Are there real-time updates to files when transactions are entered? ( n / a , n / a , n/a) GCQ objective: Control over real-time updates to

files is adequate. Are control totals from the transaction history file reconciled to the totals from the updated files? (n/a, n/a, n/a) If memo updating (A method in which the system issues memo transactions to temporarily update a copy of the file and then actual update is performed in batches overnight) is used, is the updated copy of the file matched against the master file used during actual on-line processing? (n /a n/a, n/a) CPO question: Is the data processing function tbr

this accounting cycle decentralized (Distributed Proce~ing)? (n/a, n / a , n / a ) GCQ objective: Control over distributed processing applications is adequate. Is a software log maintained for all transactions including errors and retransmissions? (n/a, n / a , n/a) Are data dictionary tools used to document and monitor the responsibilities for various data items or elements among the distributed network'? ( n / a , n/a, n/a) Arc transaction logs maintained and reviewed for

all elements of the distributed data system? ( n / a , n/a, n/a) Are software controls used to prevent update interference on central databases in distributed systems? (n/a, n / a , n / a )

CPO Question: Are telecommunications or networks used in this accounting cycle? (n/a, 100%, 100%) GCQ Objective: Control over telecommunications a n d / o r networks is adequate. Is physical access to the terminals controlled? Is the physical access of microcomputers or terminals maintained by either equipment locks or control over location? ( n / a , 100%, 100%) Is the use of terminals controlled by passwords? Are terminals automatically locked out after failed sign-on? (n/a, 100%, 96%) Are terminals automatically locked out if the terminal is inactive for a specified period of time? ( n / a , 100%, 96%) Are sign-on and sign-off procedures specified and then verified by the computer system? ( n / a , 100%, 96%) Is access to all data files that are in the process of being updated (i.e. 'live' files) controlled by password? (n/a, 100%, 100%) Do users refrain from using common passwords (first name, spouse's name, birth date, etc.)? (n/a, 100%, 100%) Do users refrain from displaying passwords externally (e.g. post-it notes) on the terminals? (n/a, 100%, 96%) Is the use of batch files to log onto the system prohibited'? (n/a, 100%, 96%) Are passwords protected and changed on a regular basis? Is the display of passwords prevented during log on? (n/a, 100%, 100%) Upon termination or resignation, are employees immediately denied any computer access? ( n / a , 100%, 100%) Are passwords stored using a on-way encryption algorithm? (n/a, 100%, 96%) Are the system user rights transaction or application specific?

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

Are system user rights established, monitored and changed by a system or network administrator? (n/a, 100%, 91%) Is the ability to change system user rights protected by a system administrator or supervisor password? ( n / a , 100%, 91%) Is the system administrator or supervisor password changed frequently? ( n / a , 100%, 91%) Has the system administrator or supervisor password been changed from default password settings? ( n / a , 100%, 91%) Is the confidentiality of the system administrator or supervisor Password maintained? ( n / a , 100%, 91%) Is the system administrator or supervisor password encrypted? ( n / a , 100%, 91%) Is the system accessible by modem? Is data encrypted during transmission and on the files? ( n / a , n / a , n / a ) Are the phone numbers changed periodically? (n/a, n / a , n / a ) Are the phone numbers listed on the modem or terminal? ( n / a , n / a , n / a ) Are there procedures to prevent line tapping, unlisted numbers and auto connection of dial-up lines? (n/a, n / a , n / a ) Is password and transaction code documentation secured against unauthorized access? ( n / a , n / a , n/a) CPO question: Does the software in this accounting cycle generate transactions or pass information to other cycles? (19%, 93%, 96%) GCQ objective: Control over computer generated transactions is adequate. Are the methods used in the program to generate the data and related control record appropriate? (19%, 93%, 96%) Is there an adequate check over the accuracy of the data generated (e.g. reasonableness check, manual review of data generated, etc.)? (19%, 93%, 96%) Are the results of the check for accuracy reviewed and approved? (19%, 93%, 96%) Are controls such as computer sequence checks, computer matching or batch totals used to ensure the completeness of input a n d / o r updates? (19%, 93%, 96%)

369

Are exception reports investigated and reconciled? (19%, 93%, 96%) CPO question: Is there a significant loss of visible audit trail in this accounting cycle? (n/a, n / a , n/a) GCQ Objective: Control over a 'paperless' audit trail is adequate. For paperless or EDI transactions, are control totals such as batch totals, sequence numbers and 'line item counts' maintained which: track missing transactions, validate completeness, ensure onetime-only receipt of a transaction, match functional acknowledgment reports, review exception reports to ensure corrections? (n/a, n / a , n / a ) CPO question: Have there been any hardware or software malfunctions which resulted in a loss of data in this accounting cycle? (n/a, n / a , n / a ) GCQ objective: Backup control is adequate. Was data restored by use of backup? Were there restart facilities which enabled processing to continue from point of interruption? (n/a, n/a, n/a) Were there procedures which identified the processing stage reached at the time of malfunction? (n/a, n / a , n / a ) Were there controls to ensure that program libraries (most recent versions) and data files (most recent backup or copy) were restored subsequent to the failure or malfunction? (n/a, n / a , n / a ) Was data restored by manual re-entry? Were controls (batch totals, control totals, etc.) used to ensure that the restoration of data was complete and accurate? (n/a, n / a , n / a ) Was data lost and not restored'? Was the nature of the data insignificant? (n/a, n/a, n/a) Was the affect upon the financial statements immaterial? ( n / a , n / a , n / a ) A.4. A c c o u n t i n g control variables (Phase 2)

Balancing receivable subledger (1 9%, 20%, 43%) Balancing run to run control totals (11%, 13%, 57%) Balancing the G / L with the subledgers (52%, 47%, 17%)

370

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

Canceling original documents (11%, 27%, 13%) Checking by computer for duplicate entries (11%, 27%, 30%) Checking by computer the numerical sequence of a file (7%, 20%, 39%) Checking for a third party signature (7%, 13%, 9%)) Checking manually the numerical sequence of a document, journal or report (0%, 20%, 17%) Checking one-to-one (26%, 7%, 9%) Comparing batch totals (4%, 7%, 43%)) Comparison of budgeted amounts to actual amounts (0%, 7%, 9%) Comparison of cash receipts listing to deposit slips (0%, 27%, 17%) Computer generation of transactions (0%, 0%,

o%) Dual control over cash receipts (11%, 20%, 4%) Electronic authorization (0%, 7%, 0%) Independent review of edit reports for data file changes (0%, 20%, 13%) Interactive dependency edit (0%, 0%, 4%) Interactive document reconciliation (0%, 0%, 4%) Interactive edit controls (0%, 7%, 0%) Interactive existence edit (0%, 0%, 0%) Interactive feedback edit (0%, 0%, 0%) Interactive format edit (0%, 0%, 0%) Interactive key verification (0%, 7%, 0%) Interactive mathematical accuracy check (7%, 20%, 9%) Interactive prior data matching (0%, 0%, 0%) Interactive reasonableness edit(O% 13%, 4%) Interactive range check edit (0%, 0%, 0%) Interactive check digit (0%, 0%, 4%) Matching to a previously validated document (37%, 40%, 30%) Matching to a previously validated file (0%, 27%, 52%) Matching to an authorized list (26%, 47%, 22%) Monthly review of bank reconciliations (0%, 13%,

4%) Monthly review of receivable (7%, 13%, 13%) Non-interactive format edit (0%, 0%, 0%) Non-interactive mathematical accuracy (0%, 0%,

Non-interactive range check edit (0%, 0%, 0%) Non-interactive reasonableness edit (0%, 0%, 0%) Non-interactive prior data matching (0%, 0%, 0%) Non-interactive check digit (0%, 0%, 0%) Performance of analytical procedures and investigation of unusual items (0%, 13%, 17%) Periodic reconciliation of books to physical (7%, 13%, 4%) Periodic revision of budgeted amounts based on updated information (0%, 0%, 4%) Physical access controls (7%, 13%, 4%) Physical safeguards (7%, 7%, 4%) Reconciliation of manual totals to run totals (0%, 7%, 17%) Reconciliation of master file balance to control account (0%, 34%, 9%) Reconciliation of master file balance to run totals (7%, 7%, 35%) Reperformance (0%, 7%, 0%) Restrictive endorsement of checks received (0%, 7%, 4%) Reviewing adjustment transactions (0%, 7%, 4%) Reviewing internal signatures (7%, 20%, 9%) Reviewing permanent data file exception reports

(0%, 0%, 0%) Reviewing reference file data (0%, 0%, 0%) Software access controls (0%, 33%, 9%) Use of a Iockbox (7%, 7%, 0%) Use of prerecorded input (OCR, MICR, OMR) (0%, 0%, 0%) Verifying mathematical accuracy (7%, 7%, 4%) Other: Review, compare invoice adjusted for out of stock items with the packing slip (11%, 14%,

0%) Other: Driver gets signature on annotated sales invoice. (19%, 0%, 0%) Other: Cost of goods entry is calculated from sales invoices and calculations reviewed (4%, 0%, 0%) Other: Credit Check on invoice customer (0%, 0%, 0%) Other: Compares amounts written off to aged trial balance and customer detail (0%, 0%, 0%)

o%)

References

Non-interactive edit controls (0%, 0%, 0%) Non-interactive existence edit (0%,0%, 0%) Non-interactive dependency edit (0%, 0%, 0%)

A I C P A , 1989. A m e r i c a n Institute o f Certified Public Accountants. Audit guide: Consideration of the internal control structure in a financial statement audit. N e w York, NY.

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

Anderson, J., 1983. A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 261-295. Bataineh, S., AI-Anbuky, A., Al-Aqtash, S., 1996. An expert system for unit commitment and power demand prediction using fuzzy logic and neural networks. Expert Systems, 29-40. Bell, T., Ribar, S., Verchio, J., 1990. Neural nets vs. logistic regression: A comparison of each model's ability to predict commercial bank failures. KPMG Peat Marwick Working paper, May 1990. Beulens, A., Van Nunen, J., 1988. The use of expert system technology in DSS. Decision Support Systems 4, 421-431. Borthick, A.F., West, O., 1987. Expert systems - A new tool for the professional. Accounting Horizons 1, 9-16. Brown, C., Coakley, J., Phillips, M.E., 1995. Neural networks enter the world of management accounting. Management Accounting, 51-57. Brown, C., Phillips, M., 1990. Expert systems for management accountants. Management Accounting, 18-22. Brown, C., O'Leary, D., 1994. Introduction to Artificial Intelligence and Expert Systems. Published Monograph, 1994. Caudill, M., 1991. CRA expert networks. Byte, 108-116. Cheng, B., Titterington, D., 1994. Neural networks: A review from a statistical perspective. Statistical Science 9, 2-54. Coakley, J., Brown., C. 1993. Artificial neural networks applied to ratio analysis in the analytical review process. International Journal of Intelligent Systems in Accounting, Finance and Management, 19-39. Craven, M., Shavlik, J., 1996. Extracting tree-structured representations of trained networks. In: Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA. Davis, E., 1994. Effects of decision aid type on auditors' going concern evaluation. Audit Judgment Symposium, Co-sponsored by Grant Thornton and University of Southem California, pp. 21-22. Davis, J. 1996. Experience and auditors' selection of relevant information for preliminary control risk assessment. Auditing: Journal of Practice and Theory, 16-37. Dcng, P., 1994. Automating knowledge acquisition and refinement for decision support: A connectionist inductive inference model. Decision Sciences 24 (2), 371-393. Dreyfus, H., Dreyfus, S., 1986. Mind Over Machine: The Power of llurnan Intuition and Expertise in the Era of the Computer. Free Press, New York, NY. Eining, M., Jones, D. and Loebbecke, J., 1994. An experimental examination of the impact of decision aids on the assessment and evaluation of management fraud. Audit Judgment Symposium, Co-sponsored by Grant Thornton and University of Southern California, February 21-22. Frisch, A., Cohn, A., 1991. Thoughts and after-thoughts on the 1988 Workshop on Principles of Hybrid Reasoning. A1 Maga zinc: Special Issue, January 1991, 77-.83. Gallant, S. 1993. Neural Network Learning and Expert Systems. MIT Press, Cambridge, MA. Gillett, P., Bamber, E., Mock, T., Trot.man, K., 1995. Audit judgment. In: Bell, T., Wright, A. (Eds.), Auditing Practice, Research, and Education: A Productive Collaboration. American Institute of Certified Public Accountants in Cooperation

371

with the Auditing Section of the American Accounting Association, New York. Grant Thornton, 1992. Information and control understanding system version 2.08. Gray, G., McKee, T., Mock, T., 1991. The future impact of expert systems and decision support systems on auditing. Advances in Accounting 9, 249-273. Greeno, J.G. 1973. The structure of memory and the process of solving problems. In: Solso, R.L. (Ed.), Contemporary Issues in Cognitive Psychology: The Loyola Symposium. Wiley, New York, NY. Gupta, U., 1994. How case-based reasoning solves new problems. Interfaces 24 (6), 110-119. Hecht-Nielsen, R., 1990. Neurocomputing. Addison-Wesley, Reading, MA. Hedberg, S., 1995. Where's AI hiding? AI Expert, 17-20. Hornick, K., Stinchcombe, M., White, M., 1989. Multilayer feedforwared networks are universal approximators. Neural Networks 2, 359-366. Jackson, P. 1990. Introduction to Expert Systems. Addison-Wesley, Workingham, MA. Johnson-Laird, P.N., 1983. Mental Models: Toward a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press, Cambridge, MA. Kandel, A., Langholz, G., 1992. Hybrid Architectures for Intelligent Systems. CRC Press, Boca Raton, FL. Koopmans, L. 1987. Introduction to Contemporary Statistical Methods. Duxbury Press, Boston, MA. Kunz. J., Kehler, T., Williams, M., 1987. Applications development using a hybrid A1 development system. In: Hawley, R. (I-M.), Artificial Intelligence in Programming Environments. Ellis Horwood, Chichester. Lacher, R., Coats, P., Shanker, C., Franklin, L., 1995. A neural network for classifying the financial health of a firm. European Journal of Operational Research 85 (1), 53-65. Liberatore, M., Stylianou, A., 1993. The development manager's advisory system: A knowledgebased DSS tool for project assessment. Decision Sciences 24 (5), 953-976. Liberatore, M., Stytianou, A., 1995. Expert support systems for new product development decision making: A modeling framework and applications. Management Science 41 (8), 1296-1316. Lin, C., Hendler, J., 1995. Examining a hybrid connections/symbolic system for the analysis of ballistic signals. In Sun, R., Bookman, L. (Eds.), Computational Architectures Integrating Neuraland Symbolic Processes. Kluwer Academic Publishers, Norwell, MA, pp. 319-348. Lippmann, R., 1987. An introduction to computing with neural nets. IEEE ASSP Magazine, 4-22. t.ymer, A., Richards, K., 1995. A hybrid-based expert system for personal pension planning in the UK. Intelligent Systems in Accounting, Finance and Management 4, 71-88. Maclntyre, J., Smith, P., Harris, T., Industrial experience: The use of hybrid systems in the power industry. In: Medsker, L. (Ed.), Hybrid Intelligent Systems. Kluwer Academic Publishers, Norwell, MA, pp. 57-74. MacLennan, B., 1993. Continuous symbol systems - The logic of

372

J.T. Davis et al. / European Journal of Operational Research 103 (1997) 350-372

connectionism. In: D.S. Levine, M. Aparicio IV (Eds.), Neural Networks for Knowledge Representation and Inference. Lawrence Erlbaum, Hillsdale, NJ. Markham, I., Ragsdale, C., 1995. Combining neural networks and statistical predictions to solve the classification problem in discriminant analysis. Decision Sciences 26 (2), 229-242. Medsker, L., 1994. Design and development of hybrid neural network and expert systems. In: Proceedings of the IEEE International Conference on Neural Networks III, Orlando, FL, pp. 1470-1474. Medsker, L., Bailey, D., 1992. Models and guidelines for integrating expert systems and neural networks. In: Kandel, A., Langholz, G. (Eds.), Hybrid Architectures for Intelligent Systems. CRC Press, Boca Raton, FL, pp. 153-171. Medsker, L., 1995. Hybrid Intelligent Systems. Kluwer Academic Publishers, Norwell MA. Minsky, M., 1986. The Society of Mind. Simon and Schuster, New York, NY. NeuralWare: Technical Publications Group, 1992. Neural computing - A technology handbook for professional ll/plus and neuralWorks explorer. Pittsburgh, PA. O'Leary, D., Watkins, P., 1991. Review of expert systems in auditing. Expert Systems Review for Business and Accounting, 3--22. Ripley, B., 1994. Neural networks and related methods for classification. Journal of the Royal Statistical Society B 56 (3), 409-456. Ripley, B., 1993. Statistical aspects of neural networks. In: Barndorff-Nielsen, O., Jensen, J., Kendall, S. (Eds.), Networks and Chaos - Statistical and Probabilistic Aspects. Chapman and Hall, London, pp. 40-123. Rosenhcad, J., 1992. Into the swamp: The analysis of social issues. Journal of Operational Research Society 43 (4), 293305.

Rumelhart, D., McClelland, J., 1986. Parallel Distributed Processing: Volumes I and II. The MIT Press, Cambridge, MA, pp. 110-146. Sarle, W., 1994. Neural networks and statistical models. In: Proceedings of the Nineteenth Annual SAS Users Group International Conference. SAS Institute, Cary NC, pp. 15381550. Scott, D., Wallace, W., 1994. A second look at an old tool: Analytical procedures. The CPA Journal, 30-35. Silverman, B., 1995. Knowledge-based systems and the decision sciences. Interfaces 25 (6), 67-82. Solso, R., 1988. Cognitive Psychology, 2nd edition. Allyn and Bacon, Boston, MA. Sun, R., Bookman, L., 1995. Computational Architectures Integrating Neural and Symbolic Processes. Kluwer Academic Publishers, Norwell, MA. Sun, R., 1994. A two-level hybrid architecture for structuring knowledge for commonsense reasoning. In: Sun, R., Bookman, L. (Eds.), Computational Architectures Integrating Neural and Symbolic Processes. Kluwer Academic Publishers, Norwell, MA, pp. 247-282. Sutton, S., Young, R., McKenzie, P., 1995. An analysis of potential legal liability incurred through audit expert systems. Intelligent Systems in Accounting, Finance and Management 4, 191-204. Tam, K., Kiang, M., 1992. Managerial applications of neural networks: The case of bank failure predictions. Management Science 38 (7), 926-947. Turban, E., Watkins, P.R., 1986. Integrating expert systems and decision support systems. MIS Quarterly 10 (2), 121-136. Yoon, Y., Guimaraes, T., Swales, G., 1993. Integrating artificial neural networks with rule-based expert systems. DSS Special Issue on Artificial Neural Networks.