The influence of a business-process focus on category knowledge and internal control evaluation

Accounting, Organizations and Society 30 (2005) 423–434 www.elsevier.com/locate/aos

The influence of a business-process focus on category knowledge and internal control evaluation Lori S. Kopp

a,*

, Ed O’Donnell

b

a

b

Faculty of Management, University of the Lethbridge, Lethbridge, AB, Canada T1K 3M4 W.P. Carey School of Business, Arizona State University, P.O. Box 873606, Tempe, AZ 85287-3606, USA

Abstract This study examines whether organizing information about internal controls around business processes instead of control objectives produces stronger category knowledge during training and improves decision performance during internal control evaluation. Novices who were trained to evaluate internal controls using business-process-focused materials developed stronger category knowledge. After training, novices who evaluated internal controls using materials organized around a process focus identified more control issues during a control evaluation task. Findings suggest that the process focus may be a more effective framework for organizing internal control evaluation tasks.
2004 Elsevier Ltd. All rights reserved.

Introduction Assurance services firms generally use one of two alternative task structures to organize the information that their auditors evaluate during financial statement audits. They employ either a categories-of-business-processes focus or a categories-of-control-objectives focus (Lemon, Tatum, & Turley, 2000). For example, while evaluating the reliability of internal controls, auditors with firms that use the process focus learn about client operations with information that has been categorized by business process, like procurement, inbound logistics, and sales. On the other hand, auditors who work for firms that use the objective focus learn with information categorized by

*

Corresponding author. E-mail addresses: [email protected] (L.S. Kopp), ed. [email protected] (E. O’Donnell).

control objectives, like authorization, accuracy, and completeness (Curtis & Viator, 2000). For most tasks, auditors acquire the same information under either task structure. The difference is the categories used to organize and present that information (Frederick, 1991). The category knowledge that auditors develop through training and experience becomes a significant element of their expertise (Tubbs, 1992; Frederick, Heiman-Hoffman, & Libby, 1994). Auditors with well-developed category knowledge tend to perform better than auditors with less-welldeveloped category knowledge (Bonner, Libby, & Nelson, 1997). However, when the categories used to present decision information do not match the categories that auditors use to store knowledge in memory, decision performance can suffer (Nelson, Libby, & Bonner, 1995). This study examines two research questions. First, does organizing training materials around business-process categories rather than

0361-3682/$ - see front matter
2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.aos.2004.03.004

424

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

control-objective categories help untrained novices develop stronger category knowledge? Category knowledge becomes stronger when the mental structures that people use to store information become more clearly defined, which enhances their ability to encode and remember the information that they acquire (Day, Arthur, & Gettman, 2001). Second, does structuring internal control evaluation tasks around a process focus rather than an objective focus help trained novices perform better by creating a more consistent mapping between the way decision information is presented and the way they organize knowledge in memory? We provide evidence that the answer to both questions is yes. The debate about whether the process focus provides a more effective task structure for evaluating client-specific information than the objective focus is currently being waged in audit practice (Lemon et al., 2000). In response to changing economic conditions, several of the largest assurance services firms have adopted a process-focused audit methodology while other firms continue to use the objective-focused approach (Eilifsen, Knechel, & Wallage, 2001). Not surprisingly, firms that use a process focus in the field also use a process focus in their training programs (Ballou & Heitger, 2003). If assurance services firms are not careful to match the structure of their audit procedures with the category knowledge that their auditors develop through firm training and field experience, audit performance could suffer. We examined the influence of these alternative task structures in the context of internal control evaluation. Internal controls are oversight practices and processing constraints an organization uses to help ensure that its assets are protected from misappropriation and its systems provide reliable information about business transactions. During an assurance engagement, professional standards require that auditors evaluate the reliability of this system of internal controls before they design their auditing tests (American Institute of Certified Public Accountants, 1995). Establishing an effective system of internal controls has become a central issue in corporate governance because of the Sarbanes–Oxley Act that was recently enacted by the United States government.

This legislation establishes accountability for the system of internal controls at the level of the corporate board of directors and imposes penalties for non-compliance. Frederick (1991) demonstrated that auditing students who learn about controls from documentation that has been organized around transaction flows remembered that information more effectively than novices who used documentation organized around categories of control objectives. Our study extends Frederick’s work on knowledge organization by evaluating how task structure influences decision performance during a diagnostic auditing procedure. We examine whether novices who perform control evaluations using documentation that has been structured around business process flows can recognize control issues more effectively than auditors who use documentation structured around categories of control objectives. The findings from this study have implications for both audit practice and decision-making research. With respect to training, strengthening category knowledge through exercises structured around process-focused categories could help assurance services firms and academics teach aspiring auditors more effectively. Evidence that category type influences learning extends the literature on knowledge development by contributing to our understanding of the association between training structure and knowledge acquisition. Regarding decision performance, if the process focus provides a more effective structure for internal control evaluation then assurance services firms may want to re-evaluate how their decision support systems present information to auditors in the field. Evidence that a processfocused task structure improves decision performance extends the literature on decision-making by contributing to our understanding of the association between information presentation and human ability to integrate and evaluate decision cues.

Theoretical framework This study examines how alternative category structures affect (a) the development of category

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

knowledge and (b) decision performance during an information processing task. Prior auditing research has demonstrated that training can influence the development of category knowledge (Bonner et al., 1997) and that stronger category knowledge can improve judgment in a variety of auditing decision contexts (see Nelson, 1994 for a review). This study examines whether using business-process categories rather than control-objective categories for training novices to evaluate internal controls fosters the development of stronger category knowledge. Prior research has also demonstrated that process-focused task structures influence recall differently from objective-focused task structures (Frederick, 1991) and that a mismatch between the categories used to structure an audit task and the categories auditors use to store knowledge in memory can impair decision performance (Nelson et al., 1995). This study examines whether using business-process categories rather than controlobjective categories to structure an internal control evaluation task fosters superior decision performance. Audit judgment researchers who have compared a process focus with an objective focus have hypothesized that differences between these alternative category structures result from the differences in the way that categories are linked to each other. For example, Frederick (1991) explains that the information components in process-focusedcategories are linked by temporal and/or spatial relationships while the information components of objective-focused categories are linked by classifications of features. He suggests that processfocused categories can enhance the accessibility of knowledge stored in memory because temporal affiliations provide more or better-ordered retrieval cues. Schank (1999, pp. 1–20) describes how human information processing relies on memory processes and the categorical structures used to organize the information stored in memory––the knowledge that people bring to a decision task. Memory processes involve instantaneous, automatic cognition that can be characterized as indexing. People need a memory index to store new information, retrieve existing knowledge, and create new indices

425

for novel information. The human mind searches this indexing system without conscious effort whenever a new input enters the realm of sensory perception. In essence, when people process information, the mind consults a mental index, which provides a link to memory structures that store the knowledge needed to understand and interpret the information. Schank’s (1999, pp. 107–122) theory of dynamic memory suggests that human memory is organized by categories––hierarchical mental structures that group knowledge by common characteristics. The memory index provides a map for navigating between higher-level categories that contain more general knowledge and lower-level categories that contain the details. Memory processes are constructive in nature. The mind relies on the memory index to establish links among related categories and assemble relevant knowledge each time a new item of information is processed. During decisionmaking tasks, sub-conscious cognitive processes construct the necessary context, procedures, and criteria on demand, when new information is integrated with knowledge stored in memory (also see Libby & Luft, 1993). Schank (1999, pp. 21–40) believes that the categories used to store information in memory have a significant influence on the outcome of a decision task. When categories are well-defined, a quality that results from learning, the indexing process has a greater chance of properly classifying a decision cue. When the mind tries to index a cue using poorly defined categories, classification errors become more likely and decision performance can suffer because classifying a decision cue improperly (by associating it with the wrong category) results in misinterpreting the cue. Schank (1999, pp. 89–106) describes how people naturally tend to categorize their knowledge in terms of temporally organized series of events and related constructs. The high-level mental structures that the human mind uses to access categories of detailed information tend to be organized around temporal sequences of events or ‘‘story lines’’. At the highest level, memory indexing processes begin by accessing or constructing a story line that provides context for the information being acquired. A story line provides the basis for

426

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

interpreting new decision cues by activating categories of more detailed information related to the active story line. In other words, using a story line to organize the memory indexing process helps prevent cues from being interpreted out of context (by being associated with the wrong knowledge category). Frederick (1991) examined how the structure of information about internal controls influenced ability to remember specific controls for both experienced auditors and undergraduate accounting students (novices). He found that experienced auditors who received documentation organized around transaction flow story lines (schematic organization) recalled more specific controls than auditors who received documentation organized around categories of control objectives (taxonomic organization). However, the type of categories used to organize control documentation had no influence on novices’ ability to recall specific controls, which suggests that at least some of the benefits of using a process-focused task structure may develop as a result of training and experience. Frederick (1991) also found that both experienced auditors and novices tended to cluster the controls better when they used transaction-flowsstory-line documentation than when they used control-objectives documentation. In other words, controls recalled under a process-focused task structure more closely followed the sequence in which the controls were presented than did controls recalled under the objectives-focused task structure. These findings support the notion that people tend to rely on mental representations that organize knowledge around memory structures organized like story lines rather than by categories of attributes. Research has demonstrated that structuring information around story lines can improve decision performance. Pennington and Hastie (1986) found that jurors, who were presented with evidence grouped by descriptive classifications tended to re-arrange that evidence into a mental representation that provided a story line about what happened. The story lines that they constructed provided a context for evaluating their choice of verdicts. In another series of experiments Pennington and Hastie (1988) found (a) that vari-

ability in the story lines people create corresponds to variability in their decisions and (b) that people construct and use story lines to provide a context for decision making. In a study where experienced auditors evaluated the likelihood that their client could stay in business, case materials structured around a story line had a stronger influence on going-concern judgments than case materials structured by categories of financial attributes (Ricchiute, 1992). In summary, Shank’s theory suggests that (a) people who develop stronger categorical knowledge structures are likely to make better decisions because they can process information more effectively and (b) because people naturally tend to think in story lines, they can process information more effectively when that information is organized around series of temporally related events (a story). Research has provided evidence supporting both of these propositions. Training categories Category knowledge becomes stronger as the mental structures used to group and store knowledge become more well defined (Glaser, 1990). As category knowledge becomes stronger, decision performance generally improves (Day et al., 2001). In auditing contexts, for example, stronger category knowledge leads to a better understanding of frequency distributions for financial statement errors (Nelson, 1993). Auditors with stronger category knowledge can process information about client activities more effectively and develop more accurate expectations about the probability that a specific event has occurred (Nelson et al., 1995). Novices with stronger category knowledge can also learn from subsequent experience better than novices with less well-developed category knowledge (Bonner et al., 1997). Category knowledge develops through training and experience. During internal control evaluation, performance improves as the category knowledge that novices develop through training becomes stronger (Curtis & Viator, 2000). Training materials organized around categories of attributes that link together to form a causal chain of correlated events produce stronger category

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

knowledge than training materials organized around categories where the causal path is less obvious (Kersten & Billman, 1997). Schank (1999) believes that, because the human mind naturally tends to organize environmental cues around a story line, learning becomes more difficult when information is organized around categories of attributes that do not form a story. He suggests that, without a story line to follow, it is more difficult to (a) construct categories for storing new information and (b) link those categories to the memory index. If individuals increase the strength of their knowledge structures when they think in stories, then novices should be able to learn more effectively when training programs are organized around story lines. Using a process focus for internal control evaluation provides a distinct story line not provided when using the controlobjective focus. These associations between ability to learn and the types of categories used to structure the learning environment give rise to our first research hypothesis: H1: Novices who learn how to perform internal control evaluation using materials organized around a business-process focus will develop stronger category knowledge than novices who learn about this task using materials organized around a control-objective focus.

Task structure Category structures affect judgment by influencing the knowledge representations that people construct from memory (Boland et al., 2001). Altering the way that task-specific information is presented changes the structure of the task, which interacts with the structure of knowledge stored in memory and influences decision performance (Devine & Kozlowski, 1995). Schank (1999, pp. 89–106) suggests that people tend to make more effective decisions when they can simply retrieve a story line from memory rather than having to construct a story line from decision information. People naturally tend to

427

think in terms of stories. When decision tasks are structured around a story line, memory indexing processes involve retrieving a comparable story line and using links provided by that mental structure to activate related categories of information. However, when decision tasks are structured around categories of attributes other than story lines, the mind must construct an appropriate story line before the indexing process can begin. The benefits of organizing information around story lines have been examined in various decision contexts. Klein (1998) describes how people rely on stories they construct from experience-based memories to evaluate the plausibility of decision alternatives. He explains that, in naturalistic decision-making contexts, people construct a story about how a potential decision will play out when they evaluate a novel decision alternative. Kinney (1997) believes that auditors who learn the story of value creation will be in a better position to recognize potential audit problems (because they have more completely activated their category knowledge of business processes). O’Donnell and Schultz (2003) found that auditors who used case materials organized around business process categories were able to identify more risk factors during analytical procedures than auditors who used materials organized around financial statement categories. If task structures that organize information around a story line are less likely to result in construction errors, and if process-focused task structures provide a story line, then processfocused task structures should be less likely to foster interpretation errors than objective-focused task structures. In other words, decision performance should be better for tasks where decision information is organized around business-process categories rather than tasks organized around control-objective categories. These associations between task structure and decision performance give rise to our second research hypothesis: H2: Novices who perform an internal control evaluation task using information organized around business-process categories will be able to identify more control issues than

428

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

novices who evaluate information organized around control-objective categories.

Method We tested our hypotheses in a laboratory experiment where participants with no audit training or experience learned about internal control evaluation then evaluated case materials and identified control issues. Our study was organized in a two-by-two, crossed-factors design. One factor was the type of categories used for training. Participants learned about internal control evaluation either with materials that organized information about control procedures around business-process-categories or control-objective categories. The other factor was the type of categories used to structure the internal control evaluation case. Participants received case materials that were organized around either the same process-focused or objectivefocused categories used for training. Participants were assigned to one of four groups so that the two experimental factors would be crossed. For example, about half of the participants who were trained with the process-focused materials received the business-process-focused internal control evaluation case materials. The other half received the control-objective-focused case materials. Participants One objective of this study was to examine whether the type of categories used for training would influence the development of category knowledge. Therefore, it was essential to use participants with no category knowledge. A total of 82 students enrolled in their first undergraduate accounting course participated in this experiment. We chose students with no accounting training or work experience because we wanted to test the influence of the training that we provided without the confounding influence of knowledge developed from other courses or work experience. Prior research has shown that students develop a dominant internal control knowledge structure after completing junior-level financial and managerial accounting courses (Curtis & Borthick, 2002). We

could not use experienced auditors as subjects because task-specific experience provides procedural knowledge that alters memory structures (Anderson, 1982) and can influence the development of category knowledge (Bonner & Walker, 1994). We could not use newly hired auditors because (a) their audit training, which provides opportunities to develop category knowledge, might not be uniform, (b) we would have no control over the type of categories used during their training, and (c) instruction provided to them during their college education leads to development of category knowledge. The students who participated were novices with regard to internal control evaluations. During their accounting course, they had received instruction about what internal controls were and why it was important for an organization to assess internal control effectiveness. However, they had received no instruction in either accounting information systems or auditing and none of the participants reported any experience in evaluating internal controls. Participation in the study was voluntary but students who chose to participate received bonus points in the class. To motivate performance, all participants were eligible for a cash prize lottery where the number of times their names were entered into the drawing depended on their level of task performance. Of the 44 participants assigned to the processfocused learning condition, 24 completed the process-focused internal control evaluation case and 20 completed the objective-focused internal control evaluation case. Of the 38 participants assigned to the objective-focused learning condition, 20 completed the process-focused internal control evaluation case and 18 completed the objective-focused internal control evaluation case. Materials All of the narrative cases used in this study were adapted from accounting information systems or auditing textbooks. Each narrative contained a description of (a) the origin of every document and record in the system, (b) all processing that takes place, (c) the disposition of every document and

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

record in the system, and (d) an discussion of the control issues of the purchasing cycle for a hypothetical company. Each case contained information about 20 specific internal control procedures, which were taken from the list developed by Frederick (1991). In process-focused cases, these procedures were described in connection with activities related to (1) purchasing, (2) receiving, (3) accounts payable, and (4) cash disbursements. In objective-focused cases, the same 20 procedures were discussed under control objectives that included (1) authorization, (2) accuracy, (3) completeness, and (4) validity. Both the process-focused and objectivefocused versions of a case contained identical sentences except for transition words and phrases. Procedures This study involved three phases. During the first phase, the novice participants received instruction about how to evaluate internal controls using either process-focused or objective-focused training materials. They were given the opportunity to ask questions about the training materials, then they evaluated a training case. The case given to the participants was compatible with the knowledge structure in which they received instruction. The instructor (one of the authors) used the case to explain internal control concepts and discuss each of the 20 specific internal control procedures. During this discussion, the instructor described conditions that would represent strengths or weaknesses in the system of internal controls. Paragraph by paragraph, the instructor demonstrated how the wording in the narrative could be used to identify whether an internal control procedure was being followed. At the end of the first phase, participants were provided with a written copy of the solution to the case, and they were instructed to review these materials before the next training session. Five days after phase one participants returned for phase two. Participants began phase two by completing a demographic questionnaire. Then they evaluated a new case and identified internal control strengths and weaknesses. As in phase one, the version of the case they evaluated during phase

429

two was compatible with the knowledge structure in which they received instruction. First, participants were asked to evaluate internal control without the use of any of the instructional materials provided to them during the first session. Then, they were allowed to review their evaluation using instructional materials from the first session. Using a discussion format, the instructor identified the internal control strengths and weaknesses verbally on a paragraph-by-paragraph basis. At the end of the second phase, participants were provided with a written solution and asked to review all instructional materials in preparation for the internal control evaluation they would perform in the next session. Two days after phase two participants returned for the third phase. During the third phase, participants were instructed to evaluate a different case and document all control issues (i.e., internal control strengths and weaknesses). They were not allowed to use any of the instructional materials used in the prior sessions. All participants completed the task within the 30 min allowed (a period of time that a pilot study indicated would be sufficient). After giving the instructor their documentation, participants completed a debriefing questionnaire and were then asked to sort the 20 internal controls discussed during the training phase into four unlabeled categories. Consistent with Nelson et al. (1995), sorting internal controls into a predetermined number of categories was used to hold constant the level of detail in the participants’ sorts.

Results During the internal control evaluation phase of the experiment, participants spent an average of 16 min evaluating case materials and documenting their findings. On average, they listed five control issues, including four of the 13 internal control strengths seeded to the case materials and one of the seven internal control weakness. While this level of performance would be considered deplorable for auditors, it may not be unreasonable for novices who have had only a few hours of instruction and practice, no formal auditing

430

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

Our first hypothesis states that participants who were trained with process-focused materials will develop stronger category knowledge than participants who used objective-focused training materials. We calculated cophenetic correlations (Sneath & Sokal, 1973) between the categories of internal controls that participants used during the training phase and groups of internal controls that they created during the free-sort task, a technique that was employed in similar studies by Nelson et al. (1995) and Bonner et al. (1997). For participants who received process-focused training materials, strength of category knowledge was measured using the correlation coefficients for categories from the process-focused training materials. Strength of category knowledge for participants assigned to the objective-focused training condition was calculated using the coefficients for objective-focused training categories. This metric provides an indication of the extent to which participants used the categories they learned during the training phase as an organizing framework for their category knowledge of internal control procedures. In other words, it measures the extent to which category knowledge has become well defined. Before testing our first hypothesis, we evaluated the validity of our measure for the strength of category knowledge.

their coefficients for objective-focused categories (a finding reported by Bonner et al., 1997). Because participants had no opportunity to develop category knowledge before this experiment (they were taking their first accounting course and they reported having no accounting experience), the correlation coefficients measure the extent to which the categories they developed during the sorting task match the categories learned during the training phase. There is a negative Pearson correlation of 0.34 between the process-focused and the objective-focused measures (one-tailed pvalue ¼ 0.0008), which suggests that these two measures differentiate types of category knowledge. As category knowledge becomes stronger, the cognitive demands of an evaluative judgment task decrease, and the level of difficulty diminishes (Schank, 1999). Accordingly, if cophenetic correlations provide a reliable surrogate for strength of category knowledge, then that metric should be negatively correlated with the perceived difficulty of the task. During the debriefing questionnaire, participants rated the average difficulty of the task at 6.75 on a an 11-point scale (0 ¼ ‘‘not at all difficult’’, 10 ¼ ‘‘extremely difficult’’). There is a negative Pearson correlation of 0.22 (one-tailed pvalue ¼ 0.0211) between participants’ difficulty ratings and the strength of their category knowledge (measured using cophenetic correlation coefficients). Because there was no difference in the average ACT scores or GPA across the four experimental conditions, it unlikely that this result was driven by between-subjects differences in learning ability. Therefore, this negative correlation suggests that participants with more welldefined knowledge categories for internal control procedures found the task to be less difficult and provides further support for the validity of our surrogate measure for strength of category knowledge.

Validity checks

Hypothesis tests

If cophenetic correlations provide a reliable surrogate for strength of category knowledge, then each participant’s coefficients for process-focused categories should be negatively correlated with

Our first research hypothesis predicts that novices trained with materials organized around process-focused categories will develop stronger category knowledge than novices trained with

training, and no field experience. This exercise would be considered a relatively simple auditing task. However, after completing the task, participants rated the difficulty level at almost seven on an 11-point scale where zero represented ‘‘not at all difficult’’ and 10 represented ‘‘extremely difficult’’. Rating the difficulty level above the midpoint could be expected of novices with very little training who tried hard but were not able to perform as well as experienced auditors. Quantifying variables

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

objective-focused materials. We evaluated our first hypothesis by using analysis of variance to test whether category knowledge was as strong for participants in the objective-focused training conditions as it was for participants in the processfocused training condition. Because the data used to quantify strength of category knowledge was gathered after the evaluation task, we included a task structure variable in the model to control for the main and interactive influence that alternative structures might have had on the development of category knowledge. Results are presented in Table 1. Category knowledge was stronger for participants assigned to the process-focused training condition. Adjusted cell means for strength of category knowledge (measured using cophenetic correlation coefficients) were 0.34 for the processfocused training group, which was significantly ðp < 0:05Þ greater than the average of 0.22 for participants assigned to the objective-focused training condition. While these cophenetic correlations appear low, they are comparable to auditors’ cophenetic correlations (range 0.01 to 0.31) in a previous study examining internal control (Davis & Davis, 1998). These findings suggest that using a process focus to organize training materials around a story line can help novices develop stronger category knowledge, which provides evidence supporting our first research hypothesis.

431

Our second research hypothesis predicts that, during a internal control evaluation task, novices who use a process-focused task structure will identify more control issues than participants who use an objective-focused task structure. We evaluated this hypothesis by using analysis of variance to test whether novices assigned to the processfocused condition identified more control issues than novices assigned to the objective-focused condition, while controlling for the influence of differences in training. Results are presented in Table 2. Participants who used a process-focused task structure identified an average of 6.00 control issues, which is significantly ðp < 0:01Þ higher than the average of 4.05 control issues identified by participants assigned to the objective-focused task structure. The types of categories used for training had no main or interactive influence on the number of control issues that participants identified. These findings suggest that using a process focus to structure internal control evaluation tasks can help novices identify more control issues, which provides evidence supporting our second research hypothesis. Additional analysis As discussed above, prior research has shown that decision performance improves as category

Table 1 Category knowledge developed during training DF

Sum of squares

Panel A: Analysis of variance Dependent variable ¼ strength of category knowledge (cophenetic correlations) Model 3 0.30 Error 78 4.74 Process-focused training ðT Þ 1 0.28 Process-focused task structure ðSÞ 1 0.01 Interaction ðT
SÞ 1 0.02

Panel B: Cell means Process-focused task structure Objective-focused task structure Condition mean

F -statistic

1.69 4.69 0.10 0.44

Process-focused training

Objective-focused training

Condition mean

0.31 ðn ¼ 24Þ 0.36 ðn ¼ 20Þ 0.34

0.23 ðn ¼ 20Þ 0.21 ðn ¼ 18Þ 0.22

0.28 0.27

t ¼ 0:72

t ¼ 0:23

t ¼ 0:32

t ¼ 1:10 t ¼ 1:93 t ¼ 2:17

432

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

Table 2 Performance during internal control evaluation DF

Sum of squares

Panel A: Analysis of variance Dependent variable ¼ number of control strengths and weaknesses documented Model 4 77.44 Error 77 1005.77 Process-focused training ðT Þ 1 0.06 Process-focused task structure ðSÞ 1 76.68 Interaction ðT
SÞ 1 0.06

Panel B: Cell means Process-focused task structure Objective-focused task structure Condition mean

F -statistic

2.00 0.00 5.95 0.00

Process-focused training

Objective-focused training

Condition mean

6.00 ðn ¼ 24Þ 4.00 ðn ¼ 20Þ 5.00

6.00 ðn ¼ 20Þ 4.11 ðn ¼ 18Þ 5.05

6.00 4.05

t ¼ 1:83

t ¼ 1:61

t ¼ 2:44

knowledge increases. Results presented in Table 1 suggest that process-focused training produced stronger category knowledge than objectivefocused training. However, results presented in Table 2 suggest that type of training had no influence on task performance. To examine these associations further, we conducted a direct test of the joint influence of category knowledge and task structure by regressing the number of control issues documented on (1) a dummy variable indicative of task structure, (2) strength of category knowledge measured by cophenetic correlations, and (3) the interaction between those two variables. Results are presented in Table 3. Both process-focused task structure and strength of category knowledge have a significant, positive influence on the number of control issues documented, but the interaction term is not significant. These findings suggest that using busi-

t ¼ 0:00 t ¼ 0:09 t ¼ 0:07

ness-process categories to organize control evaluation tasks improves performance. Furthermore, the benefits of a process-focused task structure accrue over and above the benefits of developing strong category knowledge.

Discussion Results from this study suggest (a) that using process-focused categories when training novices to perform internal control evaluation can enhance category learning and (b) that organizing decision support information around a process focus can help novice auditors make better decisions. Furthermore, these findings suggest that using a process-focused task structure influences novices’ judgment above and beyond the influence of the category knowledge that they bring to the task. At

Table 3 The influence of category knowledge and task structure on performance during internal control evaluation Intercept Process-focused task structure (PTS) Strength of category knowledge (SCN) Interaction (PTS · SCN)

Parameter estimate

T -statistic

One-tailed P -value

1.87 2.02 7.41 0.19

2.54 1.98 3.98 0.07

0.0066 0.0208 0.0001 0.9428

OLS regression model: perform ¼ b0 + b1 PFE + b3 (PFT · SCN) + e; r-square ¼ 0.33; f -statistic ¼ 13.05; p-value ¼ 0.0001. Perform ¼ number of control strengths and weaknesses documented.

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

least for novices, a process-focused task structure appears to help them identify control issues better than an objective-focused task structure. Research has shown that, when people encounter a task structure that does not match their knowledge structures, they tend to re-structure task specific information to conform to category knowledge and, as a result, decision performance can suffer (Nelson et al., 1995; Bonner et al., 1997). Future research could examine whether the need to re-structure task information when auditors with process-focused knowledge structures use an objective-focused task structure to evaluate controls accounts for differences in performance. This study extends our understanding of how the process focus can influence evaluative judgment. However, while students taking their first accounting course were appropriate subjects for this study, their performance is not necessarily indicative of the performance of experienced auditors in the field and the generalizability of these results will be limited. Future research should examine whether the proceduralized knowledge that develops through task-specific experience changes associations between category knowledge, task structure, and decision performance. Another limitation of this study must be considered when evaluating our results. Because we did not establish a baseline measure for strength of category knowledge before we began the training phase, we do not know how strong an effect that our training manipulation had on the development of categorical structures in memory. Evidence from our test of hypothesis one demonstrates that training altered category knowledge. However, we only measured the strength of category knowledge structures after the training had been delivered. It could be that the students who participated in this experiment were more familiar with business-process categories like purchasing and receiving than they were with control-objective categories like authorization and completeness. If so, the familiarity they brought to the training sessions may have influenced their learning experience and could be confounding the finding that process-focused categories resulted in stronger category knowledge than objectivefocused categories.

433

Research on how the business-risk audit methodology influences judgment has just begun to examine the effect of using alternative task structures for providing decision information to auditors (Ballou & Heitger, 2003). In a study where novices evaluated information about tests of controls, a process-focused task structure produced different reliability judgments than an objectivefocused task structure (O’Donnell, 2003). In a study where experienced auditors evaluated a strategic auditing case, those who used a businessprocess-focused task structure recognized more seeded risk factors than auditors who used the financial-statement-category task structure provided by their firm’s audit support software (O’Donnell & Schultz, 2003). Taken together, the findings of this study and the research described above suggest that delivering decision information in a way that makes it easier for auditors to construct a story line can enhance decision performance. The challenge for future research is to determine (a) why the process focus influences auditor cognition, (b) when alternative task structures can help and hinder auditor judgment, and (c) how to optimize the benefits of decision support systems that are organized around a flow of events rather than a listing of attributes.

Acknowledgements We are grateful for the helpful suggestions we received from Steve Kaplan, Fred Phillips and from participants at the 2002 ABO Section Research Conference and the 2003 Auditing Section Mid-Year Meeting of the American Accounting Association.

References American Institute of Certified Public Accountants, Statement on Auditing Standards No. 78: Consideration of Internal Control in a Financial Statement Audit: An Amendment to SAS No. 55. New York: American Institute of Certified Public Accountants, Inc. 1995. Anderson, J. (1982). Acquisition of cognitive skill. Psychological Review, 89(4), 396–406.

434

L.S. Kopp, E. O’Donnell / Accounting, Organizations and Society 30 (2005) 423–434

Ballou, B., & Heitger, D., (2003). Judgment and Decision Making Research in a Dynamic Auditing Environment. Working paper, University of Miami, Ohio. Boland, R., Singh, J., Salipante, P., Aram, J., Fay, S., & Kanawattanachai, P. (2001). Knowledge representations and knowledge transfers. Academy of Management Journal, 44(2), 393–417. Bonner, S., Libby, R., & Nelson, M. (1997). Audit category knowledge as a precondition to learning from experience. Accounting, Organizations and Society, 22(5), 387–410. Bonner, S., & Walker, P. (1994). The effects of instruction and experience on the acquisition of auditing knowledge. The Accounting Review, 69(1), 157–178. Curtis, M., & Borthick, F., (2002). Accelerating the acquisition of knowledge structure to improve performance in internal control evaluation. Working paper, University of North Texas. Curtis, M., & Viator, R. (2000). An investigation of multidimensional knowledge structure and computer auditor performance. Auditing: A Journal of Practice and Theory, 19(2), 83–103. Davis, C., & Davis, E. (1998). Audit task and audit methodology as determinants of auditors’ elicitation of alternative knowledge organization structures. Advances in Accounting Behavioral Research, 1, 101–121. Day, E., Arthur, W., & Gettman, D. (2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86(5), 1022–1033. Devine, D., & Kozlowski, S. (1995). Domain-specific knowledge and task characteristics in decision making. Organizational Behavior and Human Decision Processes, 64(3), 294–306. Eilifsen, A., Knechel, W., & Wallage, P. (2001). Application of the business risk audit model: a field study. Accounting Horizons, 15(3), 193–208. Frederick, D. (1991). Auditors’ representation and retrieval of internal control knowledge. The Accounting Review, 66(2), 240–258. Frederick, D., Heiman-Hoffman, V., & Libby, R. (1994). The structure of auditors’ knowledge of financial statement errors. Auditing: A Journal of Practice and Theory, 13(1), 1– 21. Glaser, R. (1990). The reemergence of learning theory within instructional research. American Psychologist, 45(1), 29–39. Kersten, A., & Billman, D. (1997). Event category learning. Journal of Experimental: Psychology: Learning, Memory, and Cognition, 23(6), 638–658.

Kinney, W., (1997). Foreword, auditing organizations through a strategic-systems lens. KPMG Peat Marwick, LLP. Klein, G. (1998). Sources of power Cambridge. MA: The MIT Press. Lemon, M., Tatum, K., & Turley, W. (2000). Developments in the audit methodologies of large accounting firms. London, UK: ABG Professional Information. Libby, R., & Luft, L. (1993). Determinants of judgment performance in accounting settings: ability, knowledge, motivation, and environment. Accounting, Organizations and Society, 18(5), 425–450. Nelson, M. (1993). The effects of error frequency and accounting knowledge on error diagnosis in analytical review. The Accounting Review, 68(4), 803–824. Nelson, M. (1994). The learning and use of frequency information in auditing. Journal of Accounting Literature, 13, 185–211. Nelson, M., Libby, R., & Bonner, S. (1995). Knowledge structure and the estimation of conditional probabilities in audit planning. The Accounting Review, 70(1), 27–47. O’Donnell, E. (2003). The influence of process-focused knowledge acquisition on evaluative judgment during a systems assurance task. International Journal of Accounting Information Systems, 4(2), 115–139. O’Donnell, E., & Schultz, J., (2003). The influence of businessprocess-focused audit support software on analytical procedures judgments. Auditing: A Journal of Practice and Theory, forthcoming. Pennington, N., & Hastie, R. (1986). Evidence evaluation in complex decision making. Journal of Personality and Social Psychology, 51(2), 242–258. Pennington, N., & Hastie, R. (1988). Explanation-based decision making Effects of memory structure on judgement. Journal of Experimental Psychology: Learning, Memory and Cognition, 13(3), 521–533. Ricchiute, D. (1992). Working-paper order effects and auditors’ going-concern decisions. The Accounting Review, 67(1), 46– 58. Schank, R. (1999). Dynamic memory revisited. New York, NY: Cambridge University Press. Sneath, P., & Sokal, R. (1973). Numerical taxonomy. San Francisco: W.H. Freeman. Tubbs, R. (1992). The effect of experience on the auditor’s organization and amount of knowledge. The Accounting Review, 67(4), 783–801.