The effects of feedback type on auditor judgment performance for configural and non-configural tasks

Accounting, Organizations and Society 30 (2005) 537–553 www.elsevier.com/locate/aos

The effects of feedback type on auditor judgment performance for configural and non-configural tasks Patrick W. Leung a, Ken T. Trotman b

b,*

a Hong Kong Polytechnic University, Hong Kong School of Accounting, University of New South Wales, Sydney 2052, Australia

Abstract This study examines the effect of four different types of feedback (outcome, task properties, cognitive, combined task properties/cognitive) on the risk assessment judgments of auditors on two tasks, one requiring configural cue processing and the other not requiring configural cue processing. Task properties feedback and combined feedback improved performance on both tasks. Outcome feedback was more effective for the non-configural task, while cognitive feedback was more effective for the configural task. We also found that, on the configural task, the effect of cognitive feedback was heightened by lower levels of participant self-insight. Finally, combined feedback was particularly effective in transferring knowledge across tasks. These results have direct practical application for the use of feedback in audit practice. From a theoretical perspective they also help to reconcile earlier mixed results in accounting and psychology.
2005 Elsevier Ltd. All rights reserved.

Introduction In this study we examine alternative forms of feedback (outcome, task properties, cognitive, combined task properties/cognitive) in improving judgments in two different types of audit tasks, one requiring configural cue processing and the other not. One of the fundamental aims of judgment and decision making (J/DM) research in auditing is *

Corresponding author. Tel.: +61 2 9385 5831; fax: +61 2 9662 4491. E-mail address: [email protected] (K.T. Trotman).

to improve auditor judgments (Solomon & Shields, 1995). Psychology literature has shown the effect of different forms of feedback on performance of si\mple tasks, and speculated on how these effects may be impacted by specific aspects of task complexity. In auditing, tasks range in complexity across many variables (e.g. Bonner, 1994) including whether the task requires configural processing or not (Brown & Solomon, 1991). The earlier audit judgment studies (e.g. Ashton, 1974), which examined auditor performance on tasks not requiring configural processing of cues, showed that performance (usually measured by consensus) was higher than typically reported in

0361-3682/$ - see front matter
2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.aos.2004.11.003

538

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

J/DM studies, but that there was considerable room for improvement. 1 Even where configural cue processing is considered appropriate, Brown and Solomon (1990, 1991) found that a substantial number of participants did not process configurally. This leaves scope for substantial improvement in their judgments. Three of the most frequently studied forms of feedback in the psychology literature are outcome feedback, task properties feedback and cognitive feedback (see Balzer et al., 1994; Luckett & Eggleton, 1991 for reviews). Outcome feedback (OFB) provides information on the ÔcorrectnessÕ of each judgment outcome, task properties feedback (TPF) provides information on the optimal policy in the environment, and cognitive feedback (CFB) provides information about an individualÕs own judgment policy (Balzer, Doherty, & OÕConnor, 1989; Hammond, Stewart, Brehmer, & Steinmann, 1975). There have been extensive studies on these three types of feedback in psychology (see reviews by Balzer et al., 1989, 1994, 1992; Luckett & Eggleton, 1991; Kluger & DeNisi, 1996). Overall, the results on OFB have been mixed (Kluger & DeNisi, 1996), TPF has been found to be effective across a wide variety of studies and CFB 2 feedback has generally not been shown to be effective in increasing performance (see Balzer et al., 1989). However task complexity, and its impact on feedback stud-

1 For example, Solomon and Shields (1995) noted that consensus was examined in 22 policy capturing studies and the unweighted mean correlation was 0.59. 2 Balzer et al. (1989, 1992) refer to cognitive feedback as having a number of components including information about the task (TI) and information about the participantÕs cognitive system (CI). TI includes reactions between the cues and the criterion, information about the criterion or the cues themselves or both. TI includes task predictability and information about the functional form relating the criterion to the cues. CI provides information about the subjectÕs cognitive strategy reflecting relations between the cues and the participantÕs judgments. CI can include information about judgment consistency and the weights and functional form relating each cue to each judgment. In our paper TPF is the equivalent of TI and CPF is the equivalent of CI. In both cases we provided information on cue weights and functional form (see research methods).

ies, has largely been ignored in the experimental psychology literature (Balzer et al., 1989, 1992; Hirst, Luckett, & Trotman, 1999; Kluger & DeNisi, 1996). This has resulted in calls to consider the use of cognitive feedback, in particular, in more complex tasks. Balzer et al. note that the laboratory tests investigating CFB have been of lower complexity and that as Ômany tasks in the world include a large number of cues, non-linear cue criterion relationships, and complex aggregation rulesÕ (p. 428), further investigation of CFB in more complex environments would be prudent. Similarly, Balzer et al. (1992) note that while CFB Ôwas not necessary to improve performance in the present task, perhaps it would be beneficial in more ‘‘complicated’’ tasks with highly interrelated cues, non-linear relationships between cues and criterion, environments that are highly configural and so onÕ. They also suggest that this research should use Ôreal world judges and tasksÕ. Our study extends previous feedback research in the following ways. First, while there has been a significant amount of research in auditing on ways to improve auditor judgments via such means as the review process and decision aids (Messier, 1995; Rich, Solomon, & Trotman, 1997a, Rich, Solomon, & Trotman, 1997b), little attention in the audit literature has been given to feedback, which psychological research shows has the potential to improve performance. One exception is Bonner and Walker (1994) who examined the impact of OFB and TPF feedback on studentsÕ error predictions on a task that did not require configural cue processing. Our aim in this study is to examine the impact of the different types of feedback across tasks requiring/not requiring configural cue processing. While the importance of this task dimension has been raised in the psychological literature, it has not been addressed experimentally. Second we extend the psychology and auditing literatures by providing cognitive feedback to participants who perform a task that requires configural or non-configural processing. While the importance of this has been noted above, the psychology literature has not examined cognitive feedback in tasks requiring configural processing and the auditing literature has not considered cog-

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

nitive feedback in any situation. In practice, auditing tasks include tasks that require configural cue processing and tasks that donÕt. 3 By examining CFB, we consider a potential way to improve auditor judgments that is not presently used in practice, but which has the potential to be incorporated in staff training or a decision aid. Third, the study extends previous psychology and accounting research by investigating the effect of self-insight in moderating the effectiveness of CFB. Knowledge of this relationship is important to audit firms as the introduction of any form of cognitive feedback (e.g. via training) may be most useful for levels within the firm where self-insight is expected or known to be lower. Fourth, the study extends both the accounting and psychology literatures by investigating the transfer of learning, i.e. where learning from one task enhances performance in different contexts/ tasks (Salomon & Perkins, 1989). This is important, given that the goals of training in professional settings are to increase the long-term level of post-training performance and to transfer learning to related tasks and altered contexts (Schmidt & Bjork, 1992).

Hypothesis development Outcome feedback Overall, there have been inconsistent and inconclusive results in the accounting and psychology literatures on the effectiveness of OFB in improving judgment performance. Kluger and DeNisi (1996), after an extensive review of the psychology literature and a meta-analysis on feedback intervention (their term for OFB), concluded that OFB, on average, improved judgment performance. However, over one third of the studies reviewed showed decreased performance. Kluger and DeNisi suggested that task attributes might

3 For example, Brown and Solomon (1991) note that Ôrisk of misstatementÕ tasks may involve substitutable audit procedures or not. For the former, configural cue processing is appropriate and for the latter it is not.

539

moderate the effect, and that simple-task performance benefited from OFB more than complextask performance. Others have speculated that OFB appears to be effective where the task is highly predictable and relatively simple, and where the relationship between the cues and the criterion is largely linear (Hirst & Luckett, 1992; Hirst et al., 1999). Consistent with this, Bonner and Walker (1994) suggest two conditions for OFB to work. First, the task must be sufficiently simple to enable participants to work backwards from the outcome and develop correct explanations. This requires the task to have a small number of cues and high predictability, or for it to be simplified in other ways. Second, participants need to have some relevant prior knowledge before receiving the OFB. All participants in our present study met the second condition, but differed on the first condition depending on which task they completed. We suggest that participants completing the configural task would have reduced ability to work backwards from the outcome because of the added complexity (Bonner & Walker, 1994; Hirst & Luckett, 1992) resulting from the relationship between the cues and criterion being non-linear. However, when this relationship is linear it will be much easier to work backwards from the criterion. This suggests the following hypothesis. H1a OFB is more effective in improving judgment performance in a non-configural task than a configural task.

Task properties feedback Most studies in both psychology and accounting found TPF effective in improving judgment in a wide variety of tasks (see, for example, Balzer et al., 1989; Tuttle & Stocks, 1998). This is because TPF reduces the amount of task ambiguity by providing individuals with knowledge about cue importance or knowledge about the functional form of the correct model (Hammond, Rohrbaugh, Mumpower, & Adelman, 1977). The effectiveness of TPF is robust across a wide range of

540

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

tasks and remains valid regardless of the formal characteristics of tasks (such as the number of cues and their relationship to the criterion) and the substantive characteristics of tasks (such as the content of the task). Consequently, we have no a priori reason to suggest a difference and we test the following null hypothesis. H1b TPF is equally effective in improving judgment performance in tasks involving both configural and non-configural processing.

Cognitive feedback Balzer et al. (1989) note that one reason for the original development of CFB was that there was evidence that individuals lacked insight into their judgment policies. CFB was aimed at providing this information. However, the tasks used in psychology to test the effectiveness of CFB have often been very simple. As a consequence, it is not surprising that CFB has generally not improved performance, as the level of self-insight on these tasks is likely to be above average. Balzer et al. suggest that, in a complex task, individuals may not be aware of which cues are important to their judgments and may be unaware of the functional forms relating their judgments to each cue. They argue that the provision of CFB would increase their awareness of their own judgment policies and as a result, CFB would be more effective in improving judgment policies on more complex tasks. Auditing policy capturing studies have found reasonably high levels of self-insight (Solomon & Shields, 1995) and therefore we would expect little improvement for non-configural tasks. However, for configural tasks the requisite knowledge is higher (Brown & Solomon, 1991) and there is likely to be variation in who knows how they are weighting cues and combinations of cues. This provides the opportunity for CFB to be effective on a task that requires configural cue processing. H1c CFB is more effective in improving judgment performance in a configural task than a nonconfigural task.

Combined feedback Previous studies have generally found no advantage for providing a combination of TPF and CFP (CombFB) over TPF alone (Balzer et al., 1994; Balzer et al., 1992; Nystedt & Magnusson, 1973; Steinmann, 1974). One explanation is that, if participants are aware of their judgment policies, TPF alone would allow them to compensate for inappropriate weighting strategies, particularly in low task complexity cases in which cue weights differ substantially from one another. Balzer et al. (1992) suggested that CombFB may be more effective in more complex tasks and environments that are highly configural. Hoffman, Earle, and Slovic (1981) also suggested that while TPF may be sufficient for high-level performance for some tasks, CombFB may be required in tasks of greater complexity. Following these suggestions, we posit that, for a task not requiring configural processing, participants in audit tasks will generally have reasonably high self-insight (Solomon & Shields, 1995) and therefore the inclusion of CFB in addition to TPF is likely to be of limited additional value. On the other hand, while we know less about self-insight on configural tasks, we would expect it to be lower, providing an opportunity for CFB to have an additional positive impact on performance. H1d CombFB is more effective in improving judgment performance in a configural processing task than in a non-configural processing task.

Self-insight Self-insight is a potentially important variable that has not been examined in feedback research. This is surprising as one of the reasons for CFB being developed was evidence suggesting that individuals lacked self-insight (Balzer et al., 1989). Participants often describe their policies inaccurately (Balke, Hammond, & Meyer, 1973; Reilly & Doherty, 1992; Summers, Taliaferro, & Fletcher,

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

1970), or overestimate the number of cues that are important to them (McKenzie, 1997; Slovic & Lichtenstein, 1971). As CFB provides feedback to participants about their functional forms for combining cues and their cue weights, self-insight can moderate the effectiveness of CFB in improving judgment. Participants with very high self-insight would be expected to know their own functional forms and cue weights in making judgments even without CFB. Therefore, CFB is likely to be of limited value to these participants. On the other hand, participants with low self-insight may not know the functional form or weights they put on each cue in their models. For this group of participants, CFB provides additional information on their judgment models and cue weights. This additional information allows them to make appropriate adjustments based on the information in the CFB. This suggests the following hypothesis. H2 The effect of CFB in improving judgment performance would be higher for participants with low self-insight than for those with high self-insight.

541

the opportunity to learn the configural nature of the cues. When they are required to perform other tasks that require configural cue processing, they should find it easier to transfer the knowledge and apply it to the new task than participants who did not receive this feedback. Thus we would expect TPF and CombFB participants who completed Task 1A (the initial configural cue processing task) to perform better on a subsequent configural task (Task 2) than participants who receive no feedback (NFB) on Task 1A. Predictions are not made with respect to OFB and CFB as their feedback may have given participants some useful information on the configural nature of the task. This suggests the following hypotheses. H3a Participants who receive TPF in Task 1A will outperform those participants who receive NFB for Task 1A on a new configural task (Task 2). H3b Participants who receive CombFB in Task 1A will outperform those participants who receive NFB for Task 1A on a new configural task (Task 2).

Intertask learning

Research methods

An important aspect of learning in professional settings is the transfer of learning to other tasks (or intertask learning). While the importance of intertask learning has been recognised (e.g. Balzer et al., 1989; DeCorte, 1996; Salomon & Perkins, 1989), previous research (see Steinmann, 1976 for an exception) has not investigated whether different types of feedback on a particular judgment task can be transferred to other tasks. Studies in psychology indicate that learning and the transfer of learning to novel contexts requires the explicit rules of the task to be made known to the individuals (Renkl, 1999). TPF and CombFB (which includes TPF) provide this information, as TPF states the functional form and the cue weights to complete the task. Participants who have practiced a configural task and receive TPF either separately or as part of CombFB have had

Experimental design The experiment was a 5 (feedback treatments) · 2 (configural cue processing demands) · 2 (blocks) full factorial design. Feedback and configural demands were manipulated between-subjects and block was a within-subjects variable. Five feedback treatments were manipulated in the experiment, namely: no feedback (NFB), outcome feedback (OFB), task properties feedback (TPF), cognitive feedback (CFB), and combined feedback (CombFB). NFB participants served as the control groups. Configural cue processing demands was manipulated across two tasks (Task 1A and Task 1B). Task 1A required configural processing and Task 1B did not require configural processing. For each task, there were two blocks, each of 16 cases,

542

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

representing two one-half fractional replications of a 25 factorial design (described below). To assess learning between tasks, all participants subsequently completed Task 2 as the last stage of the experiment. Tasks Task 1A was adapted from Brown and Solomon (1991, Experiment 1). It consisted of a hypothetical audit case involving judgment of the risk of misstatement of an account balance based on part of an audit program for accounts receivable. The case included background information and details on the five audit procedures undertaken. These five procedures were the same five cues manipulated in Brown and Solomon (1991, Experiment 1). 4 The participants were told that they would be given a series of checklists (completed by one of their staff auditors), each representing a partially completed portion of the accounts receivable audit program. They were then asked to assess the misstatement risk of the account balance based on the audit evidence to date. Confirmation of accounts receivable (Procedure A) and verification of sales transactions and observation of subsequent collections (Procedure B) were intended to be substitutable audit procedures with respect to the existence assertion. Procedures C through to E were intended to be accounts receivable audit procedures less directly related to the existence assertion. All five audit procedures (A, B, C, D, and E) were factorially manipulated at two levels each (either Ôcompleted with no exceptions notedÕ or Ônot completed as of this dateÕ). Two different presentation orders of the five procedures were used.

4 The original task in Brown and Solomon (1991) has six cues, one of which was kept constant. This constant cue was incorporated in the background information of our case. Including a constant cue in the task, which would not account for any variance in the subjectsÕ statistical models, would result in subjects assigning a certain weight to that cue when they were asked to externalise the importance of each cue in their judgments. This would unfairly understate their self- insight. Therefore, it was decided to include only the five manipulated cues in Brown and Solomon (1991, Experiment 1) task.

The substitution effect 5 requires auditors to evaluate the results of audit procedures in a particular pattern or form of configuration—a compensatory-form ordinal interaction (Brown & Solomon, 1991). That is, auditors need to recognise that the two tests are substitutable, with the existence of one compensating for the absence of the other. The requirement to configurally process evidence (cues) makes the judgment more complex than a task which only requires linear processing of evidence (cues), such as Task 1B (see discussion below). Task 1B is similar to Task 1A, with only one difference—Procedure B ‘‘verify sales transactions for and observe collections of a 95% reliability sample of year end A/R’’ was replaced by another audit procedure ‘‘verify adequacy of allowance for bad debts using analytical procedures’’. This new audit procedure is not directed at the existence assertion and is not intended to be substitutable for Procedure A. Task 1B only requires auditors to make a risk assessment by a linear combination of audit cues. Task 2, which was used to assess inter-task learning, was adopted from Brown and Solomon (1990). Participants were presented with a series of 16 checklists of cash disbursement controls by a staff auditor and asked to assess the risk that cash disbursements could be materially misstated as a result of checks being written and/or disbursed for improper purposes. The cues were selected so that configural cue processing was appropriate.

5 An effective and commonly used procedure for the existence assertion of the accounts receivable balance is confirmation of individual account balances with debtors. If the confirmation requests were returned from debtors without exceptions noted, the auditor should assess that the risk of misstatement would be low. However, if a sufficient number of confirmation requests were not returned or returned with exceptions noted, the auditor may have to expand his/her substantive audit procedures, such as via verification of sales transactions and observation of subsequent collection, to collect further evidence. In other words, confirmation can be considered as the primary audit procedure and verification of sales transactions and observation of subsequent collection may be viewed as alternative or substitute procedures.

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Participants Participants were 224 Big-5 audit seniors with two to five years of audit experience in Hong Kong. They were personally contacted and accepted the invitation to participate in the experiment and were given a small souvenir and a ticket in a lottery for ten cash prizes (about US$130 each). 6 The average audit experience of participants is 2.9 years. Procedures Participants were randomly assigned to one of the ten treatment groups (5 feedback · 2 tasks) when they agreed to participate in the experiment. A convenient time and date was scheduled for each subject to come to the university laboratory. The study was programmed using IBM personal computers, in order to standardise instructions. Only one or two participants were scheduled for each session. All participants (with the exception of those in the NFB and OFB groups) completed six stages: (1) trial stage; (2) first block of 16 cases of Task 1A or Task 1B; (3) self-insight and demographic data questions and a filler task; (4) provision of feedback then second block of 16 cases of Task 1A or Task 1B; (5) manipulation checks (participants in the NFB and OFB groups did not complete this stage as they did not receive any feedback information on cue weightings); (6) completion of Task 2. The details of each stage are outlined below. Stage 1 aimed to familiarise participants with the operations of the computer system. It included practice cases where each subject evaluated the risk involved in two payroll cases which were not related to the subsequent experimental tasks in the study.

6

Originally there were 236 subjects, but the responses from five subjects were not useable because of diskette problems. In addition, the responses of seven subjects were excluded because they failed the manipulation check in recalling and ranking the three most important cues in the feedback presented to them. As a result, there were 224 valid responses (113 female and 111 male auditors).

543

In stage 2, participants completed the first block of 16 cases of Task 1A or Task 1B, depending on the treatment group. During the experiment, they read the instructions on the screen and clicked on their risk assessment for each case. Participants in the OFB treatments received OFB after each of the 16 cases in the first block while participants in the other treatment groups did not receive any type of feedback at this stage. Stage 3 solicited participantsÕ self-insight into their own judgment models in the first block of trials. They rated the importance of each cue from 0 (not important at all) to 100 (very important). ParticipantsÕ demographic data were obtained in this stage. All participants were asked to complete an unrelated filler task so that sufficient time was available (normally about 15 min) to model participantsÕ judgment policies using their responses in the first block of 16 cases, in order to provide CFB to participants in the CFB and CombFB groups. At the start of Stage 4, TPF, CFB and CombFB was provided to the respective groups before they started the second block of 16 cases. All participants completed the second block of 16 cases of Task 1A or Task 1B. There was no further OFB provided to the OFB groups in this stage. Stage 5 included the following manipulation checks. Participants receiving TPF, CFB and CombFB were asked to recall the feedback provided to them at the start of Stage 4, and to rank the importance attached to the three most important audit procedures by the partner (themselves). The ranking used was simple (1 = most important; 2 = second important; 3 = third important). Participants in the NFB and OFB groups were not required to complete this part as they did not receive this information. In stage 6, participants completed Task 2 in order to assess whether the learning in Task 1A and Task 1B under different feedback conditions could be transferred to another task that requires configural cue processing. Here we only examine the transfer of learning to the configural task, Task 1A. Presentation of feedback information The following information was obtained as the basis for providing feedback and evaluating

544

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

performance. For Task 1A, we asked five Big-5 partners to complete the task (32 cases) and selected one of the partnersÕ judgments as the benchmark. This partnerÕs judgments were selected because his judgments were close to the average of all the partners, and he exhibited the most configural cue processing in the desired direction among the audit partners. For Task 1B, we asked another three Big-5 partners to complete the task (32 cases) and we chose the partnerÕs judgments that were closest to the average of their judgments. These two partnerÕs judgments were used to provide OFB, TPF, CFB and CombFB in Tasks 1A and 1B respectively, and were also used as the benchmark to compare with the judgments of participants completing Tasks 1A and 1B. OFB was provided to participants in the OFB treatment groups after each case in the first block of 16 cases. After participants made a judgment of the risk of misstatement for each case and clicked on their answer, a message with the partnerÕs risk assessment was shown on the screen immediately below their answer. Where the partnerÕs answer was 60, the message read: ‘‘The risk assessment of this case made by a very experienced partner, who is recognised as an expert in this audit task, is 60’’. Therefore, the subject had his/her own judgment together with the partnerÕs judgment on that case on the one screen. TPF, CFB, and CombFB were presented to participants belonging to the respective treatment groups before they started the second block of 16 trials. Participants were allowed to study the information for as long as they liked. They then proceeded to the second block of 16 cases. TPF refers to information about the relationship between the cues (audit procedures) and the criterion (the correct risk judgment). In this study, TPF was information about the importance of each cue and its combination with other cues in influencing the risk assessment. Following Brown and Solomon (1990, 1991), the importance was defined as the influence of each cue on the amount of variation in the partnerÕs risk assessments in the respective task. For Task 1A, two of the cues (Procedures A and B) are expected to relate to the risk assessment in a compensatory-form ordinal interaction

(Brown & Solomon, 1991). To make sure that participants understood the feedback information, both a bar chart and a diagram were presented to them, together with a brief narrative description to explain the meaning of the interaction effect of these two cues. The diagrammatic information and the related descriptions are shown in Exhibit 1. For Task 1B, the cues related to the risk assessments in a linear manner and there was no interaction effect among the cues. The relative importance of the cues was presented in the form of bar charts with the numerical percentage also shown on the chart. CFB provides information about a subjectÕs cognitive strategy. A commonly used CFB is the relationship between the cues and the subjectÕs judgments (Remus, OÕConnor, & Griggs, 1996). In this study, we modelled participantsÕ judgments based on the first block of 16 cases. 7 Following Brown and Solomon (1991), the weight of each cue (audit procedure) was ‘‘computed by dividing the sum of the squares for the term by the total sum of squares for the model’’ (p. 106). For each subject receiving CFB, their respective judgment weight for each of the five cues (the main effects) was included in the feedback information. For the ten possible two-way interactions, we presented the largest two interactions provided the weight of each interaction was greater than, or equal to 4%. 8 Participants in the CombFB groups were provided with both TPF and CFB. A sample of the CombFB information in bar chart format presented to participants who completed Task 1A is shown in Exhibit 2. Interpretation of the bar charts and diagrammatic representations of

7

As the 16 cases represented a one-half fractional replication of a 25 factorial design, only the main effects and 10 twoway interaction effects were estimated. 8 The decision to present a maximum of two interactions was made to avoid information overload for the subjects. This was particularly the case for subjects in the CombFB groups. Also, it would have been difficult for subjects to interpret many interaction terms. The feedback information in the CFB was very similar to that of TPF with the exception that the description and explanation clearly indicated they were the judgment weights of the subject.

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Judgment variations (out of 100 percent) attached to each audit procedure by the partner

45% 40% 35% 30% 25% 20% 15% 10% 5% 0%

40% 28% 25%

A

B

4%

2%

1%

C

D

E

AxB

Audit Procedures

Interpretation of the bar chart: For the partner, Procedure A was the most important audit procedure, accounting for 40% of all the variations of the risk assessments that the year-end A/R balance could be materially misstated as a result of non-existence (the risk assessments). Procedure B was the second most important, accounting for 28% of all the variations in the partner’s A/R risk assessments. Procedures C, D, and E were less important in the partner’s risk assessments and respectively accounted for 4%, 2%, and 1% of all the variations in the A/R risk assessments. Apart from the importance of Procedure A or Procedure B considered individually, these two procedures considered jointly accounted for another 25% of all the variations in the partner’s A/R risk assessments.

Assessed Risk Level

This joint impact of these two audit procedures in the partner’s risk assessments can be shown diagrammatically as follows: Procedure B

High

Low

S R P

Q l

l

Confirmation of No Confirmation of A/R A/R

Subsequent Verification and Observation of Collection of A/R No Subsequent Verification or Observation of Collection of A/R

Procedure A

Interpretation of the line chart: You will notice that the lowest risk assessment occurs at P (both procedures have been completed), but the risk assessment does not increase much when only one of the procedures is completed (Q and R). However, when neither procedure has been completed (S), the risk assessment is considerably higher. Exhibit 1. TPF information provided to participants completing Task 1A.

545

546

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Judgment variations (out of 100 percent) attached to each audit procedure by the partner and by you

45% 40%

40% 35%

35% 32%

30%

28%

Partner You

25%

25% 20% 15%

11%

10%

8% 6%

6%

4%

5%

2%

1% 2%

0%

0% A

B

C

D

E

AxB BxE

Audit Procedures Exhibit 2. Combined feedback information bar chart provided to participants completing Task 1A.

interactions were in the same format as for the TPF and CFB treatment. Measurement of variables Following Libby and Libby (1989) and Solomon and Shields (1995, p. 14) we used a measure of accuracy in which ÔtruthÕ was surrogated by the response of an expert. 9 ParticipantsÕ risk judgments were compared to the selected audit partnerÕs judgments and the mean absolute error/ difference (MAE) provided the measure of participantsÕ judgment accuracy. The smaller the MAE, the more accurate was the subjectÕs judgment. We also measured judgment achievement (ra), a lens model parameter, which is a commonly used

measure of judgment performance. Judgment achievement was computed for each subject by correlating his/her actual risk assessments with the assessments of the selected partner described above. These ra results are only reported if they are different from the MAE results. Participants were asked to rate the importance of each of the five cues in their risk assessments. The rating used a 100-point scale where 0 was not important at all and 100 was very important. Rating of cue importance has been found to be better than point allocation in terms of being closer to theoretical valid weight (Zhu & Anderson, 1991), easier for participants to perform (Doyle, Green, & Bottomley, 1997), and of high test–retest reliability (Bottomley, Doyle, & Green, 2000). An index of self-insight for each subject was computed by correlating a subjectÕs subjective 10 weights with

9

Libby and Libby (1989) used the consensus of a panel of experts. However, in this study it was not possible to meaningfully aggregate configural judgments, so the judgment of one partner was used, as described above. Participants received feedback from only one partner whose judgments were not extreme. The range of feedback across partners can vary substantially and the impact of the four types of feedback may vary if the feedback came from a different partner with more extreme judgments.

10

For each subject, the relative weight assigned to each audit procedure (cue) was calculated by dividing the points assigned to that cue by the sum of the points awarded to all five cues. That is, the sum of the weights of all five cues should equal one. These are referred to as the subjective weights.

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

their objective 11 weights. For each of the two tasks separately we divided participants into high or low self-insight groups by taking a median split.

Results Manipulation checks After they had completed the second block of trials, participants in the TPF, CFB, and CombFB groups were asked to recall and rank the three most important audit procedures from a list of all five audit procedures (cues). As noted earlier, there were seven participants (three in the TPF groups, one in the CFB groups, and three in the CombFB groups) who failed the manipulation check and their responses were excluded. This left 224 useable responses. The development of H1c and H1d suggests that self-insight is likely to be higher for the non-configural task (Task 1B). Self-insight data was collected in stage 3 and the mean self-insight for Task 1B participants of 0.72 was significantly higher than for Task 1A participants 0.65 (t = 2.525; p = 0.01). Statistical analysis There is no statistically significant difference between the order of cue presentation and judgment performance in each treatment group. Therefore, the participants in both cue presentation orders in each treatment group are combined together in further data analysis. The distribution of participants across these ten treatment groups is quite even, ranging from 21 to 24 (see Table 2, Panel A). Judgment accuracy (using MAE) was analysed with a repeated measures analysis of variance involving a 5 (feedback treatments) · 2 (task com11 An ANOVA model was computed for each subject based on the first block of 16 trials. The variation in judgment attributed to each audit procedure (cue) in a subjectÕs model was computed by dividing the sum of the squares for each cue by the total sum of squares for that model. These are referred to as the objective weights.

547

plexity levels) · 2 (blocks) design. Feedback and task complexity were manipulated between-subjects and block was a within-subjects variable. The results of the overall ANOVA are presented in Table 1 and descriptive statistics of MAE are presented in Table 2, Panel A. The main effects for both the between-subjects variables (task and feedback) were significant. The significant task main effect was caused by the fact that judgment accuracy in Task 1A was significantly lower than that in Task 1B (consistent with Task 1A being a more complex task). Also, the various feedback conditions had significantly different effects on participantsÕ judgment accuracy, as shown below in the multiple comparisons (using Tukey HSD tests) of the different feedback groups in Table 2. Block was the within-subjects variable and was significant in the ANOVA model due to the significant increase in judgment accuracy in the second block of trials. The three-way interaction (Block * Task * Feedback) was significant (p = 0.008). Table 2, Panel A shows the mean group judgment accuracy across blocks and task. TPF, CFB, and CombFB led to significant judgment accuracy improvement in block 2 in the configural task (Task 1A). However, OFB and NFB groups did not significantly improve their performance on this task. For the non-configural task (Task 1B), OFB, TPF, and CombFB led to significant improvement in judgment accuracy in block 2, but this was not the case for NFB and CFB. The pairwise comparisons in Panels B and C of Table 2 show that there was a significant difference in judgment improvement across the two blocks of trials for the different feedback conditions. Different feedback conditions had different effects in improving judgment accuracy across the two tasks. In a task requiring configural cue processing (Task 1A), participants in the TPF, CFB, and CombFB groups had a significantly higher improvement in judgment accuracy than their NFB and OFB counterparts. In a task not requiring configural cue processing (Task 1B), participants in the OFB, TPF, and CombFB groups showed significantly higher improvements in judgment accuracy than participants in the NFB group.

548

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Table 1 Judgment accuracy (MAE) across feedback, task and block SS

df

MS

Panel A: Overall ANOVA Between-subjects Task Feedback Task * feedback Error

186.15 500.84 77.73 6549.53

1 4 4 214

186.15 125.21 19.43 30.61

Within-subjects Block Block * task Block * feedback Block * task * feedback Error

800.92 2.40 293.67 137.11 2076.35

1 1 4 4 214

800.92 2.40 73.42 34.28 9.70

F ratio

6.082 4.091 0.635

0.014 0.003 0.638

82.55 0.25 7.57 3.53

Panel B: Planned contrasts H1a—(OFB across 2 tasks) H1c—(CFB across 2 tasks) H1d—(CombFB across 2 tasks) *

P

0.000 0.620 0.000 0.008

0.010* 0.014* 0.232

6.813 6.093 1.435

Significant at p 6 0.017.

Table 2 Judgment accuracy (MAE) for each feedback treatment in Tasks 1A and 1B Feedback

Task 1A n

Task 1B Mean MAE in 1st Block

Mean MAE in 2nd Block

Diff. in mean MAE

Panel A: Average judgment accuracy (MAE) in both blocks of trials NFB 22 17.67 17.61 0.06 OFB 23 16.39 15.95 0.44 TPF 21 16.61 12.68 3.93** CFB 23 18.37 14.08 4.29** CombFB 22 17.64 12.24 5.40** Average 17.34 14.54 Difference in mean (column row) NFB

OFB

n

Mean MAE in 1st Block

Mean MAE in 2nd Block

Diff. in mean MAE

22 22 24 23 22 2.80

16.42 15.26 15.44 15.90 16.48

16.62 11.39 11.35 14.81 12.67 15.89

0.20 3.87** 4.09** 1.09 3.81** 13.35

TPF

CFB

2.54

CombFB

Panel B: Pairwise comparison using the Tukey HSD test on mean difference in MAE between the first and second blocks of trials for Task 1A NFB 0 0.38 3.87* 4.23** 5.34** * * OFB 0 3.49 3.85 4.96** TPF 0 0.36 1.47 CFB 0 1.11 CombFB 0 Panel C: Pairwise comparison using the Tukey HSD test on difference in MAE between the first and the second blocks of trials for Task 1B NFB 0 4.07* 4.29* 1.29 4.01* OFB 0 0.22 2.78 0.06 TPF 0 3.00 0.28 CFB 0 2.72 CombFB 0 *

Significant at 0.05 level. Significant at 0.01 level.

**

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Test of hypotheses Testing H1a, H1c and H1d involved a planned analysis of the three relevant comparisons using the Bonferroni procedure to control the Type I error rate at 0.05. That is, the significance level for each of the three contrasts is at 0.05/3 = 0.017. Table 1, Panel B, presents the results of the contrast tests associated with these hypotheses. H1a states that OFB is more effective in improving judgment performance in a non-configural task than a configural task. In testing this hypothesis, a contrast was written to compare the judgment accuracy improvement of the OFB groups across the two tasks. As shown in Table 2, judgment accuracy improvement for the OFB groups in Task 1A and Task 1B was 0.44 and 3.87, respectively. As shown in Table 1 Panel B, OFB leads to significant higher improvement in performance in a non-configural task (F = 6.813; p = 0.01), thus supporting H1a. H1b tests the null hypothesis that TPF is equally effective in improving judgment performance in tasks involving both configural and non-configural processing. In testing this hypothesis, a contrast was written to compare the judgment accuracy improvement of the TPF groups across the two tasks. Table 2 shows judgment accuracy improvement for the TPF groups in Task 1A and Task 1B was 3.93 and 4.09, respectively (F = 0.015; p = 0.90). Thus the null hypothesis could not be rejected. 12 H1c predicts that CFB is more effective in improving judgment performance in a configural task than in a non-configural task. To test H1c, a contrast was written to compare the judgment accuracy improvement of the CFB groups across two tasks. Table 2 shows the judgment accuracy improvement in Task 1A was 4.29 and in Task 1B was 1.09 (F = 6.813; p = 0.014). The result supports H1c. H1d predicts that CombFB leads to greater improvement in judgment performance on a confi-

gural task than a non-configural task. A contrast was written to compare the judgment accuracy improvement of the CombFB groups across two tasks. The judgment accuracy improvement in Task 1A and Task 1B was 5.40 and 3.81, respectively (F = 1.435; p = 0.232). The result does not support H1d. H2 predicts that the effect of CFB in improving judgment performance would be higher for participants with low self-insight than for those with high self-insight. To test this hypothesis, two ttests were conducted to compare the judgment accuracy improvement for the CFB groups in the two tasks. We used the Bonferroni procedure to control the Type 1 error rate at the 0.05 level. As shown in Table 3, the improvement in judgment accuracy in Task 1A for the high and low self-insight participants was 2.33 and 6.09 respectively. The difference is significant (t21 = 2.561; p = 0.018). The judgment accuracy improvement in Task 1B for the high and low self-insight participants were 0.62 and 1.51 respectively. The difference was not significant (t21 = 0.679; p = 0.505). The results support H2 for the configural task but not for the non-configural task. H3a and H3b test for learning by examining how performance on Task 2 is impacted by which treatment group the participants were in. A oneway ANOVA comparing judgment performance 13 in Task 2 across the five treatment groups showed significant differences (F = 5.081, p = 0.001). Results of the pairwise multiple comparisons of mean judgment accuracy in Task 2 for the five treatment groups are shown in Table 4. H3a posits that the TPF Task 1A treatment group will outperform the NFB Task 1A treatment group on Task 2. The means are shown in Table 4 (14.64 versus 17.59) and the difference of 2.95 is significant at p = 0.054. H3b posits that the Task 1A CombFB participants would

13

12 As this is a no-effects hypothesis, the inferential error of concern is incorrect acceptance of the null hypothesis.

549

We again asked four different partners to complete Task 2. We chose the judgments of the partner with the highest level of configural processing and whose judgments were around the mean of the four partners. SubjectsÕ judgments were compared to this partnerÕs judgments to calculate MAE.

550

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Table 3 Average judgment accuracy for CFB participants with high and low self-insight Task 1A

Block 1 Block 2 Difference

Task 1B

Participants with low self-insight (n = 12)

Participants with high self-insight (n = 11)

Participants with low self-insight (n = 12)

Participants with high self-insight (n = 11)

20.00 13.91 6.09

16.59 14.26 2.33

17.24 15.73 1.51

14.43 13.81 0.62

Table 4 Mean judgment accuracy (MAE) in Task 2 for five feedback treatments for subjects who completed Task 1A Task 1A Panel A: Mean judgment accuracy Group NFB Mean 17.59

CFB 15.84

CombFB 13.69

Panel B: Pairwise comparisons using Tukey HSD test of the differences in mean judgment accuracy Group NFB OFB TPF CFB NFB 0 0.39 2.95* 1.75 OFB 0 2.56 1.36 TPF 0 1.20 CFB 0 CombFB

CombFB 3.90** 3.51** 0.95 2.15 0

*

OFB 17.20

TPF 14.64

Significant at 0.1 level. Significant at 0.01 level.

**

outperform the Task 1A NFB participants on Task 2. Support for H3b is shown in Table 4 (13.69 versus 17.59) and the difference of 3.90 is significant (p 6 0.01).

Discussion and conclusion This study has extended the auditing and psychology literatures by examining the impact of four different types of feedback on the risk assessments of auditors on two tasks of different demands for configural cue processing. We found that the effectiveness of different feedback treatments varied across the two tasks. Our results show that, while some types of feedback such as task properties feedback (TPF) were useful across both our tasks, the performance of other types of feedback such as outcome feedback (OFB) and cognitive feedback (CFB) depends on the level of

configural cue processing required. Specifically, OFB was more effective in the task where configural cue processing was not required, while CFB was more effective in the task requiring configural cue processing. These results have direct implications for the type of feedback that should be provided in audit practice where both types of task exist. Of particular note, while CFB has not been studied in the audit literature and is not a form of feedback that is presently used in audit practice, our study shows that it does have potential to be an effective means of improving audit judgment performance on more complex tasks. The results support the suggestions in the psychology literature that the nature of the task, including the level of configural cue processing required, will moderate the effects of different types of feedback. In particular, while the psychology literature has not found CFB to be very useful with simple, generic tasks, this study supports the

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

suggestions in that literature that CFB is likely to be useful in more complex tasks. 14 OFB was more effective in improving performance on the non-configural task than the configural task, but the opposite was true for CFB. While the effectiveness of OFB and CFB varied with the task, this was not the case for TPF or CombFB. CombFB feedback improved performance on both tasks due to its TPF component, but the CFB component did not result in an additional improvement on the configural task. We suggest that this result may depend on the degree of difference in self-insight between the two tasks. Future research could address this issue. This study examined the moderating effect of self-insight on the effectiveness of feedback on judgment performance. It was found that a lower degree of self-insight heightened the effect of CFB in improving judgment performance on the configural task. This improvement for the participants with low self-insight appears to result from the gaining of additional knowledge of their own judgment policies. This may partly explain previous findings in psychology studies that CFB was generally not effective. Most psychology studies employ simple generic tasks involving a number of cues not requiring configural processing, where participants are likely to have reasonably high selfinsight and CFB is therefore unlikely to be effective. For our non-configural task, self-insight did not modify the improvements in performance. This may be due to the small range of self-insight on the non-configural task. Future research could examine this impact in situations where the range of self-insight is likely to be larger. The study of the transfer of learning is new to the accounting and auditing literatures, yet it has important implications for the quality of audit work. It is generally agreed that learning is most effective when it can be transferred to other tasks.

14

While there is general recognition in the audit judgment literature of the importance of configural cue processing (Ashton, 1974; Brown & Solomon, 1990, 1991) there has been very limited research on how to increase the level of configural cue processing. While not reported here, our results provide evidence that TPF, CFB, and CombFB can significantly improve auditorsÕ learning of configural cue processing.

551

Our results suggest that on configural tasks the use of CFB and CombFB are effective in promoting intertask learning. In this study, participants who practiced on a configural task (Task 1A) and received CombFB had a significantly better performance in Task 2 (a configural case) when compared with their NFB counterparts. In addition to the practical implications, this suggests that future research examining feedback (and, for that matter, other alternatives for improving judgment performance) could consider both the improvement on the same task and the improvement on other related tasks via the transfer of learning. In addition to the usual limitations of experimental studies, there are a few issues that are specific to this study. In order to provide CFB it was necessary to use a factorial design. Use of a factorial experimental design places a limit on the number of cues that can be used and the number of cue levels that can be manipulated. The use of a factorial design also results in the use of cases that are not entirely representative of the task ecology (Cooksey, 1996; Trotman, 1990). The inclusion of these extreme cases which are not representative of task ecology can impact accuracy and selfinsight measures. Second, self-insight is a measured rather than a manipulated variable and there is the potential for a correlated omitted variable. Third, we only examined the effect of one combination of feedback—TPF and CFB (CombFB). Studying the effect of more combinations of feedback would add to our knowledge and help identify the most efficient and effective combination of feedback conditions to improve auditorsÕ judgment performance. Fourth, while Bonner (1994) provides a very rich model of task complexity, we only manipulated one dimension. Different dimensions of task complexity in BonnerÕs (1994) model can be manipulated to generate tasks of different complexity. Fifth, consistent with previous research studying the effectiveness of different feedback conditions, judgment performance was measured over a short period of time. However, effective learning should last for a long period of time (Christina & Bjork, 1991). Therefore, it is important to investigate the effect of different feedback conditions in improving judgment performance over a longer time period.

552

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553

Acknowledgments We wish to thank Ira Solomon for suggesting the inclusion of the self-insight variable and Amna Khalifa, Gary Monroe, Jennifer Joe, Roger Simnett and William Wright for helpful comments. The financial support of an Australian Research Council Grant to Ken Trotman is gratefully acknowledged. References Ashton, R. H. (1974). An experimental study of internal control judgments. Journal of Accounting Research, 12(1), 143–157. Balke, W. M., Hammond, K. R., & Meyer, G. D. (1973). An alternative approach to labor–management relations. Administrative Science Quarterly, 18, 311–327. Balzer, W. K., Doherty, M. E., & OÕConnor, R. Jr., (1989). Effects of cognitive feedback on performance. Psychological Bulletin, 106(3), 410–433. Balzer, W. K., Hammer, L. B., Sumner, K. E., Birchenough, T. R., Martens, S. P., & Raymark, P. H. (1994). Effects of cognitive feedback components, display format, and elaboration on performance. Organizational Behavior and Human Decision Processes, 58, 369–385. Balzer, W. K., Sulsky, L. M., Hammer, L. B., Sumner, K. E., Birchenough, T. R., Martens, S. P., et al. (1992). Task information, cognitive information, or functional validity information: which components of cognitive feedback affect performance? Organizational Behavior and Human Decision Processes, 53, 35–54. Bonner, S. E. (1994). A model of the effects of audit task complexity. Accounting, Organizations and Society, 19(3), 213–234. Bonner, S. E., & Walker, P. L. (1994). The effects of instruction and experience on the acquisition of auditing knowledge. Accounting Review, 69(1), 157–178. Bottomley, P. A., Doyle, J. R., & Green, R. H. (2000). Testing the reliability of weight elicitation methods: direct rating versus point allocation. Journal of Marketing Research, 37, 508–513. Brown, C. E., & Solomon, I. (1990). Auditor configural information processing in control risk assessment. Auditing: A Journal of Practice and Theory, 9(3), 17–38. Brown, C. E., & Solomon, I. (1991). Configural information processing in auditing: the role of domain-specific knowledge. The Accounting Review, 66(1), 100–119. Christina, R. W., & Bjork, R. A. (1991). Optimizing long-term retention and transfer. In D. Druchman & R. A. Bjork (Eds.), In the mind’s eye: enhancing human performance (pp. 23–56). Washington, DC: National Academy Press. Cooksey, R. W. (1996). Judgment analysis: theory, methods, and applications. San Diego, Cal.: Academic Press, Inc.

DeCorte, E. (1996). Learning theory and instructional science. In P. Reimann & H. Spada (Eds.), Towards an interdisciplinary learning science. Oxford: Elsevier Science Ltd. Doyle, J. R., Green, R. H., & Bottomley, P. A. (1997). Judging relative importance: direct rating and point allocation are not equivalent. Organizational Behavior and Human Decision Processes, 70(1), 65–72. Hammond, K. R., Rohrbaugh, J. W., Mumpower, J. L., & Adelman, L. (1977). Social judgment theory: applications in problem settings. In M. F. Kaplan & S. Schwartz (Eds.), Human judgment and decision processes in applied settings (pp. 1–27). New York: Academic Press. Hammond, K. R., Stewart, T. R., Brehmer, B., & Steinmann, D. O. (1975). Social judgment theory. In M. F. Kaplan & S. Schwartz (Eds.), Human judgment and decision processes (pp. 271–312). New York: Academic Press. Hirst, M. K., & Luckett, P. F. (1992). The relative effectiveness of different types of feedback in performance evaluation. Behavioral Research in Accounting, 4, 1–22. Hirst, M. K., Luckett, P. F., & Trotman, K. T. (1999). Effects of feedback and task predictability on task learning and judgment accuracy. Abacus, 35(3), 286–301. Hoffman, P. J., Earle, T. C., & Slovic, P. (1981). Multidimensional functional learning (MFL) and some new conceptions of feedback. Organizational Behavior and Human Performance, 27, 75–102. Kluger, V. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a metaanalysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. Libby, R., & Libby, P. A. (1989). Expert measurement and mechanical combination in control reliance decisions. The Accounting Review, 64(4), 729–747. Luckett, P. F., & Eggleton, I. R. C. (1991). Feedback and management accounting : a review of research into behavioral consequences. Accounting, Organizations and Society, 16(4), 371–394. McKenzie, C. R. M. (1997). Underweighting alternatives and overconfidence. Organizational Behavior and Human Decision Processes, 71(2), 141–160. Messier, W. F. Jr., (1995). Research in and development of audit decision aids: a review. In A. H. Ashton & R. H. Ashton (Eds.), Judgment and decision making research in accounting and auditing (pp. 207–228). New York: Cambridge University Press. Nystedt, L., & Magnusson, D. (1973). Cue relevance and feedback in a clinical prediction task. Organizational Behavior and Human Performance, 9, 100–109. Reilly, B. A., & Doherty, M. E. (1992). The assessment of selfinsight in judgment policies. Organizational Behavior and Human Decision Processes, 53, 285–309. Remus, W., OÕConnor, M., & Griggs, K. (1996). Does feedback improve the accuracy of recurrent judgmental forecasts?. Organizational Behavior and Human Decision Processes, 66(1), 22–30. Renkl, A. (1999). Learning mathematics from worked-out examples: analyzing and fostering self-explanations.

P.W. Leung, K.T. Trotman / Accounting, Organizations and Society 30 (2005) 537–553 European Journal of Psychology of Education, 14(4), 477–488. Rich, J. R., Solomon, I., & Trotman, K. (1997a). The audit review process: a characterization from the persuasion perspective. Accounting, Organizations and Society, 22(5), 481–505. Rich, J. R., Solomon, I., & Trotman, K. (1997b). Multi-auditor judgment/decision making research: a decade later. Journal of Accounting Literature, 16, 86–126. Salomon, G., & Perkins, D. N. (1989). Rocky roads to transfer: rethinking mechanisms of a neglected phenomenon. Educational Psychologist, 24(2), 113–142. Schmidt, R. A., & Bjork, R. A. (1992). Common principles in three paradigms suggest new concepts for training. Psychological Science, 3(4), 207–217. Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational Behavior and Human Performance, 6, 649–744. Solomon, I., & Shields, M. D. (1995). Judgment and decision research in auditing. In A. H. Ashton & R. H. Ashton

553

(Eds.), Judgment and decision making research in accounting and auditing (pp. 137–175). New York: Cambridge University Press. Steinmann, D. O. (1974). Transfer of lens model training. Organizational Behavior and Human Performance, 12, 1–16. Steinmann, D. O. (1976). The effects of cognitive feedback and task complexity in multiple-cue probability learning. Organizational Behavior and Human Performance, 15, 168–179. Summers, D. A., Taliaferro, J. D., & Fletcher, D. J. (1970). Judgment policy and interpersonal learning. Behavioral Science, 15, 514–521. Trotman, K. T. (1990). The effect of the distribution of cases on judgment consensus. Accounting and Finance, 30(1), 39–50. Tuttle, B., & Stocks, M. H. (1998). The use of outcome feedback and task property information by subjects with accounting—domain knowledge to predict financial distress. Behavioral Research in Accounting, 10, 76–108. Zhu, S.-H., & Anderson, N. H. (1991). Self-estimation of weight parameter in multiattribute analysis. Organizational Behavior and Human Decision Processes, 48, 36–54.