- Email: [email protected]
The effect of outcome knowledge on auditors' judgmental evaluations
Accounting O~unizationsandSociety, Printed in Great Britain
Vol. 17, No. 2, pp. 185-194,
1992.
0361-3682192 115.Ot+.OO Pergamon Press plc
THE EFFECT OF OUTCOME KNOWLEDGE ON AUDITORS’ JUDGMENTAL EVALUATIONS*
JANE L. REIMERS Florida State University and STEPHEN A. BUTLER University of Oklahoma
Abstract This study investigates the effect of outcome knowledge on auditors’ judgments. Specifically, we varied the degree of surprise of the outcome to determine its effect on hindsight judgments in an auditing task. Results indicate that when the outcome is surprising, ax ante judgments are aEected. However, when the outcome is not surprismg, foresight and hindsight evaluations are not signilicantly different. We also tested a simple decision aid designed to reduce the hindsight effect. While the results were weak, we were able to move foresight judgments in the direction of the related hindsight judgment.
Decision research in both psychology and accounting has documented a phenomenon known as the hindsight effect: people tend to judge a possible outcome as ex ante more likely when they know that outcome has occurred. That is, they believe they “knew it all along”. (See Hawkins & Hastie, 1990, for an excellent review of this research.) Because they are involved in judgments and decisions which are often evaluated in hindsight, auditors provide an appropriate and rich context for study of this phenomenon. The purpose of this research is to investigate the effect of outcome knowledge on audit judgments. First, we hypothesize that the size of the hindsight effect depends on the consistency between the evidence and the outcome, that is, the degree of surprise of the outcome. We test for this surprise effect in two different auditing
contexts. In the first, the consistent evidenceoutcome scenario involves a negative outcome. In the second, the consistent evidence-outcome condition involves a positive outcome. We believe that including two different auditing tasks will increase the potential to generalize the results. In addition, a novel strategy for reducing the hindsight effect is introduced and tested with both tasks. We developed a decision aid designed to decrease the surprise of a potential outcome, which has the effect of increasing the consistency between the evidence and the outcome. We hope to reduce the difference between the foresight and hindsight judgments by altering the foresight judgment, rather than the traditional attempts of reducing this difference by altering the hindsight judgment. In this paper, the research questions are
The authors gratefully acknowledge the helpful comments of Robert Ashton, Sarah Bonner, Terry Campbell, Robert Libby, an anonymous reviewer, and participants in the accounting workshop at the Ilniversity of Oklahoma on earlier versions of this paper. l
185
186
J.L. REIMERS and S. A. BUTLER
a d d r e s s e d w i t h t w o e x p e r i m e n t s . T h e first e x p e r i m e n t c o n s i s t e d o f a n i n t e r n a l c o n t r o l task with four outcome knowledge conditions (foresight, c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t , inconsistent evidence-outcome hindsight, and d e c i s i o n - a i d e d f o r e s i g h t ) to i n v e s t i g a t e t h e effect of the degree of surprise of the outcome o n t h e e x t e n t o f t h e h i n d s i g h t effect. In p r i o r s t u d i e s , t h e h i n d s i g h t bias has b e e n d e f i n e d as a s i g n i f i c a n t l y h i g h e r c o n d i t i o n a l p r o b a b i l i t y a s s i g n e d b y s u b j e c t s to an o u t c o m e in t h e h i n d s i g h t c o n d i t i o n (full o u t c o m e k n o w l e d g e ) t h a n in t h e f o r e s i g h t c o n d i t i o n ( n o outcome knowledge). We have operationalized t h e h i n d s i g h t e f f e c t in a s l i g h t l y d i f f e r e n t w a y , to provide increased external validity to our tasks. I n s t e a d o f a p r o b a b i l i t y j u d g m e n t o f an o u t c o m e e v e n t , o u r d e p e n d e n t v a r i a b l e is a more realistic audit judgment. In o u r first e x p e r i m e n t , s u b j e c t s w e r e g i v e n information, about the internal control system of a small company. The information was t h e s a m e f o r all s u b j e c t s . T h e n , t w o g r o u p s of subjects were given information about an o u t c o m e ; o n e g r o u p r e c e i v e d a r a t h e r unusual outcome (embezzlement) while another received a more usual outcome (small errors). Rather than requesting probability judgments for t h e o u t c o m e e v e n t itself, w e a s k e d s u b j e c t s t o g i v e an o v e r a l l e v a l u a t i o n o f t h e i n t e r n a l controls. W h i l e this is n o t p r e c i s e l y t h e s a m e o p e r a t i o n a l i z a t i o n o f t h e h i n d s i g h t e f f e c t as t h e o n e f o u n d in p r i o r p s y c h o l o g y studies,~ it is m o r e r e a l i s t i c for o u r task a n d o u r s u b j e c t s . W h e n a u d i t o r s ' j u d g m e n t s a r e e v a l u a t e d in h i n d s i g h t , they want their work to demonstrate due p r o f e s s i o n a l c a r e . In o u r first e x p e r i m e n t , t h e
e v i d e n c e f o r this is f o u n d in t h e i n t e r n a l c o n t r o l e v a l u a t i o n s , r a t h e r t h a n in a p r o b a b i l i t y j u d g m e n t o f an u n k n o w n o u t c o m e . N o m a t t e r w h a t t h e e v e n t u a l o u t c o m e , it s h o u l d n o t affect auditors' ex ante internal control evaluations. O u r results s h o w e d that a surprising o u t c o m e w a s a s s o c i a t e d w i t h a significant h i n d s i g h t effect. H o w e v e r , an o u t c o m e w h i c h w a s n o t s u r p r i s i n g r e s u l t e d in n o s i g n i f i c a n t h i n d s i g h t effect. T h e d e c i s i o n aid p r o v e d e f f e c t i v e in r e d u c i n g , b u t n o t e l i m i n a t i n g , t h e h i n d s i g h t effect. The second experiment c o n s i s t e d o f an o p i n i o n f o r m u l a t i o n task. 2 It w a s i n c l u d e d to s h o w t h a t t h e s u r p r i s e h i n d s i g h t e f f e c t e x i s t s for p o s i t i v e as w e l l as n e g a t i v e s u r p r i s i n g o u t c o m e s . Again, w e p r o v i d e d o u t c o m e i n f o r m a t i o n b u t d i d n o t u s e t h e p r o b a b i l i t y o f t h a t o u t c o m e as the dependent variable. Rather, we asked s u b j e c t s for t h e l i k e l i h o o d t h a t t h e y w o u l d h a v e q u a l i f i e d t h e o p i n i o n , g i v e n t h e s e t o f facts p r o v i d e d . As in t h e first e x p e r i m e n t , w e s e l e c t e d a more realistic dependent variable. Such a j u d g m e n t s h o u l d n o t b e a f f e c t e d b y t h e outcome. The results confirmed the surprise effect f o u n d in t h e first e x p e r i m e n t . T h e r e m a i n d e r o f t h e p a p e r is o r g a n i z e d as follows. The next section contains a review of t h e r e l e v a n t l i t e r a t u r e a n d t h e h y p o t h e s e s to b e tested. Then, two sections provide the details of the experiments, including the results of each. T h e final s e c t i o n s u m m a r i z e s o u r r e s u l t s a n d provides a discussion of their implications.
P R I O R RESEARCH A N D D E V E L O P M E N T O F HYPOTHESES Many researchers have documented the t e n d e n c y to o v e r e s t i m a t e t h e p r e d i c t a b i l i t y o f
Hawkins & Hastie (1990) discuss "several types of judgment phenomena that are closely related to hindsight effects", and our context is consistent with their examples. The important point is that "all of these phenomena occur because subjects are unable or unwilling to ignore certain types of information when making a judgment that, at least from some perspectives, ought not to be based on the itlformation" (p. 319). 2 We designed this experiment and gathered the data before the Auditing Standards Board revised the standards pertaining to qualified opinions in the United States. Statement on Auditing Standards No. 59 ha~s eliminated the "subject to" opinion, but there is still a significant reporting modification required (Ellingsen e t al., 1989). So while technically this decision is no longer made as stated in our experiment, the substance of the problem still exists. Further, because this decision provides an effective example of a surprise-favorable outcome (in contrast to the surprise-unfavurable outcome from Experiment I ), it is reported here to demonstrate the robusmess of the surprise effect on hindsight judgments.
THE EFFECTOF OUTCOMEKNOWLEDGE reported outcomes (e.g. Fischhoff, 1975, 1977; Hasher et al., 1981). The effect has been d o c u m e n t e d in a variety of contexts, such as nursing (Mitchell & Kalb, 1981), football (Leary, 1981) and medicine (Arkes et al., 1981). The hindsight p h e n o m e n o n has also been demonstrated in accounting contexts. Under specific conditions set forth by Brown & Solomon (1987), performance evaluations based on accounting data were subject to a hindsight effect. In a series of experiments about internal audit decision making, Bailey ( 1 9 8 6 ) found that o u t c o m e knowledge affected the internal control judgments of internal auditors.
Explanation f o r the hindsight effect While many researchers have d o c u m e n t e d the existence of a hindsight effect, few have attempted to explain the reasons for its existence. Fischhoff (1975) suggested that decision makers immediately integrate o u t c o m e knowledge with their previous knowledge in an attempt to form a coherent and consolidated explanation with the information. The suggestion is that this is a cognitive process where actual m e m o r y encodings are altered. It would follow logically, then, that foresight judgments and hindsight judgments would be different. In contrast to this explanation, Hasher et al. ( 1 9 8 1 ) showed that, when new information is added to a previous knowledge state, the previous state remains in memory. However, it appears that subjects must "exert unusual effort to retrieve pre-updated information" (p. 95). A combination of these ideas has recently been used to explain the hindsight effect. Called a cognitive schemata explanation (Brown & ~ l o m o n , 1989), it suggests that the availability of a coherent schema to explain a priori evidence and the revealed outcome dominates any previous schema in memory. Factors such as recency, output interference and availability may influence the relative strengths of the pre- and postoutcome schemata. The crucial point is that there is a difference between the original pre-outcome perspective and the post-outcome perspective. Thus, it is of interest to examine potential causes for increasing or decreasing that difference.
187
Surprise outcome: evidence-outcome inconsistency One interesting result found in prior research is that the strength of the hindsight effect is a function of h o w surprising or improbable the outcome is. In terms of the discussion above, a surprising o u t c o m e (i.e. an o u t c o m e inconsistent with the evidence) causes the posto u t c o m e perspective to be very different from the pre-outcome perspective. For example, Fischhoff ( 1 9 7 7 ) found that unlikely outcomes were associated with a greater hindsight effect than likely outcomes in a study using almanac questions. Arkes et al. ( 1981 ) obtained similar results with a medical judgment task. They found that, if the symptoms were highly predictive of the resulting disease, doctors' judgments showed no hindsight effect. However, if the symptoms were less predictive, doctors' judgments showed a significant hindsight effect. Schkade & Kilbourne ( 1 9 8 9 ) demonstrated this effect in an employee evaluation task. Buchman ( 1 9 8 5 ) tested for this surprise effect in a bankruptcy prediction task. In his experiment, o u t c o m e knowledge (foresight or hindsight) and type of audit opinion (qualified or unqualified) were varied. Hindsight subjects were told that the audited firm did file for bankruptcy. Subjects w h o received the unqualified opinion (bankruptcy less likely) were expected to show a greater hindsight effect than subjects w h o received the going concern qualification (bankruptcy more likely). However, both groups showed a significant hindsight effect of similar magnitude. Although Buchman was unable to demonstrate it, we agree that the size of the hindsight effect should be affected by the degree of surprise of the eventual outcome. With the benefit of hindsight, we suggest that his results could be attributed to several factors. First, he manipulated the evidence, not the outcome. Since financial statements were provided in both o u t c o m e conditions, the opinion portion of the evidence may have simply been redundant. Another explanation consistent with his results is that, given the financial statements he
188
j.L. REIMERSand S. A. BUTLER
used, the surprise was the unqualified o p i n i o n , not the b a n k r u p t c y . In o r d e r to b e t t e r assure a test of the effect of the d e g r e e of surprise of the o u t c o m e , w e m a n i p u l a t e d the o u t c o m e itself. W e u s e d a m a n i p u l a t i o n s c h e c k and empirical k n o w l e d g e a b o u t the o c c u r r e n c e of errors a n d irregularities in d e t e r m i n i n g the d e g r e e of surprise of the o u t c o m e s . O u r first hypothesis suggests the e x i s t e n c e of a significant h i n d s i g h t effect w h e n the e v i d e n c e and o u t c o m e are i n c o n s i s t e n t , i.e. the o u t c o m e is a surprise. O u r s e c o n d hypothesis suggests that, w h e n the e v i d e n c e and o u t c o m e are consistent, h i n d s i g h t j u d g m e n t s will n o t b e significantly different than foresight j u d g m e n t s . H~: Internal control evaluations made with knowledge that is inconsistent with the evidence will be significantly different than judgments. H2: Internal control evaluations made with knowledge that is consistent with the a priori will not be significantly different than judgments.
outcome a priori foresight outcome evidence foresight
D e c i s i o n a i d to reduce the h i n d s i g h t effect A n u m b e r of r e s e a r c h e r s have a t t e m p t e d to r e d u c e or e l i m i n a t e the hindsight effect. Most often, the d e s i r e d result is to make the h i n d s i g h t j u d g m e n t m o r e like the foresight j u d g m e n t . For example, Hasher et al. ( 1 9 8 1 ) f o u n d that certain i n s t r u c t i o n s d i s c r e d i t i n g the o u t c o m e information w e r e successful in reducing the hindsight effect. W h e n told that the outc o m e i n f o r m a t i o n was a mistake, subjects w e r e s o m e t i m e s able to disregard it, e l i m i n a t i n g the hindsight effect. SIovic & Fischhoff ( 1 9 7 7 ) s h o w e d that forcing hindsight subjects to c o n s i d e r o t h e r possible o u t c o m e s in addition to the actual o u t c o m e r e d u c e d the hindsight effect. Unfortunately, most hindsight "debiasing" attempts have not b e e n s u c c e ~ f u l (see Fischhoff, 1982, for an e x t e n s i v e review). In a r e c e n t a t t e m p t to e l i m i n a t e the h i n d s i g h t effect, Arkes et al. ( 1 9 8 8 ) had subjects evaluate the p r o b a b i l i t y of t h r e e o u t c o m e s ( r a t h e r than the usual o n e ) . T w o t r e a t m e n t groups, o n e foresight and o n e hindsight, w e r e asked to give a reason for each possible o u t c o m e . This aid
r e d u c e d the h i n d s i g h t effect: the n u m b e r of s u b j e c t s d e m o n s t r a t i n g the h i n d s i g h t bias was l o w e r in the h i n d s i g h t - r e a s o n s g r o u p than in the h i n d s i g h t g r o u p n o t asked to list reasons. A n o t h e r i n t e r e s t i n g result in this s t u d y was that s u b j e c t s in the foresight-reasons g r o u p gave j u d g m e n t s that w e r e closer to those of the h i n d s i g h t g r o u p than the j u d g m e n t s given by s u b j e c t s in the foresight g r o u p w h o did n o t give reasons. H o w e v e r , Arkes et al. ( 1 9 8 8 ) r e p o r t e d this data w i t h o u t c o m m e n t o r analysis. T h e a p p r o a c h to r e d u c i n g the h i n d s i g h t effect suggested b y that result m a y b e feasible in a u d i t i n g contexts. That is, rather than altering hindsight j u d g m e n t s , it m i g h t b e useful to make the foresight j u d g m e n t m o r e like the h i n d s i g h t j u d g m e n t . Consider, for example, the evaluation of i n t e r n a l controls. If t h e r e is a w e a k n e s s in the i n t e r n a l c o n t r o l s t r u c t u r e that is o v e r l o o k e d by an auditor, the result c o u l d b e a r e d u c t i o n in the efficiency or effectiveness of the audit. If the o u t c o m e of this w e a k n e s s w e r e s o m e e r r o r or irregularity, t h e n c o n s i d e r a t i o n of this o u t c o m e w o u l d b e likely to cause the a u d i t o r to n o t i c e the weakness. Potential o u t c o m e k n o w l e d g e c o u l d cause an a u d i t o r to focus o n an i n t e r n a l c o n t r o l weakness, i n f o r m a t i o n that m i g h t o t h e r w i s e be overlooked. With regard to aids in the e v a l u a t i o n of internal control, Cushing & Loebbecke ( 1 9 8 6 ) f o u n d that m a n y firms have d e v e l o p e d extensive q u e s t i o n n a i r e s a n d checklists for auditors. However, in their sample of the 12 firms w h i c h i n c l u d e d the w o r l d ' s eight largest firms, o n l y o n e p r o v i d e d "an e x t e n s i v e d i s c u s s i o n of, a n d emphasis on, h o w errors a n d irregularities c o u l d be c o n c e a l e d in c o n n e c t i o n w i t h their e v a l u a t i o n of i n t e r n a l c o n t r o l " ( C u s h i n g & Loebbecke, 1986, p. 20). This i n f o r m a t i o n c o u l d r e d u c e the h i n d s i g h t effect b y m a k i n g foresight j u d g m e n t s m o r e like hindsight judgments, rather than the m o r e traditional approach of "debiasing" the h i n d s i g h t j u d g m e n t s w h i c h has had little success. W e w a n t e d to see if a simple c u e that c a p t u r e d the e s s e n c e of that p e r s p e c t i v e , i n c r e a s i n g a t t e n t i o n to possible negative o u t c o m e s , w o u l d r e d u c e o r e l i m i n a t e
THE EFFECTOF OUTCOME KNOWLEDGE the h i n d s i g h t effect that Bailey ( 1 9 8 6 ) f o u n d a n d that w e h o p e d to d e m o n s t r a t e in o u r experiment. Simple d e c i s i o n aids, c a u s i n g d e c i s i o n makers to focus o n i m p o r t a n t i n f o r m a t i o n , have b e e n effective in altering j u d g m e n t s . For example, Levi & Pryor ( 1 9 8 7 ) had s u b j e c t s i m a g i n e b o t h reasons for an o u t c o m e and various o u t c o m e s . T h e y f o u n d that the subjects' self-generated reasons for an o u t c o m e affected the subjects' predictions. Such aids have also altered judgm e n t s in a u d i t i n g c o n t e x t s (e.g. Butler, 1985; K a c h e l m e i e r & Messier, 1990). O u r d e c i s i o n aid was d e s i g n e d to help subjects focus o n an o u t c o m e as well as p o t e n t i a l causes. W e suggest the following hypothesis: H3: When auditors are given a cue to consider a potential outcome, their internal control evaluations will not be significantlydifferent than judgments made by auditors who had actual knowledge that the outcome had occurred. In t e r m s of the d i s c u s s i o n of the e x p l a n a t i o n of the h i n d s i g h t effect, w e w a n t to alter the original p e r s p e c t i v e ( s c h e m a ) w i t h a simple d e c i s i o n aid, m a k i n g it m o r e like the posto u t c o m e perspective. Since it is difficult to influence the h i n d s i g h t e v a l u a t i o n of p e r s o n s w h o did n o t make the original j u d g m e n t , the idea is to make sure that the p e r s o n m a k i n g the original j u d g m e n t c o n s i d e r s the p o t e n t i a l outc o m e so that the original j u d g m e n t is m o r e like the j u d g m e n t of that s e c o n d evaluator. T h e d e c i s i o n aid suggests a c u e that the d e c i s i o n m a k e r does n o t ignore. What the success of s u c h an aid w o u l d argue for, in an applied setting, is m a k i n g sure, with checklists for example, that auditors carefully c o n s i d e r all possibilities before m a k i n g their decisions.
Positive vs negative outcomes In o u r first e x p e r i m e n t , u s i n g an i n t e r n a l control judgment, the c o n d i t i o n with consistent e v i d e n c e a n d o u t c o m e involves a positive o u t c o m e , w h i l e the i n c o n s i s t e n t c o n d i t i o n
189
involves a n e g a t i v e o u t c o m e . Because it has b e e n suggested that n e g a t i v e e v e n t s are given g r e a t e r w e i g h t t h a n positive e v e n t s (e.g. Skowronski & Carlston, 1989), w e tested the first two h y p o t h e s e s in a different c o n t e x t . In the s e c o n d e x p e r i m e n t , w e used an o p i n i o n formulation task w h e r e the consistent evidenceo u t c o m e involved a negative o u t c o m e ; similarly, the surprise c o n d i t i o n i n v o l v e d a positive outcome.
EXPERIMENT 1
Method Subjects. A total of 52 auditors from five i n t e r n a t i o n a l a c c o u n t i n g firms p a r t i c i p a t e d in the first experiment. The auditors w e r e primarily s e n i o r s a n d s u p e r v i s o r s w i t h b e t w e e n t h r e e and s e v e n years of a u d i t i n g e x p e r i e n c e . Mean and m e d i a n years of e x p e r i e n c e w e r e 4.5 and 5 years, respectively. Design a n d procedure. Auditors w e r e rand o m l y assigned to o n e of four o u t c o m e k n o w l e d g e c o n d i t i o n s : foresight ( n o o u t c o m e knowledge), inconsistent evidence-outcome hindsight, c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight, a n d the d e c i s i o n - a i d e d foresight condition. T h e r e w e r e 13 s u b j e c t s in each c o n d i t i o n . Most of the s u b j e c t s c o m p l e t e d the experim e n t a l tasks d u r i n g regular m o n t h l y st'aft meetings with the e x p e r i m e n t e r s present. A small n u m b e r w e r e r e t u r n e d b y mail. 3 T h e i n t e r n a l c o n t r o l e v a l u a t i o n task was m o d e l e d after the o n e used b y Bailey ( 1 9 8 6 ) . Subjects w e r e given b r i e f i n s t r u c t i o n s followed b y five pieces of i n f o r m a t i o n a b o u t the i n t e r n a l c o n t r o l s in the sales/receivables cycle a n d t h e n asked to evaluate the s t r e n g t h of the c o n t r o l s o n a scale from 1 ( v e r y w e a k ) to 7 ( v e r y strong). T h e i n f o r m a t i o n c o n t a i n e d a crucial item that i n d i c a t e d a serious w e a k n e s s in the i n t e r n a l controls: s e p a r a t i o n of duties was violated w i t h r e s p e c t to the a c c o u n t s receivable billing and c o l l e c t i o n functions. In the
No significant differences were fimnd due to method of data collection.
190
J.L. REIMERS and S. A. BUTLER
foresight condition, subjects recorded their b e l i e f s a f t e r r e a d i n g t h e i n t e r n a l c o n t r o l evid e n c e . In t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight condition, subjects were given outc o m e i n f o r m a t i o n b e f o r e t h e y w e r e a s k e d to r e c o r d t h e i r beliefs. A f t e r r e a d i n g t h e i n t e r n a l c o n t r o l data, t h e s u b j e c t s w e r e t o l d that, a f t e r t h e audit, it w a s d i s c o v e r e d t h a t t h e a c c o u n t s receivable clerk had embezzled over $150,0004 b y a l t e r i n g b i l l i n g s a n d d e p o s i t s . In t h e c o n sistent evidence-outcome hindsight condition, t h e s u b j e c t s w e r e t o l d that, after t h e audit, several small (not material) errors were found when the billings and collections were reconc i l e d b y t h e a c c o u n t s r e c e i v a b l e clerk. B o t h h i n d s i g h t c o n d i t i o n s w e r e d e s i g n e d to b r i n g a t t e n t i o n to t h e w e a k n e s s in t h e i n t e r n a l c o n t r o l s t r u c t u r e . A f t e r all t h e i n f o r m a t i o n h a d b e e n p r e s e n t e d , t h e s u b j e c t s w e r e a s k e d to r a t e the internal control structure using only the i n f o r m a t i o n a v a i l a b l e at t h e t i m e o f t h e audit. In all t r e a t m e n t s , t h e i n t e r n a l c o n t r o l s y s t e m w a s h e l d c o n s t a n t . Thus, t h e e v a l u a t i o n o f its strength should not be affected by outcome k n o w l e d g e o f a n y type. 5 The surprising outcome, that over $150,000 h a d b e e n e m b e z z l e d , is a r e l a t i v e l y r a r e e v e n t . Although the description of the controls allows for this p o s s i b i l i t y , t h e p r o b a b i l i t y o f e m b e z z l e m e n t c o n d i t i o n e d o n t h e e v i d e n c e is l o w . W h i l e it is p o s s i b l e , it w o u l d still b e s u r p r i s i n g t o an a u d i t o r t o find s u c h an i r r e g u l a r i t y . O n t h e o t h e r h a n d , w h e n s u c h an i n t e r n a l c o n t r o l w e a k n e s s exists, it is n o t u n u s u a l t o find s m a l l errors. 6
In t h e d e c i s i o n - a i d e d f o r e s i g h t c o n d i t i o n , subjects were given the same internal control i n f o r m a t i o n as o t h e r s u b j e c t s . T h e n , b e f o r e t h e y w e r e a s k e d to r a t e t h e s t r e n g t h o f t h e i n t e r n a l control system, they were asked to think about the possibility of embezzlement and the cont r o l s n e e d e d to p r e v e n t it. Finally, s u b j e c t s rated the strength of the controls on the same 1 to 7 scale. T h e p u r p o s e o f t h e d e c i s i o n aid is to a l t e r t h e e n c o d i n g o r i n t e r p r e t a t i o n of the original evidence, causing subjects to simulate judgment under surprise hindsight. W e t h i n k this t y p e o f c u i n g w o u l d b e e a s y a n d a p p r o p r i a t e in s o m e a u d i t i n g tasks. The dependent variable was the rating of the i n t e r n a l c o n t r o l s y s t e m . T h i s is an e x t e r n a l l y valid judgment; and although the hindsight e f f e c t has t r a d i t i o n a l l y b e e n d e m o n s t r a t e d w i t h a probability judgment (of the outcome), the j u d g m e n t w e u s e d s h o u l d also b e u n a f f e c t e d b y outcome knowledge.
Results T h e m e a n j u d g m e n t w a s 4 . 0 8 for s u b j e c t s in t h e f o r e s i g h t c o n d i t i o n ; 3.92 in t h e c o n s i s t e n t evidence-outcome hindsight condition; and 2 . 6 9 in t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t c o n d i t i o n . T h e m e a n j u d g m e n t for the decision-aided foresight treatment was 3.46. T h e d i f f e r e n c e s in m e a n s a r e all in t h e predicted direction. A 4 × 1 analysis o f v a r i a n c e ( A N O V A ) c o n f i r m e d a s i g n i f i c a n t m a i n e f f e c t d u e to outcome knowledge, F(3, 48) = 4.281, p < 0.01. W e c o m p a r e d t h e m e a n f o r e s i g h t j u d g -
We do not consider the dollar amount here relevant to the treatment of the problem. Since the problem involves fraud, any amount should trigger a similar response. s Complete experimental materials are available upon request. Send requests to J. I.. Reimers, College of Business R53-A, Florida State University, Tallahassee, FI. 32306-1042, U.S.A. 6 Independent auditors were asked to evaluate the surprise of the outcomes. A small sample of auditors was given the evidence and then asked to give the probability that a particular outcome would occur. The surprise (inconsistent evidence-outcome) outcomes were assigned probabilities between O.10 and 0.42, with a mean of 0.25. The not-surprising outcomes (consistent evidence-outcome) were assigned probabilities between 0.60 and 0,95, with a mean of 0.76. Additionally, empirical evidence supports uur manipulations in both cases. Several accounting researchers have ducumented the perception by auditors that irregularities are more unusual than errors (e.g. Coakley & Loebbecke, 1985 ). With respect to the opinion formulation case, we relied on an actual case for our consistent evidence-outcome scenario.
THE EFFECTOF OUTCOMEKNOWLEDGE m e n t ( 4 . 0 8 ) to each of the hindsight c o n d i t i o n s w i t h p l a n n e d c o m p a r i s o n s , 7 finding a significant difference b e t w e e n the m e a n j u d g m e n t in the inconsistent e v i d e n c e - o u t c o m e hindsight condit i o n ( 2 . 6 9 ) a n d the foresight c o n d i t i o n , F (1, 4 8 ) ---- 10.7, p < 0.01. However, the difference b e t w e e n the consistent e v i d e n c e - o u t c o m e hindsight c o n d i t i o n ( 3 . 9 2 ) a n d the foresight condit i o n was n o t significant, F (1, 4 8 ) = 0.14, p > 0.75. This s u p p o r t s the first t w o hypotheses: an o u t c o m e that is i n c o n s i s t e n t w i t h the e v i d e n c e is associated w i t h a hindsight j u d g m e n t w h i c h is significantly different from a foresight judgm e n t , w h i l e a c o n s i s t e n t o u t c o m e is associated w i t h a h i n d s i g h t j u d g m e n t w h i c h is n o t significantly different from a foresight judgment. The third hypothesis, p r e d i c t i n g the usefulness of a simple d e c i s i o n aid in r e d u c i n g the h i n d s i g h t effect, was tested b y c o m p a r i n g the m e a n j u d g m e n t of the decision-aided foresight g r o u p ( 3 . 4 6 ) and that of the i n c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t g r o u p (2.9.6). W e k n o w from the test of the first hypothesis that t h e r e is a significant h i n d s i g h t effect (i.e. 4.08 is different than 2.96). If w e can alter the foresight j u d g m e n t w i t h a d e c i s i o n aid, replacing the m e a n of 4.08 w i t h a n e w m e a n j u d g m e n t of 3.46, does the significant hindsight effect still exist? C o m p a r i n g the m e a n j u d g m e n t of the d e c i s i o n - a i d e d foresight s u b j e c t s to that of the i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight subjects, w e f o u n d that those j u d g m e n t s are n o t significantly different, F (1, 4 8 ) = 2.62, p > 0.10. Thus, the s e c o n d hypothesis is weakly s u p p o r t e d : s a simple d e c i s i o n aid alters foresight j u d g m e n t s in such a way as to r e d u c e the hindsight effect.
191
EXPERIMENT 2
Method Subjects. Fifty-two additional auditors from the same a c c o u n t i n g firms p a r t i c i p a t e d in the s e c o n d e x p e r i m e n t . As in the p r e v i o u s subjectpool, the auditors w e r e s e n i o r s an supervisors w i t h average e x p e r i e n c e of 4.5 years.
Design and procedure. For the second experim e n t , w e s e l e c t e d an o p i n i o n f o r m u l a t i o n task m o d e l e d after an actual case w h e r e the o p i n i o n had b e e n qualified (Accounting Trends and Techniques, 1984). W e gave subjects inform a t i o n a b o u t a c o m p a n y ' s financial risk resulting from asbestos litigation a n d t h e n asked t h e m to give the likelihood that they w o u l d or w o u l d n o t issue a qualified o p i n i o n . Again, w e used a scale from 1 (definitely w o u l d qualify) to 7 (definitely w o u l d n o t qualify). Like the first e x p e r i m e n t , this e x p e r i m e n t c o n s i s t e d of four treatments: the foresight c o n d i t i o n , the c o n s i s t e n t e v i d e n c e - o u t c o m e c o n d i t i o n , the inconsistent evidence-outcome condition and the decision-aided foresight t r e a t m e n t . Again, w e u s e d t w o different o u t c o m e s , o n e c o n s i s t e n t w i t h the e v i d e n c e a n d the o t h e r inconsistent. The first was that t h e r e was additional litigation c o n c e r n i n g the c o l l e c t i o n of the i n s u r a n c e settlement. This o u t c o m e k n o w l e d g e was given after o t h e r i n f o r m a t i o n that was c o n s i s t e n t with such an o u t c o m e . T h e o t h e r o u t c o m e knowledge c o n d i t i o n c o n s i s t e d of an o u t c o m e that was m o r e surprising, given the evidence. G i v e n the pessimistic e v i d e n c e a b o u t the t r o u b l e s of a c o m p a n y i n v o l v e d in asbestos litigation, the surprising o u t c o m e was that t h e r e was n o
7 Comparisons were planned prior to inspecting the data and calculated according to Wirier (1971, pp. 170-175).
HActually, the mean of the decision aid judgments was not significantlydifferent from either the foresight or surprising hindsight judgments. So while we were able to move the judgment in the right direction, we did not eliminate the hindsight effect.
192
J.I.. REIMERS and S. A. BUTLER
f u r t h e r litigation. This s i t u a t i o n is realistic and is o n e w h i c h allows us to r e v e r s e the consistentp o s i t i v e o u t c o m e r e l a t i o n s h i p f o u n d in t h e first e x p e r i m e n t . In this case, u n l i k e that u s e d in t h e first e x p e r i m e n t , t h e o u t c o m e c o n s i s t e n t w i t h t h e e v i d e n c e is n e g a t i v e a n d t h e s u r p r i s i n g o u t c o m e is positive. T h e d e c i s i o n - a i d e d f o r e s i g h t t r e a t m e n t for this e x p e r i m e n t w a s similar to that o f t h e first e x p e r i m e n t . Before m a k i n g a j u d g m e n t , subjects in this t r e a t m e n t g r o u p w e r e asked to consider the worst possible outcome. Because t h e u n f a v o r a b l e o u t c o m e is n o t s u r p r i s i n g in this c a s e a n d w e e x p e c t a u d i t o r s in t h e u n a i d e d f o r e s i g h t c o n d i t i o n to r e c o g n i z e this, a d e c i s i o n aid in this c a s e is n o t e x p e c t e d to affect t h e auditors' judgments. Numerous auditing studies have s h o w n that a u d i t o r s c o n s i s t e n t l y w e i g h t n e g a t i v e e v i d e n c e m o r e h e a v i l y than p o s i t i v e e v i d e n c e ( A s h t o n & Ashton, 1988; Butt & C a m p b e l l , 1989; Kida, 1984). Since t h e p o s s i b l e n e g a t i v e o u t c o m e is so crucial, a d e c i s i o n aid to remove the surprise of a favorable outcome w o u l d n o t b e e x p e c t e d to h a v e any significant effect. O u r d e c i s i o n aid t r e a t m e n t w a s i n c l u d e d in t h e s t u d y for c o m p l e t e n e s s . S u b j e c t s r a t e d t h e l i k e l i h o o d o f t h e firm r e c e i v i n g a qualified o p i n i o n , o n a scale f r o m 1 ( d e f i n i t e l y w o u l d qualify) to 7 ( d e f i n i t e l y w o u l d n o t qualify). O n c e again o u r d e p e n d e n t variable was not a probability judgment of the o u t c o m e . Rather, w e s e l e c t e d a m o r e e x t e r n a l l y valid j u d g m e n t . As in t h e p r e v i o u s task, outc o m e k n o w l e d g e s h o u l d h a v e n o effect o n auditors' judgments.
R e s u Its
T h e r e s u l t s s u p p o r t t h e first t w o h y p o t h e s e s : the inconsistent evidence-outcome hindsight t r e a t m e n t was associated w i t h a larger hindsight effect than t h e c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight treatment. The foresight condition y i e l d e d a m e a n r e s p o n s e o f 2.85, w h i l e t h e m e a n r e s p o n s e s o f s u b j e c t s in t h e consistent and inconsistent evidence-outcome conditions w e r e 2.23 a n d 4.08, r e s p e c t i v e l y . T h e
m e a n j u d g m e n t for t h e d e c i s i o n - a i d e d foresight t r e a t m e n t w a s 2.69. A 4 x 1 ANOVA ( o u t c o m e k n o w l e d g e × t a s k ) c o n f i r m e d that t h e r e w a s a significant m a i n effect for t h e o u t c o m e k n o w l e d g e c o n d i tion, F (3, 4 8 ) ---- 6.284, p < 0.01. P l a n n e d c o m p a r i s o n s c o n f i r m e d that t h e j u d g m e n t s given b y s u b j e c t s in t h e i n c o n s i s t e n t e v i d e n c e o u t c o m e h i n d s i g h t c o n d i t i o n w e r e significantly different f r o m t h e j u d g m e n t s g i v e n b y t h e foresight subjects, F ( 1, 4 8 ) = 9.85, p < 0.01. H o w e v e r , t h e r e w a s n o statistically significant d i f f e r e n c e b e t w e e n t h e foresight a n d c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t c o n d i t i o n s F (1, 4 8 ) --- 1 . 9 1 , p > 0.10. T h e s e results c o n f i r m o u r first t w o h y p o t h e s e s . Thus, o u r results are c o n s i s t e n t for p o s i t i v e and n e g a t i v e o u t c o m e s . Unlike t h e r e s u l t s from t h e first e x p e r i m e n t , t h e m e a n j u d g m e n t o f t h e d e c i s i o n - a i d e d foresight s u b j e c t s ( 2 . 6 9 ) w a s significantly different from t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight s u b j e c t s ( 4 . 0 8 ) , F (1, 4 8 ) = 12.5, p < 0.01. T h e d e c i s i o n aid m o v e d t h e m e a n judgm e n t in t h e right d i r e c t i o n ; but, as e x p e c t e d , it was n o t a significant m o v e m e n t . As d i s c u s s e d above, this m a y b e due, in part, to a u d i t o r s ' conservative n a t u r e and professional skepticism, s i n c e r e v e r s i n g t h e h i n d s i g h t effect in this case m e a n t m a k i n g a less c o n s e r v a t i v e j u d g m e n t .
SUMMARY AND DISCUSSION W e have d e m o n s t r a t e d that t h e c o n s i s t e n c y between evidence and a subsequent outcome can h a v e an i m p a c t o n t h e d e g r e e o f the h i n d s i g h t effect. Specifically, w e have s h o w n that a u d i t o r s ' j u d g m e n t s r e v e a l a h i n d s i g h t effect w h e n t h e o u t c o m e is i n c o n s i s t e n t w i t h t h e e v i d e n c e , i.e. a surprise. H o w e v e r , w h e n t h e o u t c o m e is c o n s i s t e n t w i t h t h e e v i d e n c e , t h e h i n d s i g h t effect is n o t significant. Similar results h a v e b e e n n o t e d in m o r e g e n e r a l tasks b u t h a v e n o t b e e n p r e v i o u s l y c o n f i r m e d in an a u d i t i n g context. W e h a v e t a k e n t h e r e s e a r c h o n e s t e p further, a r g u i n g that t h e h i n d s i g h t effect m a y b e
THE EFFECT OF OUTCOME KNOWLEDGE reduced by altering the foresight judgment r a t h e r t h a n b y d e b i a s i n g t h e h i n d s i g h t judgm e n t . A u d i t o r s l i v e in a w o r l d w h e r e t h e i r d e c i s i o n s a r e o f t e n j u d g e d in h i n d s i g h t a n d t h e y may be unable to influence those who make the hindsight evaluations. Using a simple decision aid, w e a t t e m p t e d to h a v e a u d i t o r s m e n t a l l y simulate a surprising outcome when they did not have outcome knowledge. While not overw h e l m i n g l y s u c c e s s f u l , t h e r e s u l t s i n d i c a t e that f o r e s i g h t j u d g m e n t s c a n b e m o v e d in t h e d i r e c t i o n o f h i n d s i g h t j u d g m e n t s . M o r e sophistic a t e d aids c o u l d i n c r e a s e t h e e f f e c t i v e n e s s o f this a p p r o a c h . The practical implication of these results r e l a t e s to t h e t y p e s o f o u t c o m e s a u d i t o r s f a c e a n d t h e p o t e n t i a l for r e d u c i n g t h e a m o u n t o f s u r p r i s e in t h o s e o u t c o m e s . B e c a u s e a u d i t o r s w h o a r e e n g a g e d as e x p e r t w i t n e s s e s , for
193
example, are unrelated to the original decision m a k e r s , it is difficult to a l t e r t h e i r d e c i s i o n s . Instead, auditors can specifically consider more a l t e r n a t i v e s w h e n e n g a g e d in t h e o r i g i n a l decision making, thereby reducing the surprise in t h e a c t u a l o u t c o m e . T h e s e r e s u l t s s u g g e s t their decisions would then be more closely aligned w i t h ex p o s t d e c i s i o n makers/evaluators. T h e l i m i t e d s u c c e s s o f t h e d e c i s i o n aid indicates that t h e r e is a p o t e n t i a l g a i n f r o m t h e u s e o f e x t e n s i v e c h e c k l i s t a n d q u e s t i o n n a i r e s a n d that m o r e r e s e a r c h s h o u l d b e d o n e to i n v e s t i g a t e t h e e f f e c t s o f s u c h a u d i t aids. In a d d i t i o n to a u d i t i n g , t h e r e are o t h e r accounting and business contexts where the hindsight effect may be important. Since many b u s i n e s s d e c i s i o n s a r e e v a l u a t e d in h i n d s i g h t , w e h o p e t o e x t e n d this r e s e a r c h t o m o r e general managerial contexts.
BIBLIOGRAPHY Accounting Trends and Techniques (New York: American Institute of Certified Public Accountants, 1984). Arkes, H. R., Wortmann, R. L., Saville, P. & Harkness, A. R., Hindsight Bias Among Physicians Weighing the Likelihood of Diagnoses,Journal of Applied Psychology ( 1981 ) pp. 252-254. Arkes, H. R., Faust, D., Guilmette, T. J. & Hart, K., Eliminating the Hindsight Bias, Journal of Applied Psychology ( 1988 ) pp. 305-307. Ashton, A. H. & Ashton, R. H., Sequential Belief Revision in Auditing, The Accounting Review (October 1988) pp. 623---641. Bailey, C. D., Awfiding Errors in Judgment, The Internal Auditor (1986) pp. 25-28. Brown, C. E. & Solomon, I., Effects of Outcome Information on Evaluations of Managerial Decisions, The Accounting Review (July 1987) pp. 564-577. Brown, C. E. & Solomon, I., An Experimental Investigation of Alternative Explanations for Hindsight Effocts in Agpraisal of Capital Budgeting Decisions (Unpublished manuscript, University of illinois, 1~<)). Buchman, T. A., An Effect of Hindsight on Predicting Bankruptcy with Accounting Information, Accounting~ Organizations and Society (1985) pp. 267-285. Butler, S. A., Application of a Decision Aid in the Judgmental Evaluation of Substantive Test of Details Samples, Journal of Accounting Research (Autumn 1985) pp. 513-526. Butt, J. L. & Campbell, T. L., Information Order and Hypothesis-Testing Strategies, Accounting Organizations and Society (1989) pp. 471-479. Coakley, J. R. & Loebbecke, J. K., "l]ae Expectation of Accounting Errors in Medium-sized Manufacturing Firms, Advances in Accounting (1985). Cushing, B. E. & Loebbecke, J. M., Comparison of Audit Methodologies of Large Accounting Firms (Studies in Accounting Research 26, American Accounting Association, 1986). Ellingsen, J. E., Pany, IC & Fagan, P., SAS No. 59: How to Evaluate Going Concern,Journal of Accountancy (January 1989)pp. 24--31. Fischhoff, B., Hindsight # Foresight: the Effect of Outcome Knowledge on Judgment Under Uncertainty, Journal of Experimental Psychology: Human Perception and Performance ( 1975 ) pp. 288--299.
194
J.L. REIMERSand S. A. BUTLER Fischhoff, B., Perceived Informativeness of Facts, Journal o f Experimental Psychology.. Human Perception and Performance (1977) pp. 349-358. Fischhoff, B., Debiasing, in Kahneman, D., Slovic, P. & Tversky, A. (eds),Judgment under Uncertainty: Heuristics and Biases (New York: Cambridge, 1982). Hasher, L., Attig, M. S. & Alba,J. W., I Knew It All Along: Or, Did I?,Journal o f VerbalLearning and Verbal Behavior ( 1981 ) pp. 86-96. Hawkins, S. A. & Hastie, R., Hindsight: Biased Judgments of Past Events After the Outcomes Are Known, n _ . . ,
,
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
J O U R N A L O F BUSINESS F I N A N C E A N D A C C O U N T I N G
Volume
18, N u m b e r
September
5
1991
Editor: Richard Briston CONTENTS M a c r o e c o n o m i c Factors and t h e UK Stock Market S. P O O N and S. J. TAYLOR G o i n g Private: C o r p o r a t e R e s t r u c t u r i n g U n d e r I n f o r m a t i o n A s y m m e t r y and A g e n c y P r o b l e m s WI SAENG KIM and ESMERALDA O. LYN Financial Ratios and Different Failure P r o c e s s e s ERKKI K. LAITINEN M a n a g e m e n t A n t i c i p a t i o n o f I n t e r e s t Rates: T h e Case o f C o m m e r c i a l Banks JENNIFER FRANCIS A l t e r n a t i v e M o d e l s for Assessing t h e Financial C o n d i t i o n o f Business in G r e e c e PANAYIOTIS T H E O D O S S I O U A g g r e g a t i o n and R e c i p r o c a l S e r v i c e Cost Allocation CHARLES SUTCLIFFE T h e I m p a c t o f S o c i o e c o n o m i c Variables and C r e d i t Ratings o n Municipal B o n d Risk P r e m i a PU LIU and FAZEL J. SEWYED R e d u c i n g t h e U n c e r t a i n t y - I n d u c e d Bias in Capital B u d g e t i n g D e c i s i o n s - - A H u r d l e Rate Approach TARUN K. MUKHERJEE T h e Valuation o f Natural Resources: E v i d e n c e from Oil and Ga~s Pure Plays J O H N L CRAIN and A. M. M. JAMAL T h e Relationship B e t w e e n H o l d i n g Gains and Equity Value: A Neoclassical I n t e r p r e t a t i o n KEVIN C. W. CHEN
Vol. 17, No. 2, pp. 185-194,
1992.
0361-3682192 115.Ot+.OO Pergamon Press plc
THE EFFECT OF OUTCOME KNOWLEDGE ON AUDITORS’ JUDGMENTAL EVALUATIONS*
JANE L. REIMERS Florida State University and STEPHEN A. BUTLER University of Oklahoma
Abstract This study investigates the effect of outcome knowledge on auditors’ judgments. Specifically, we varied the degree of surprise of the outcome to determine its effect on hindsight judgments in an auditing task. Results indicate that when the outcome is surprising, ax ante judgments are aEected. However, when the outcome is not surprismg, foresight and hindsight evaluations are not signilicantly different. We also tested a simple decision aid designed to reduce the hindsight effect. While the results were weak, we were able to move foresight judgments in the direction of the related hindsight judgment.
Decision research in both psychology and accounting has documented a phenomenon known as the hindsight effect: people tend to judge a possible outcome as ex ante more likely when they know that outcome has occurred. That is, they believe they “knew it all along”. (See Hawkins & Hastie, 1990, for an excellent review of this research.) Because they are involved in judgments and decisions which are often evaluated in hindsight, auditors provide an appropriate and rich context for study of this phenomenon. The purpose of this research is to investigate the effect of outcome knowledge on audit judgments. First, we hypothesize that the size of the hindsight effect depends on the consistency between the evidence and the outcome, that is, the degree of surprise of the outcome. We test for this surprise effect in two different auditing
contexts. In the first, the consistent evidenceoutcome scenario involves a negative outcome. In the second, the consistent evidence-outcome condition involves a positive outcome. We believe that including two different auditing tasks will increase the potential to generalize the results. In addition, a novel strategy for reducing the hindsight effect is introduced and tested with both tasks. We developed a decision aid designed to decrease the surprise of a potential outcome, which has the effect of increasing the consistency between the evidence and the outcome. We hope to reduce the difference between the foresight and hindsight judgments by altering the foresight judgment, rather than the traditional attempts of reducing this difference by altering the hindsight judgment. In this paper, the research questions are
The authors gratefully acknowledge the helpful comments of Robert Ashton, Sarah Bonner, Terry Campbell, Robert Libby, an anonymous reviewer, and participants in the accounting workshop at the Ilniversity of Oklahoma on earlier versions of this paper. l
185
186
J.L. REIMERS and S. A. BUTLER
a d d r e s s e d w i t h t w o e x p e r i m e n t s . T h e first e x p e r i m e n t c o n s i s t e d o f a n i n t e r n a l c o n t r o l task with four outcome knowledge conditions (foresight, c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t , inconsistent evidence-outcome hindsight, and d e c i s i o n - a i d e d f o r e s i g h t ) to i n v e s t i g a t e t h e effect of the degree of surprise of the outcome o n t h e e x t e n t o f t h e h i n d s i g h t effect. In p r i o r s t u d i e s , t h e h i n d s i g h t bias has b e e n d e f i n e d as a s i g n i f i c a n t l y h i g h e r c o n d i t i o n a l p r o b a b i l i t y a s s i g n e d b y s u b j e c t s to an o u t c o m e in t h e h i n d s i g h t c o n d i t i o n (full o u t c o m e k n o w l e d g e ) t h a n in t h e f o r e s i g h t c o n d i t i o n ( n o outcome knowledge). We have operationalized t h e h i n d s i g h t e f f e c t in a s l i g h t l y d i f f e r e n t w a y , to provide increased external validity to our tasks. I n s t e a d o f a p r o b a b i l i t y j u d g m e n t o f an o u t c o m e e v e n t , o u r d e p e n d e n t v a r i a b l e is a more realistic audit judgment. In o u r first e x p e r i m e n t , s u b j e c t s w e r e g i v e n information, about the internal control system of a small company. The information was t h e s a m e f o r all s u b j e c t s . T h e n , t w o g r o u p s of subjects were given information about an o u t c o m e ; o n e g r o u p r e c e i v e d a r a t h e r unusual outcome (embezzlement) while another received a more usual outcome (small errors). Rather than requesting probability judgments for t h e o u t c o m e e v e n t itself, w e a s k e d s u b j e c t s t o g i v e an o v e r a l l e v a l u a t i o n o f t h e i n t e r n a l controls. W h i l e this is n o t p r e c i s e l y t h e s a m e o p e r a t i o n a l i z a t i o n o f t h e h i n d s i g h t e f f e c t as t h e o n e f o u n d in p r i o r p s y c h o l o g y studies,~ it is m o r e r e a l i s t i c for o u r task a n d o u r s u b j e c t s . W h e n a u d i t o r s ' j u d g m e n t s a r e e v a l u a t e d in h i n d s i g h t , they want their work to demonstrate due p r o f e s s i o n a l c a r e . In o u r first e x p e r i m e n t , t h e
e v i d e n c e f o r this is f o u n d in t h e i n t e r n a l c o n t r o l e v a l u a t i o n s , r a t h e r t h a n in a p r o b a b i l i t y j u d g m e n t o f an u n k n o w n o u t c o m e . N o m a t t e r w h a t t h e e v e n t u a l o u t c o m e , it s h o u l d n o t affect auditors' ex ante internal control evaluations. O u r results s h o w e d that a surprising o u t c o m e w a s a s s o c i a t e d w i t h a significant h i n d s i g h t effect. H o w e v e r , an o u t c o m e w h i c h w a s n o t s u r p r i s i n g r e s u l t e d in n o s i g n i f i c a n t h i n d s i g h t effect. T h e d e c i s i o n aid p r o v e d e f f e c t i v e in r e d u c i n g , b u t n o t e l i m i n a t i n g , t h e h i n d s i g h t effect. The second experiment c o n s i s t e d o f an o p i n i o n f o r m u l a t i o n task. 2 It w a s i n c l u d e d to s h o w t h a t t h e s u r p r i s e h i n d s i g h t e f f e c t e x i s t s for p o s i t i v e as w e l l as n e g a t i v e s u r p r i s i n g o u t c o m e s . Again, w e p r o v i d e d o u t c o m e i n f o r m a t i o n b u t d i d n o t u s e t h e p r o b a b i l i t y o f t h a t o u t c o m e as the dependent variable. Rather, we asked s u b j e c t s for t h e l i k e l i h o o d t h a t t h e y w o u l d h a v e q u a l i f i e d t h e o p i n i o n , g i v e n t h e s e t o f facts p r o v i d e d . As in t h e first e x p e r i m e n t , w e s e l e c t e d a more realistic dependent variable. Such a j u d g m e n t s h o u l d n o t b e a f f e c t e d b y t h e outcome. The results confirmed the surprise effect f o u n d in t h e first e x p e r i m e n t . T h e r e m a i n d e r o f t h e p a p e r is o r g a n i z e d as follows. The next section contains a review of t h e r e l e v a n t l i t e r a t u r e a n d t h e h y p o t h e s e s to b e tested. Then, two sections provide the details of the experiments, including the results of each. T h e final s e c t i o n s u m m a r i z e s o u r r e s u l t s a n d provides a discussion of their implications.
P R I O R RESEARCH A N D D E V E L O P M E N T O F HYPOTHESES Many researchers have documented the t e n d e n c y to o v e r e s t i m a t e t h e p r e d i c t a b i l i t y o f
Hawkins & Hastie (1990) discuss "several types of judgment phenomena that are closely related to hindsight effects", and our context is consistent with their examples. The important point is that "all of these phenomena occur because subjects are unable or unwilling to ignore certain types of information when making a judgment that, at least from some perspectives, ought not to be based on the itlformation" (p. 319). 2 We designed this experiment and gathered the data before the Auditing Standards Board revised the standards pertaining to qualified opinions in the United States. Statement on Auditing Standards No. 59 ha~s eliminated the "subject to" opinion, but there is still a significant reporting modification required (Ellingsen e t al., 1989). So while technically this decision is no longer made as stated in our experiment, the substance of the problem still exists. Further, because this decision provides an effective example of a surprise-favorable outcome (in contrast to the surprise-unfavurable outcome from Experiment I ), it is reported here to demonstrate the robusmess of the surprise effect on hindsight judgments.
THE EFFECTOF OUTCOMEKNOWLEDGE reported outcomes (e.g. Fischhoff, 1975, 1977; Hasher et al., 1981). The effect has been d o c u m e n t e d in a variety of contexts, such as nursing (Mitchell & Kalb, 1981), football (Leary, 1981) and medicine (Arkes et al., 1981). The hindsight p h e n o m e n o n has also been demonstrated in accounting contexts. Under specific conditions set forth by Brown & Solomon (1987), performance evaluations based on accounting data were subject to a hindsight effect. In a series of experiments about internal audit decision making, Bailey ( 1 9 8 6 ) found that o u t c o m e knowledge affected the internal control judgments of internal auditors.
Explanation f o r the hindsight effect While many researchers have d o c u m e n t e d the existence of a hindsight effect, few have attempted to explain the reasons for its existence. Fischhoff (1975) suggested that decision makers immediately integrate o u t c o m e knowledge with their previous knowledge in an attempt to form a coherent and consolidated explanation with the information. The suggestion is that this is a cognitive process where actual m e m o r y encodings are altered. It would follow logically, then, that foresight judgments and hindsight judgments would be different. In contrast to this explanation, Hasher et al. ( 1 9 8 1 ) showed that, when new information is added to a previous knowledge state, the previous state remains in memory. However, it appears that subjects must "exert unusual effort to retrieve pre-updated information" (p. 95). A combination of these ideas has recently been used to explain the hindsight effect. Called a cognitive schemata explanation (Brown & ~ l o m o n , 1989), it suggests that the availability of a coherent schema to explain a priori evidence and the revealed outcome dominates any previous schema in memory. Factors such as recency, output interference and availability may influence the relative strengths of the pre- and postoutcome schemata. The crucial point is that there is a difference between the original pre-outcome perspective and the post-outcome perspective. Thus, it is of interest to examine potential causes for increasing or decreasing that difference.
187
Surprise outcome: evidence-outcome inconsistency One interesting result found in prior research is that the strength of the hindsight effect is a function of h o w surprising or improbable the outcome is. In terms of the discussion above, a surprising o u t c o m e (i.e. an o u t c o m e inconsistent with the evidence) causes the posto u t c o m e perspective to be very different from the pre-outcome perspective. For example, Fischhoff ( 1 9 7 7 ) found that unlikely outcomes were associated with a greater hindsight effect than likely outcomes in a study using almanac questions. Arkes et al. ( 1981 ) obtained similar results with a medical judgment task. They found that, if the symptoms were highly predictive of the resulting disease, doctors' judgments showed no hindsight effect. However, if the symptoms were less predictive, doctors' judgments showed a significant hindsight effect. Schkade & Kilbourne ( 1 9 8 9 ) demonstrated this effect in an employee evaluation task. Buchman ( 1 9 8 5 ) tested for this surprise effect in a bankruptcy prediction task. In his experiment, o u t c o m e knowledge (foresight or hindsight) and type of audit opinion (qualified or unqualified) were varied. Hindsight subjects were told that the audited firm did file for bankruptcy. Subjects w h o received the unqualified opinion (bankruptcy less likely) were expected to show a greater hindsight effect than subjects w h o received the going concern qualification (bankruptcy more likely). However, both groups showed a significant hindsight effect of similar magnitude. Although Buchman was unable to demonstrate it, we agree that the size of the hindsight effect should be affected by the degree of surprise of the eventual outcome. With the benefit of hindsight, we suggest that his results could be attributed to several factors. First, he manipulated the evidence, not the outcome. Since financial statements were provided in both o u t c o m e conditions, the opinion portion of the evidence may have simply been redundant. Another explanation consistent with his results is that, given the financial statements he
188
j.L. REIMERSand S. A. BUTLER
used, the surprise was the unqualified o p i n i o n , not the b a n k r u p t c y . In o r d e r to b e t t e r assure a test of the effect of the d e g r e e of surprise of the o u t c o m e , w e m a n i p u l a t e d the o u t c o m e itself. W e u s e d a m a n i p u l a t i o n s c h e c k and empirical k n o w l e d g e a b o u t the o c c u r r e n c e of errors a n d irregularities in d e t e r m i n i n g the d e g r e e of surprise of the o u t c o m e s . O u r first hypothesis suggests the e x i s t e n c e of a significant h i n d s i g h t effect w h e n the e v i d e n c e and o u t c o m e are i n c o n s i s t e n t , i.e. the o u t c o m e is a surprise. O u r s e c o n d hypothesis suggests that, w h e n the e v i d e n c e and o u t c o m e are consistent, h i n d s i g h t j u d g m e n t s will n o t b e significantly different than foresight j u d g m e n t s . H~: Internal control evaluations made with knowledge that is inconsistent with the evidence will be significantly different than judgments. H2: Internal control evaluations made with knowledge that is consistent with the a priori will not be significantly different than judgments.
outcome a priori foresight outcome evidence foresight
D e c i s i o n a i d to reduce the h i n d s i g h t effect A n u m b e r of r e s e a r c h e r s have a t t e m p t e d to r e d u c e or e l i m i n a t e the hindsight effect. Most often, the d e s i r e d result is to make the h i n d s i g h t j u d g m e n t m o r e like the foresight j u d g m e n t . For example, Hasher et al. ( 1 9 8 1 ) f o u n d that certain i n s t r u c t i o n s d i s c r e d i t i n g the o u t c o m e information w e r e successful in reducing the hindsight effect. W h e n told that the outc o m e i n f o r m a t i o n was a mistake, subjects w e r e s o m e t i m e s able to disregard it, e l i m i n a t i n g the hindsight effect. SIovic & Fischhoff ( 1 9 7 7 ) s h o w e d that forcing hindsight subjects to c o n s i d e r o t h e r possible o u t c o m e s in addition to the actual o u t c o m e r e d u c e d the hindsight effect. Unfortunately, most hindsight "debiasing" attempts have not b e e n s u c c e ~ f u l (see Fischhoff, 1982, for an e x t e n s i v e review). In a r e c e n t a t t e m p t to e l i m i n a t e the h i n d s i g h t effect, Arkes et al. ( 1 9 8 8 ) had subjects evaluate the p r o b a b i l i t y of t h r e e o u t c o m e s ( r a t h e r than the usual o n e ) . T w o t r e a t m e n t groups, o n e foresight and o n e hindsight, w e r e asked to give a reason for each possible o u t c o m e . This aid
r e d u c e d the h i n d s i g h t effect: the n u m b e r of s u b j e c t s d e m o n s t r a t i n g the h i n d s i g h t bias was l o w e r in the h i n d s i g h t - r e a s o n s g r o u p than in the h i n d s i g h t g r o u p n o t asked to list reasons. A n o t h e r i n t e r e s t i n g result in this s t u d y was that s u b j e c t s in the foresight-reasons g r o u p gave j u d g m e n t s that w e r e closer to those of the h i n d s i g h t g r o u p than the j u d g m e n t s given by s u b j e c t s in the foresight g r o u p w h o did n o t give reasons. H o w e v e r , Arkes et al. ( 1 9 8 8 ) r e p o r t e d this data w i t h o u t c o m m e n t o r analysis. T h e a p p r o a c h to r e d u c i n g the h i n d s i g h t effect suggested b y that result m a y b e feasible in a u d i t i n g contexts. That is, rather than altering hindsight j u d g m e n t s , it m i g h t b e useful to make the foresight j u d g m e n t m o r e like the h i n d s i g h t j u d g m e n t . Consider, for example, the evaluation of i n t e r n a l controls. If t h e r e is a w e a k n e s s in the i n t e r n a l c o n t r o l s t r u c t u r e that is o v e r l o o k e d by an auditor, the result c o u l d b e a r e d u c t i o n in the efficiency or effectiveness of the audit. If the o u t c o m e of this w e a k n e s s w e r e s o m e e r r o r or irregularity, t h e n c o n s i d e r a t i o n of this o u t c o m e w o u l d b e likely to cause the a u d i t o r to n o t i c e the weakness. Potential o u t c o m e k n o w l e d g e c o u l d cause an a u d i t o r to focus o n an i n t e r n a l c o n t r o l weakness, i n f o r m a t i o n that m i g h t o t h e r w i s e be overlooked. With regard to aids in the e v a l u a t i o n of internal control, Cushing & Loebbecke ( 1 9 8 6 ) f o u n d that m a n y firms have d e v e l o p e d extensive q u e s t i o n n a i r e s a n d checklists for auditors. However, in their sample of the 12 firms w h i c h i n c l u d e d the w o r l d ' s eight largest firms, o n l y o n e p r o v i d e d "an e x t e n s i v e d i s c u s s i o n of, a n d emphasis on, h o w errors a n d irregularities c o u l d be c o n c e a l e d in c o n n e c t i o n w i t h their e v a l u a t i o n of i n t e r n a l c o n t r o l " ( C u s h i n g & Loebbecke, 1986, p. 20). This i n f o r m a t i o n c o u l d r e d u c e the h i n d s i g h t effect b y m a k i n g foresight j u d g m e n t s m o r e like hindsight judgments, rather than the m o r e traditional approach of "debiasing" the h i n d s i g h t j u d g m e n t s w h i c h has had little success. W e w a n t e d to see if a simple c u e that c a p t u r e d the e s s e n c e of that p e r s p e c t i v e , i n c r e a s i n g a t t e n t i o n to possible negative o u t c o m e s , w o u l d r e d u c e o r e l i m i n a t e
THE EFFECTOF OUTCOME KNOWLEDGE the h i n d s i g h t effect that Bailey ( 1 9 8 6 ) f o u n d a n d that w e h o p e d to d e m o n s t r a t e in o u r experiment. Simple d e c i s i o n aids, c a u s i n g d e c i s i o n makers to focus o n i m p o r t a n t i n f o r m a t i o n , have b e e n effective in altering j u d g m e n t s . For example, Levi & Pryor ( 1 9 8 7 ) had s u b j e c t s i m a g i n e b o t h reasons for an o u t c o m e and various o u t c o m e s . T h e y f o u n d that the subjects' self-generated reasons for an o u t c o m e affected the subjects' predictions. Such aids have also altered judgm e n t s in a u d i t i n g c o n t e x t s (e.g. Butler, 1985; K a c h e l m e i e r & Messier, 1990). O u r d e c i s i o n aid was d e s i g n e d to help subjects focus o n an o u t c o m e as well as p o t e n t i a l causes. W e suggest the following hypothesis: H3: When auditors are given a cue to consider a potential outcome, their internal control evaluations will not be significantlydifferent than judgments made by auditors who had actual knowledge that the outcome had occurred. In t e r m s of the d i s c u s s i o n of the e x p l a n a t i o n of the h i n d s i g h t effect, w e w a n t to alter the original p e r s p e c t i v e ( s c h e m a ) w i t h a simple d e c i s i o n aid, m a k i n g it m o r e like the posto u t c o m e perspective. Since it is difficult to influence the h i n d s i g h t e v a l u a t i o n of p e r s o n s w h o did n o t make the original j u d g m e n t , the idea is to make sure that the p e r s o n m a k i n g the original j u d g m e n t c o n s i d e r s the p o t e n t i a l outc o m e so that the original j u d g m e n t is m o r e like the j u d g m e n t of that s e c o n d evaluator. T h e d e c i s i o n aid suggests a c u e that the d e c i s i o n m a k e r does n o t ignore. What the success of s u c h an aid w o u l d argue for, in an applied setting, is m a k i n g sure, with checklists for example, that auditors carefully c o n s i d e r all possibilities before m a k i n g their decisions.
Positive vs negative outcomes In o u r first e x p e r i m e n t , u s i n g an i n t e r n a l control judgment, the c o n d i t i o n with consistent e v i d e n c e a n d o u t c o m e involves a positive o u t c o m e , w h i l e the i n c o n s i s t e n t c o n d i t i o n
189
involves a n e g a t i v e o u t c o m e . Because it has b e e n suggested that n e g a t i v e e v e n t s are given g r e a t e r w e i g h t t h a n positive e v e n t s (e.g. Skowronski & Carlston, 1989), w e tested the first two h y p o t h e s e s in a different c o n t e x t . In the s e c o n d e x p e r i m e n t , w e used an o p i n i o n formulation task w h e r e the consistent evidenceo u t c o m e involved a negative o u t c o m e ; similarly, the surprise c o n d i t i o n i n v o l v e d a positive outcome.
EXPERIMENT 1
Method Subjects. A total of 52 auditors from five i n t e r n a t i o n a l a c c o u n t i n g firms p a r t i c i p a t e d in the first experiment. The auditors w e r e primarily s e n i o r s a n d s u p e r v i s o r s w i t h b e t w e e n t h r e e and s e v e n years of a u d i t i n g e x p e r i e n c e . Mean and m e d i a n years of e x p e r i e n c e w e r e 4.5 and 5 years, respectively. Design a n d procedure. Auditors w e r e rand o m l y assigned to o n e of four o u t c o m e k n o w l e d g e c o n d i t i o n s : foresight ( n o o u t c o m e knowledge), inconsistent evidence-outcome hindsight, c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight, a n d the d e c i s i o n - a i d e d foresight condition. T h e r e w e r e 13 s u b j e c t s in each c o n d i t i o n . Most of the s u b j e c t s c o m p l e t e d the experim e n t a l tasks d u r i n g regular m o n t h l y st'aft meetings with the e x p e r i m e n t e r s present. A small n u m b e r w e r e r e t u r n e d b y mail. 3 T h e i n t e r n a l c o n t r o l e v a l u a t i o n task was m o d e l e d after the o n e used b y Bailey ( 1 9 8 6 ) . Subjects w e r e given b r i e f i n s t r u c t i o n s followed b y five pieces of i n f o r m a t i o n a b o u t the i n t e r n a l c o n t r o l s in the sales/receivables cycle a n d t h e n asked to evaluate the s t r e n g t h of the c o n t r o l s o n a scale from 1 ( v e r y w e a k ) to 7 ( v e r y strong). T h e i n f o r m a t i o n c o n t a i n e d a crucial item that i n d i c a t e d a serious w e a k n e s s in the i n t e r n a l controls: s e p a r a t i o n of duties was violated w i t h r e s p e c t to the a c c o u n t s receivable billing and c o l l e c t i o n functions. In the
No significant differences were fimnd due to method of data collection.
190
J.L. REIMERS and S. A. BUTLER
foresight condition, subjects recorded their b e l i e f s a f t e r r e a d i n g t h e i n t e r n a l c o n t r o l evid e n c e . In t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight condition, subjects were given outc o m e i n f o r m a t i o n b e f o r e t h e y w e r e a s k e d to r e c o r d t h e i r beliefs. A f t e r r e a d i n g t h e i n t e r n a l c o n t r o l data, t h e s u b j e c t s w e r e t o l d that, a f t e r t h e audit, it w a s d i s c o v e r e d t h a t t h e a c c o u n t s receivable clerk had embezzled over $150,0004 b y a l t e r i n g b i l l i n g s a n d d e p o s i t s . In t h e c o n sistent evidence-outcome hindsight condition, t h e s u b j e c t s w e r e t o l d that, after t h e audit, several small (not material) errors were found when the billings and collections were reconc i l e d b y t h e a c c o u n t s r e c e i v a b l e clerk. B o t h h i n d s i g h t c o n d i t i o n s w e r e d e s i g n e d to b r i n g a t t e n t i o n to t h e w e a k n e s s in t h e i n t e r n a l c o n t r o l s t r u c t u r e . A f t e r all t h e i n f o r m a t i o n h a d b e e n p r e s e n t e d , t h e s u b j e c t s w e r e a s k e d to r a t e the internal control structure using only the i n f o r m a t i o n a v a i l a b l e at t h e t i m e o f t h e audit. In all t r e a t m e n t s , t h e i n t e r n a l c o n t r o l s y s t e m w a s h e l d c o n s t a n t . Thus, t h e e v a l u a t i o n o f its strength should not be affected by outcome k n o w l e d g e o f a n y type. 5 The surprising outcome, that over $150,000 h a d b e e n e m b e z z l e d , is a r e l a t i v e l y r a r e e v e n t . Although the description of the controls allows for this p o s s i b i l i t y , t h e p r o b a b i l i t y o f e m b e z z l e m e n t c o n d i t i o n e d o n t h e e v i d e n c e is l o w . W h i l e it is p o s s i b l e , it w o u l d still b e s u r p r i s i n g t o an a u d i t o r t o find s u c h an i r r e g u l a r i t y . O n t h e o t h e r h a n d , w h e n s u c h an i n t e r n a l c o n t r o l w e a k n e s s exists, it is n o t u n u s u a l t o find s m a l l errors. 6
In t h e d e c i s i o n - a i d e d f o r e s i g h t c o n d i t i o n , subjects were given the same internal control i n f o r m a t i o n as o t h e r s u b j e c t s . T h e n , b e f o r e t h e y w e r e a s k e d to r a t e t h e s t r e n g t h o f t h e i n t e r n a l control system, they were asked to think about the possibility of embezzlement and the cont r o l s n e e d e d to p r e v e n t it. Finally, s u b j e c t s rated the strength of the controls on the same 1 to 7 scale. T h e p u r p o s e o f t h e d e c i s i o n aid is to a l t e r t h e e n c o d i n g o r i n t e r p r e t a t i o n of the original evidence, causing subjects to simulate judgment under surprise hindsight. W e t h i n k this t y p e o f c u i n g w o u l d b e e a s y a n d a p p r o p r i a t e in s o m e a u d i t i n g tasks. The dependent variable was the rating of the i n t e r n a l c o n t r o l s y s t e m . T h i s is an e x t e r n a l l y valid judgment; and although the hindsight e f f e c t has t r a d i t i o n a l l y b e e n d e m o n s t r a t e d w i t h a probability judgment (of the outcome), the j u d g m e n t w e u s e d s h o u l d also b e u n a f f e c t e d b y outcome knowledge.
Results T h e m e a n j u d g m e n t w a s 4 . 0 8 for s u b j e c t s in t h e f o r e s i g h t c o n d i t i o n ; 3.92 in t h e c o n s i s t e n t evidence-outcome hindsight condition; and 2 . 6 9 in t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t c o n d i t i o n . T h e m e a n j u d g m e n t for the decision-aided foresight treatment was 3.46. T h e d i f f e r e n c e s in m e a n s a r e all in t h e predicted direction. A 4 × 1 analysis o f v a r i a n c e ( A N O V A ) c o n f i r m e d a s i g n i f i c a n t m a i n e f f e c t d u e to outcome knowledge, F(3, 48) = 4.281, p < 0.01. W e c o m p a r e d t h e m e a n f o r e s i g h t j u d g -
We do not consider the dollar amount here relevant to the treatment of the problem. Since the problem involves fraud, any amount should trigger a similar response. s Complete experimental materials are available upon request. Send requests to J. I.. Reimers, College of Business R53-A, Florida State University, Tallahassee, FI. 32306-1042, U.S.A. 6 Independent auditors were asked to evaluate the surprise of the outcomes. A small sample of auditors was given the evidence and then asked to give the probability that a particular outcome would occur. The surprise (inconsistent evidence-outcome) outcomes were assigned probabilities between O.10 and 0.42, with a mean of 0.25. The not-surprising outcomes (consistent evidence-outcome) were assigned probabilities between 0.60 and 0,95, with a mean of 0.76. Additionally, empirical evidence supports uur manipulations in both cases. Several accounting researchers have ducumented the perception by auditors that irregularities are more unusual than errors (e.g. Coakley & Loebbecke, 1985 ). With respect to the opinion formulation case, we relied on an actual case for our consistent evidence-outcome scenario.
THE EFFECTOF OUTCOMEKNOWLEDGE m e n t ( 4 . 0 8 ) to each of the hindsight c o n d i t i o n s w i t h p l a n n e d c o m p a r i s o n s , 7 finding a significant difference b e t w e e n the m e a n j u d g m e n t in the inconsistent e v i d e n c e - o u t c o m e hindsight condit i o n ( 2 . 6 9 ) a n d the foresight c o n d i t i o n , F (1, 4 8 ) ---- 10.7, p < 0.01. However, the difference b e t w e e n the consistent e v i d e n c e - o u t c o m e hindsight c o n d i t i o n ( 3 . 9 2 ) a n d the foresight condit i o n was n o t significant, F (1, 4 8 ) = 0.14, p > 0.75. This s u p p o r t s the first t w o hypotheses: an o u t c o m e that is i n c o n s i s t e n t w i t h the e v i d e n c e is associated w i t h a hindsight j u d g m e n t w h i c h is significantly different from a foresight judgm e n t , w h i l e a c o n s i s t e n t o u t c o m e is associated w i t h a h i n d s i g h t j u d g m e n t w h i c h is n o t significantly different from a foresight judgment. The third hypothesis, p r e d i c t i n g the usefulness of a simple d e c i s i o n aid in r e d u c i n g the h i n d s i g h t effect, was tested b y c o m p a r i n g the m e a n j u d g m e n t of the decision-aided foresight g r o u p ( 3 . 4 6 ) and that of the i n c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t g r o u p (2.9.6). W e k n o w from the test of the first hypothesis that t h e r e is a significant h i n d s i g h t effect (i.e. 4.08 is different than 2.96). If w e can alter the foresight j u d g m e n t w i t h a d e c i s i o n aid, replacing the m e a n of 4.08 w i t h a n e w m e a n j u d g m e n t of 3.46, does the significant hindsight effect still exist? C o m p a r i n g the m e a n j u d g m e n t of the d e c i s i o n - a i d e d foresight s u b j e c t s to that of the i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight subjects, w e f o u n d that those j u d g m e n t s are n o t significantly different, F (1, 4 8 ) = 2.62, p > 0.10. Thus, the s e c o n d hypothesis is weakly s u p p o r t e d : s a simple d e c i s i o n aid alters foresight j u d g m e n t s in such a way as to r e d u c e the hindsight effect.
191
EXPERIMENT 2
Method Subjects. Fifty-two additional auditors from the same a c c o u n t i n g firms p a r t i c i p a t e d in the s e c o n d e x p e r i m e n t . As in the p r e v i o u s subjectpool, the auditors w e r e s e n i o r s an supervisors w i t h average e x p e r i e n c e of 4.5 years.
Design and procedure. For the second experim e n t , w e s e l e c t e d an o p i n i o n f o r m u l a t i o n task m o d e l e d after an actual case w h e r e the o p i n i o n had b e e n qualified (Accounting Trends and Techniques, 1984). W e gave subjects inform a t i o n a b o u t a c o m p a n y ' s financial risk resulting from asbestos litigation a n d t h e n asked t h e m to give the likelihood that they w o u l d or w o u l d n o t issue a qualified o p i n i o n . Again, w e used a scale from 1 (definitely w o u l d qualify) to 7 (definitely w o u l d n o t qualify). Like the first e x p e r i m e n t , this e x p e r i m e n t c o n s i s t e d of four treatments: the foresight c o n d i t i o n , the c o n s i s t e n t e v i d e n c e - o u t c o m e c o n d i t i o n , the inconsistent evidence-outcome condition and the decision-aided foresight t r e a t m e n t . Again, w e u s e d t w o different o u t c o m e s , o n e c o n s i s t e n t w i t h the e v i d e n c e a n d the o t h e r inconsistent. The first was that t h e r e was additional litigation c o n c e r n i n g the c o l l e c t i o n of the i n s u r a n c e settlement. This o u t c o m e k n o w l e d g e was given after o t h e r i n f o r m a t i o n that was c o n s i s t e n t with such an o u t c o m e . T h e o t h e r o u t c o m e knowledge c o n d i t i o n c o n s i s t e d of an o u t c o m e that was m o r e surprising, given the evidence. G i v e n the pessimistic e v i d e n c e a b o u t the t r o u b l e s of a c o m p a n y i n v o l v e d in asbestos litigation, the surprising o u t c o m e was that t h e r e was n o
7 Comparisons were planned prior to inspecting the data and calculated according to Wirier (1971, pp. 170-175).
HActually, the mean of the decision aid judgments was not significantlydifferent from either the foresight or surprising hindsight judgments. So while we were able to move the judgment in the right direction, we did not eliminate the hindsight effect.
192
J.I.. REIMERS and S. A. BUTLER
f u r t h e r litigation. This s i t u a t i o n is realistic and is o n e w h i c h allows us to r e v e r s e the consistentp o s i t i v e o u t c o m e r e l a t i o n s h i p f o u n d in t h e first e x p e r i m e n t . In this case, u n l i k e that u s e d in t h e first e x p e r i m e n t , t h e o u t c o m e c o n s i s t e n t w i t h t h e e v i d e n c e is n e g a t i v e a n d t h e s u r p r i s i n g o u t c o m e is positive. T h e d e c i s i o n - a i d e d f o r e s i g h t t r e a t m e n t for this e x p e r i m e n t w a s similar to that o f t h e first e x p e r i m e n t . Before m a k i n g a j u d g m e n t , subjects in this t r e a t m e n t g r o u p w e r e asked to consider the worst possible outcome. Because t h e u n f a v o r a b l e o u t c o m e is n o t s u r p r i s i n g in this c a s e a n d w e e x p e c t a u d i t o r s in t h e u n a i d e d f o r e s i g h t c o n d i t i o n to r e c o g n i z e this, a d e c i s i o n aid in this c a s e is n o t e x p e c t e d to affect t h e auditors' judgments. Numerous auditing studies have s h o w n that a u d i t o r s c o n s i s t e n t l y w e i g h t n e g a t i v e e v i d e n c e m o r e h e a v i l y than p o s i t i v e e v i d e n c e ( A s h t o n & Ashton, 1988; Butt & C a m p b e l l , 1989; Kida, 1984). Since t h e p o s s i b l e n e g a t i v e o u t c o m e is so crucial, a d e c i s i o n aid to remove the surprise of a favorable outcome w o u l d n o t b e e x p e c t e d to h a v e any significant effect. O u r d e c i s i o n aid t r e a t m e n t w a s i n c l u d e d in t h e s t u d y for c o m p l e t e n e s s . S u b j e c t s r a t e d t h e l i k e l i h o o d o f t h e firm r e c e i v i n g a qualified o p i n i o n , o n a scale f r o m 1 ( d e f i n i t e l y w o u l d qualify) to 7 ( d e f i n i t e l y w o u l d n o t qualify). O n c e again o u r d e p e n d e n t variable was not a probability judgment of the o u t c o m e . Rather, w e s e l e c t e d a m o r e e x t e r n a l l y valid j u d g m e n t . As in t h e p r e v i o u s task, outc o m e k n o w l e d g e s h o u l d h a v e n o effect o n auditors' judgments.
R e s u Its
T h e r e s u l t s s u p p o r t t h e first t w o h y p o t h e s e s : the inconsistent evidence-outcome hindsight t r e a t m e n t was associated w i t h a larger hindsight effect than t h e c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight treatment. The foresight condition y i e l d e d a m e a n r e s p o n s e o f 2.85, w h i l e t h e m e a n r e s p o n s e s o f s u b j e c t s in t h e consistent and inconsistent evidence-outcome conditions w e r e 2.23 a n d 4.08, r e s p e c t i v e l y . T h e
m e a n j u d g m e n t for t h e d e c i s i o n - a i d e d foresight t r e a t m e n t w a s 2.69. A 4 x 1 ANOVA ( o u t c o m e k n o w l e d g e × t a s k ) c o n f i r m e d that t h e r e w a s a significant m a i n effect for t h e o u t c o m e k n o w l e d g e c o n d i tion, F (3, 4 8 ) ---- 6.284, p < 0.01. P l a n n e d c o m p a r i s o n s c o n f i r m e d that t h e j u d g m e n t s given b y s u b j e c t s in t h e i n c o n s i s t e n t e v i d e n c e o u t c o m e h i n d s i g h t c o n d i t i o n w e r e significantly different f r o m t h e j u d g m e n t s g i v e n b y t h e foresight subjects, F ( 1, 4 8 ) = 9.85, p < 0.01. H o w e v e r , t h e r e w a s n o statistically significant d i f f e r e n c e b e t w e e n t h e foresight a n d c o n s i s t e n t e v i d e n c e - o u t c o m e h i n d s i g h t c o n d i t i o n s F (1, 4 8 ) --- 1 . 9 1 , p > 0.10. T h e s e results c o n f i r m o u r first t w o h y p o t h e s e s . Thus, o u r results are c o n s i s t e n t for p o s i t i v e and n e g a t i v e o u t c o m e s . Unlike t h e r e s u l t s from t h e first e x p e r i m e n t , t h e m e a n j u d g m e n t o f t h e d e c i s i o n - a i d e d foresight s u b j e c t s ( 2 . 6 9 ) w a s significantly different from t h e i n c o n s i s t e n t e v i d e n c e - o u t c o m e hindsight s u b j e c t s ( 4 . 0 8 ) , F (1, 4 8 ) = 12.5, p < 0.01. T h e d e c i s i o n aid m o v e d t h e m e a n judgm e n t in t h e right d i r e c t i o n ; but, as e x p e c t e d , it was n o t a significant m o v e m e n t . As d i s c u s s e d above, this m a y b e due, in part, to a u d i t o r s ' conservative n a t u r e and professional skepticism, s i n c e r e v e r s i n g t h e h i n d s i g h t effect in this case m e a n t m a k i n g a less c o n s e r v a t i v e j u d g m e n t .
SUMMARY AND DISCUSSION W e have d e m o n s t r a t e d that t h e c o n s i s t e n c y between evidence and a subsequent outcome can h a v e an i m p a c t o n t h e d e g r e e o f the h i n d s i g h t effect. Specifically, w e have s h o w n that a u d i t o r s ' j u d g m e n t s r e v e a l a h i n d s i g h t effect w h e n t h e o u t c o m e is i n c o n s i s t e n t w i t h t h e e v i d e n c e , i.e. a surprise. H o w e v e r , w h e n t h e o u t c o m e is c o n s i s t e n t w i t h t h e e v i d e n c e , t h e h i n d s i g h t effect is n o t significant. Similar results h a v e b e e n n o t e d in m o r e g e n e r a l tasks b u t h a v e n o t b e e n p r e v i o u s l y c o n f i r m e d in an a u d i t i n g context. W e h a v e t a k e n t h e r e s e a r c h o n e s t e p further, a r g u i n g that t h e h i n d s i g h t effect m a y b e
THE EFFECT OF OUTCOME KNOWLEDGE reduced by altering the foresight judgment r a t h e r t h a n b y d e b i a s i n g t h e h i n d s i g h t judgm e n t . A u d i t o r s l i v e in a w o r l d w h e r e t h e i r d e c i s i o n s a r e o f t e n j u d g e d in h i n d s i g h t a n d t h e y may be unable to influence those who make the hindsight evaluations. Using a simple decision aid, w e a t t e m p t e d to h a v e a u d i t o r s m e n t a l l y simulate a surprising outcome when they did not have outcome knowledge. While not overw h e l m i n g l y s u c c e s s f u l , t h e r e s u l t s i n d i c a t e that f o r e s i g h t j u d g m e n t s c a n b e m o v e d in t h e d i r e c t i o n o f h i n d s i g h t j u d g m e n t s . M o r e sophistic a t e d aids c o u l d i n c r e a s e t h e e f f e c t i v e n e s s o f this a p p r o a c h . The practical implication of these results r e l a t e s to t h e t y p e s o f o u t c o m e s a u d i t o r s f a c e a n d t h e p o t e n t i a l for r e d u c i n g t h e a m o u n t o f s u r p r i s e in t h o s e o u t c o m e s . B e c a u s e a u d i t o r s w h o a r e e n g a g e d as e x p e r t w i t n e s s e s , for
193
example, are unrelated to the original decision m a k e r s , it is difficult to a l t e r t h e i r d e c i s i o n s . Instead, auditors can specifically consider more a l t e r n a t i v e s w h e n e n g a g e d in t h e o r i g i n a l decision making, thereby reducing the surprise in t h e a c t u a l o u t c o m e . T h e s e r e s u l t s s u g g e s t their decisions would then be more closely aligned w i t h ex p o s t d e c i s i o n makers/evaluators. T h e l i m i t e d s u c c e s s o f t h e d e c i s i o n aid indicates that t h e r e is a p o t e n t i a l g a i n f r o m t h e u s e o f e x t e n s i v e c h e c k l i s t a n d q u e s t i o n n a i r e s a n d that m o r e r e s e a r c h s h o u l d b e d o n e to i n v e s t i g a t e t h e e f f e c t s o f s u c h a u d i t aids. In a d d i t i o n to a u d i t i n g , t h e r e are o t h e r accounting and business contexts where the hindsight effect may be important. Since many b u s i n e s s d e c i s i o n s a r e e v a l u a t e d in h i n d s i g h t , w e h o p e t o e x t e n d this r e s e a r c h t o m o r e general managerial contexts.
BIBLIOGRAPHY Accounting Trends and Techniques (New York: American Institute of Certified Public Accountants, 1984). Arkes, H. R., Wortmann, R. L., Saville, P. & Harkness, A. R., Hindsight Bias Among Physicians Weighing the Likelihood of Diagnoses,Journal of Applied Psychology ( 1981 ) pp. 252-254. Arkes, H. R., Faust, D., Guilmette, T. J. & Hart, K., Eliminating the Hindsight Bias, Journal of Applied Psychology ( 1988 ) pp. 305-307. Ashton, A. H. & Ashton, R. H., Sequential Belief Revision in Auditing, The Accounting Review (October 1988) pp. 623---641. Bailey, C. D., Awfiding Errors in Judgment, The Internal Auditor (1986) pp. 25-28. Brown, C. E. & Solomon, I., Effects of Outcome Information on Evaluations of Managerial Decisions, The Accounting Review (July 1987) pp. 564-577. Brown, C. E. & Solomon, I., An Experimental Investigation of Alternative Explanations for Hindsight Effocts in Agpraisal of Capital Budgeting Decisions (Unpublished manuscript, University of illinois, 1~<)). Buchman, T. A., An Effect of Hindsight on Predicting Bankruptcy with Accounting Information, Accounting~ Organizations and Society (1985) pp. 267-285. Butler, S. A., Application of a Decision Aid in the Judgmental Evaluation of Substantive Test of Details Samples, Journal of Accounting Research (Autumn 1985) pp. 513-526. Butt, J. L. & Campbell, T. L., Information Order and Hypothesis-Testing Strategies, Accounting Organizations and Society (1989) pp. 471-479. Coakley, J. R. & Loebbecke, J. K., "l]ae Expectation of Accounting Errors in Medium-sized Manufacturing Firms, Advances in Accounting (1985). Cushing, B. E. & Loebbecke, J. M., Comparison of Audit Methodologies of Large Accounting Firms (Studies in Accounting Research 26, American Accounting Association, 1986). Ellingsen, J. E., Pany, IC & Fagan, P., SAS No. 59: How to Evaluate Going Concern,Journal of Accountancy (January 1989)pp. 24--31. Fischhoff, B., Hindsight # Foresight: the Effect of Outcome Knowledge on Judgment Under Uncertainty, Journal of Experimental Psychology: Human Perception and Performance ( 1975 ) pp. 288--299.
194
J.L. REIMERSand S. A. BUTLER Fischhoff, B., Perceived Informativeness of Facts, Journal o f Experimental Psychology.. Human Perception and Performance (1977) pp. 349-358. Fischhoff, B., Debiasing, in Kahneman, D., Slovic, P. & Tversky, A. (eds),Judgment under Uncertainty: Heuristics and Biases (New York: Cambridge, 1982). Hasher, L., Attig, M. S. & Alba,J. W., I Knew It All Along: Or, Did I?,Journal o f VerbalLearning and Verbal Behavior ( 1981 ) pp. 86-96. Hawkins, S. A. & Hastie, R., Hindsight: Biased Judgments of Past Events After the Outcomes Are Known, n _ . . ,
,
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
J O U R N A L O F BUSINESS F I N A N C E A N D A C C O U N T I N G
Volume
18, N u m b e r
September
5
1991
Editor: Richard Briston CONTENTS M a c r o e c o n o m i c Factors and t h e UK Stock Market S. P O O N and S. J. TAYLOR G o i n g Private: C o r p o r a t e R e s t r u c t u r i n g U n d e r I n f o r m a t i o n A s y m m e t r y and A g e n c y P r o b l e m s WI SAENG KIM and ESMERALDA O. LYN Financial Ratios and Different Failure P r o c e s s e s ERKKI K. LAITINEN M a n a g e m e n t A n t i c i p a t i o n o f I n t e r e s t Rates: T h e Case o f C o m m e r c i a l Banks JENNIFER FRANCIS A l t e r n a t i v e M o d e l s for Assessing t h e Financial C o n d i t i o n o f Business in G r e e c e PANAYIOTIS T H E O D O S S I O U A g g r e g a t i o n and R e c i p r o c a l S e r v i c e Cost Allocation CHARLES SUTCLIFFE T h e I m p a c t o f S o c i o e c o n o m i c Variables and C r e d i t Ratings o n Municipal B o n d Risk P r e m i a PU LIU and FAZEL J. SEWYED R e d u c i n g t h e U n c e r t a i n t y - I n d u c e d Bias in Capital B u d g e t i n g D e c i s i o n s - - A H u r d l e Rate Approach TARUN K. MUKHERJEE T h e Valuation o f Natural Resources: E v i d e n c e from Oil and Ga~s Pure Plays J O H N L CRAIN and A. M. M. JAMAL T h e Relationship B e t w e e n H o l d i n g Gains and Equity Value: A Neoclassical I n t e r p r e t a t i o n KEVIN C. W. CHEN