- Email: [email protected]
The use of operant theory in the design of performance reporting systems
Management Accounting Research, 1992, 3, 273-289
The use of operant theory in the design of performance reporting systems Linda M. Lovata* Operant theory is a branch of psychology that focuses on reinforcement and punishment. In general, reinforcement is preferred over punishment and a combination of both should be optimal. Outcome feedback can provide either reinforcement or punishment depending on the cues transmitted. Therefore, the receipt of only positive feedback should elicit better task performance than the receipt of only negative feedback. Also, a combination of the two should be the best motivator. A laboratory experiment was designed in which subjects participated in a computer simulation of a raw materials purchasing scenario. Performance was measured by costs incurred and decision speed. The group receiving a variance report when costs were outside of a specified range, both positively and negatively, made the quickest decisions without sacrificing cost performance. Receiving only negative performance reports resulted in the worst task performance. Even though the results of the experiment provide very limited support of operant theory, they do suggest that this theory may be useful in the design of better reporting systems. Also, the results indicate that the most common reporting systems may not be the most effective. Key words: performance reporting; operant theory; outcome feedback.
1. Introduction Outcome feedback is an important systems design variable which has received very little attention in the literature. T h e timing of cognitive feedback has been shown to affect decision maker performance (Te'eni, 1991). The study reported here examines the timing of outcome feedback on decision makers. While some anecdotal and experimental evidence is available (At Emery. . . , 1973; Beatty and Schneier, 1975; Hirst and Luckett, 1992), no empirical work has explored the most effective use of outcome feedback. Perhaps the reason for this omission is that only with current technology are alternative systems viable. In the past, systems designers were constrained by the manual processes necessary for report preparation. It often took a week or month to input data prior to processing, so fdes were only updated periodically. The online processing allowed by present technology makes it feasible to implement more creative reporting systems because of the timeliness of the available information. Practitioners and academics alike have espoused the benefits of emphasizing outcome feedback that highlights positive *School of Business, Department of Accounting, Southern Illinois University at Edwardsville, IL 620261104, U.S.A. Received 23 July 1992; accepted 1 November 1992.
01992 Academic Press Limited
274
L. M. Lovata
performance (Birnberg and Nath, 1967; Beatty and Schneier, 1975; Brownell, 1983). This capability would have been expensive several years ago since it requires close traclung of performance, but currently is easily implemented. Therefore, it is now important to investigate the effect of the timing of outcome feedback on decision maker performance. Since the purpose of many internal reports is to motivate users to improve performance (Horngren and Foster, 1987), behaviour modification theory (also called operant theory) may provide insight into ways of increasing the effectiveness of reporting systems. Operant theory states that behaviour is modified based upon the stimulus that occurs immediately following a particular action. Behaviour which is followed by a reward will increase, while behaviour which is punished will be eliminated. This theory, then, may provide guidance as to how to use outcome feedback to maximize decision maker performance. The purpose of this study is to investigate within an operant theory framework the effect of outcome feedback on decision maker performance. The study reveals that the nature of the outcome feedback impacts performance and provides limited support of operant theory. Specifically, providing feedback arbitrarily can interfere with effective decision making, and providing a balance of favourable and unfavourable feedback can enhance performance over reporting only negative results. First, operant theory will be discussed. In the next section, related research is examined. Then operant theory is explained in relation to this particular study and the hypotheses developed. This is followed by a methodology section. The data analysis and results are then discussed, and a summary concludes the paper.
2. Operant theory Operant theory has been tested extensively in the areas of mental health and education often using animals, students or handicapped individuals as subjects. Its application in organizations is referred to as Organizational Behaviour Management (OBM). Originally it was feared that many of the tenets of operant theory only applied to very fundamental behaviours with naive subjects, but research in OBM has found these techniques to be valuable for a wide range of organizational behaviours (O'Hara et al., 1985). Several types of stimuli may occur subsequent to any behaviour. If the stimulus encourages the behaviour to recur, it is considered reinforcement. Rewards such as money or food are considered positive reinforcement if their presence increases the likelihood that the behaviour will recur. Negative reinforcement also increases the likelihood of the behaviour recurring, but this type of reinforcement occurs when an unpleasant stimulus is removed subsequent to the desired behaviour being performed. Any stimulus can be used as reinforcement. In fact, the definition of a reinforcer is simply a stimulus that increases the likelihood that the behaviour will recur. Therefore, the effectiveness of different stimuli will differ across individuals. While reinforcement will increase the likelihood of a behaviour recurring, there are two ways to reduce the likelihood: ignore or punish the behaviour. Behaviours that are ignored, i.e., neither reinforced nor punished, will be reduced. In order to eliminate these behaviours effectively, however, it is important to note that many different stimuli can serve as reinforcement. Even though tangible rewards are eliminated, if other
The Use of Operant Theory
275
s t i m u l ~such as social or intrinsic reinforcers continue, the behaviour will not be extinguished. Punishment is the more direct approach to eliminate behaviour. It consists of either the presentation of an undesirable stimulus or the removal of a pleasant stimuli subsequent to the behaviour. In both instances, the behaviour of interest should decrease. Operant theory warns of the excessive use of punishment. In general, the receipt of only punishment tends to result in behaviour which is, '. . . rigid, non-realisti~,and inappropriately strong' (Thompson, 1978, p. 67). This is due in part to recipients of only punishment receiving clear signals as to which behaviours are inappropriate without equally clear indications of the appropriate behaviours. Accordingly, they tend to overreact to the specific behaviours being punished. One key to using operant theory is to define appropriate rewards and punishment. Although a reward is often associated with money or food, and punishment may be electric shock or removal of privileges, outcome feedback can also serve as rewards or punishment (Duncan and Bruwelheide, 1985; Miller, 1978; Potter, 1980). In fact, within an organization, outcome feedback has the advantage over many rewards in that it is inexpensive and easy to administer (Miller, 1978, pp. 116- 117). When feedback is viewed in this manner, there are pervasive implications for the design of information systems. If feedback is favourable, it may reinforce behaviour; if it is negative, it is punishment (Ilgen et al., 1979). According to operant theory, then, the type of feedback received is critical to the subsequent behaviour of the decision maker. The majority of reporting systems currently used in organizations ignore this behavioural aspect of reporting, issuing reports based on an arbitrary passage of time, or if contingent on performance, emphasizing only negative performance deviations (Birnberg and Nath, 1967; Brownell, 1983; Keithley, 1975; Ludwig, 1973). The management report most directly related to rewards or punishment is a performance report. This report generally relates performance to a standard, budget, or goal.' If operant theory holds, task performance should improve when reports are issued based upon deviations from expected performance rather thsn the arbitrary passage of time. Also, highlighting favourable performance in addition to unfavourable performance should elicit better behaviour than emphasizing only unfavourable performance. Since prior research has not integrated operant theory and accounting reports, this study examines one specific report to determine its effect. While several different types of rewards can be made contingent on a behaviour, for this preliminary study of operant theory, only one potential stimulus will be isolated. Once tested, then subsequent studies can examine other reward structures to determine their incremental effects. The study reported here investigates variance reports within the context of operant theory. The results provide some general support for the use of operant theory in the design of variance reporting systems. 3. Prior research
Two areas of research relate to this investigation. The first studies deal with time-based reporting. Though many of these studies examine aggregation levels, contrasting, for example, weekly and monthly reports (Bruns, 1980; Firth, 1980), one study specifically 'Feedback is especially effective when it compares performance to a goal or standard (Potter, 1980). Therefore, a variance report was chosen as the mode of feedback.
276
L. M. Lovata
tests reporting intervals in combination with aggregation levels (Cook, 1967) and finds that more frequent reporting yields the best performance. Cook (1967) divided teams of students participating in a simulated business game into three groups. One group received the results of their past decisions prior to making subsequent ones. A second group received reports presenting the results of the four prior decisions every fourth period, while a third group received no feedback until the end of the game. The group receiving the most feedback performed the best. This study was criticized on several dimensions (Becker, 1967; Jensen, 1%7), two of which will be discussed here. First, the information withheld was critical to the successful completion of the business game. Second, the group participation over an entire semester made communication among groups likely. The study reported in this paper attempts to circumvent these problems in a similar decision environment. The manipulated report is not essential to the decision maker's analysis of the task. Also, the business simulation developed for this study can be completed in one session so as to diminish the probability of communication among participants. Two recent studies by Te'eni (1991) and Hirst and Luckett (1992) also examine the timing of outcome feedback. Hirst and Luckett (1992) find that when the task is perfectly predictable, outcome feedback is useful. Te'eni (1991) finds that the timing of cognitive feedback influences performance. Both studies manipulated reports based on periodicity, not performance. A second area of related research lies in the management by exception (MBE) literature. Two studies have systematically tested the effects of MBE and the results of both are equivocal. The first by Brownell (1983) examines MBE in an expectancy theory framework. By analysing 122 responses to a questionnaire, he tests both the use of favourable and unfavourable exception reporting and the interaction among participation, motivation, and MBE. The questionnaire measured participation through established, validated scales and motivation by eliciting intrinsic and extrinsic valences (utilities) for 17 scenarios. The use of MBE was determined by responses to questions dealing with when superiors investigate variances. Brownell finds that although negative exception reporting and motivation are inversely related as hypothesized, the relationship is not statistically significant. Also, the effect of MBE on motivation may differ given the level of budgetary participation. The finding most relevant to this study is that there is, in fact, a bias toward emphasizing unfavourable variances. Benbasat and Schroeder (1977), using an experimental computer simulation (business game), operationalize MBE by placing an asterisk beside reports revealing greater than a 10% change from the prior period. This manipulation did not provide significant changes in performance. Benbasat and Schroeder state: 'While this (lack of significance) may lead to the conclusion that exception reporting is ineffective, we feel that the size and nature of the reporting system used in the experiment were factors that reduced the potential effectiveness of exception reporting' ( P 46).
They then appeal for more research in the area. While Brownell focuses on the unfavourable-only type of exception reporting and Benbasat and Schroeder investigate the favourable and unfavourable condition, no study has contrasted these types of reporting systems. Also, prior accounting studies
The Use of Operant Theory
277
have not examined the differential impact due to the content of the performance reports. The management literature discusses the importance of this distinction. In the OBM literature, several authors have noted the need to better specify feedback content in order to assess the reinforcing nature of reports. As Ilgen et al. (1979) state: 'Theoretically, to assume that feedback per se is a positive reinforcer ignores the fact that feedback varies along some positive-to-negative continuum. It is well established in the operant conditioning literature that positive reinforcers have different effects on responses than does punishment (Reynolds, 1968). It is difficult to conceive of negative feedback serving as a positive reinforcer in many settings. Yet, much of the research on the frequency of feedback fails to deal with the sign of the feedback. . .' (p. 360).
More recently, several review articles have noted the lack of empirical research in this area (Prue and Fairbank, 1981; Balcazar et al., 1985; Duncan and Bruwelheide, 1985). The study reported in this paper attempts to address these omissions in OBM literature. The sign of the feedback is explicitly considered and analysed within the operant theory framework. In the next section, this framework is used to develop the specific hypotheses tested.
4. Operant theory and hypotheses development While periodic reporting transmits information regarding past performance, operant theory suggests that certain types of reports are more effective if they are tied directly to exceptional performance. The reports that would be appropriate for this type of schedule are those dealing with outcome feedback, which Prue and Fairbank (1981) define as being information concerning the quantity or quality of past performance. Limiting reports as suggested by operant theory is not applicable to reports that relay critical data concerning the decision environment. Therefore, the reporting systems for which operant theory may apply include performance evaluations whether they be quality control reports, personnel evaluations, or variance reports. These types of reports can exhibit reinforcing qualities if they are issued only when the desired behaviour is observed. While traditional reporting systems that issue reports at regular intervals, such as weekly or monthly, are effective, this theory suggests that the value of performance reports will increase if received based upon significant performance variations. Accountants refer to this type of system as exception reporting, in which performance reports are received only when deviations from standard are observed. Unfortunately, the most common type of exception reporting system is one where negative variances are scrutinized (Brownell, 1983; Birnberg and Nath, 1967). In operant theory terms, this provides only punishment to the decision maker. At the other extreme is an exception reporting system that reports only positive variances. With this system, rewarding reports are received for good performance while poor performance is ignored. A derivative of rhis type of system was implemented at Emery Air Freight and Michigan Bell with remarkably positive results (At Emery. . . , 1973; Beatty and Schneier, 1975). Emery Air Freight chose to exclude monetary incentives in their positive reinforcement program due primarily to practical problems in administering monetary rewards. They reported sales and productivity increases resulting in estimated benefits of over $3 million per year. Operant theory predicts that this type of system results in better task performance than the unfavourable only system.
278
L. M. Lovata
A practical problem with a strictly positive system is that cues are not provided when performance is out of control. Extremely poor performance may be allowed to continue for an extended period of time before an individual recognizes the problem and corrects it, which may result in frustration on the part of the decision maker (Martin and Pear, 1983, p. 53). Accordingly, a complete exception reporting system that generates reports when performance is exceptionally favourable as well as unfavourable should be optimal. This system balances rewards and punishment and prevents the situation where performance is extremely poor unbeknownst to the decision maker. This type of system should discourage an individual from continuing poor performance while rewarding appropriate responses. Therefore, the hypotheses to be tested require an exception reporting framework. In order to operationalize the exception reporting systems, a range of acceptable but not exceptional performance must be specified. If this is not done, then receiving both positive and negative feedback would be equivalent to receiving a report every period. Exception reporting systems that emphasize favourable performance should elicit better performance than those concentrating on negative performance, and a balance of favourable and unfavourable reporting should yield the highest level of performance. Two hypotheses are developed for this analysis. First, the best contingent reporting system (i. e., exception reporting system) should be superior to non-contingent reporting schemes. Stated in the alternative form: H1: Receipt of both favourable and unfavourable performance reports will increase task performance over receipt of reports on a non-contingent basis.
Second, receipt of both favourable and unfavourable reports should outperform the receipt of only favourable reports which should outperform the receipt of only negative reports. H2: Receipt of both favourable and unfavourable performance reports will increase task performance over receipt of only unfavourable or only favourable performance reports.
In order to provide strong support for operant theory, the null of both hypotheses should be rejected. While there is no explicit theory to predict how the less desirable contingent reporting systems compare with non-contingent systems, it is anticipated that positive reporting systems will increase task performance. Given the negative impact of receiving only punishment, the relation between the unfavourable only group and the others is unclear. Therefore, all groups will be compared in order to discern relationships. Table 1 presents a summary of the hypothesized results.
5. Method Control over the lag between the decision making activity and the receipt of the report is crucial because operant theory states that the reward or punishment must follow the behaviour as rapidly as possible. As a result, the experimental materials were presented to subjects on microcomputers as a business game. By using a computer simulated business game, the time lag is controlled precisely, and decision time and cost performance are measured unobtrusively.
The Use of Operant Theory
279
Table 1 Hypothesized dtfferences*
No report Fourth Always Unfavourable Favourable
Fourth
Always
Unfavourable
Favourable
Both
?
? ?
? ? ?
BETTER BETTER BETTER BETTER
BETTER BETTER BETTER BETTER BETTER
*BETTER indicates performance is hypothesized to be better for the group with the heading across the top of the table.
Subjects Sixty graduate business students were randomly assigned to one of six groups, resulting in 10 subjects per cell. The average participant was 26-years-old with three years of work experience. The experiment required no special knowledge or experience since the task was self-contained. Also, it was a decision making task not an attitudinal test. Ashton and Kramer (1980) provide an extensive review of the literature contrasting business people with students and conclude that student's performance is not significantly different in decision malung tasks. Since this is an experiment concerning basic human behaviour, not expert judgment, students were deemed a p p r ~ p r i a t e . ~ Each subject participated in the experiment individually, and the entire procedure was completed in one session. Variables The independent variable in this research is the timing of the variance report. The variance report is based only on the most current decision period and in no way accumulated the results of prior decisions. The timing of the report, then, not the lag factor or aggregation level, is manipulated. Six levels of this variable are tested: (1) (2) (3) (4)
Variance report never received (No Report); Variance report received every fourth period (Fourth); Variance report received every period (Always); Variance report received only when an unfavourable variance greater than 3% of standard is observed (Unfavourable); (5) Variance report received only when a favourable variance greater than 3% of standard is observed (Favourable); and (6) Variance report received when either a favourable or unfavourable variance greater than 3% of standard is observed (Both). Subjects were assigned to one of the six treatment conditions and received only that treatment for the duration of the experiment. The two performance measures used as dependent variables were cost performance and decision time. These were measured unobtrusively by the simulator, and the averages for each subject used. The time measure is examined because it has been shown that the lag between subsequent responses is often directly related to rewards or 2This was confirmed when analysis showed that age and experience did not affect decision maker performance.
280
L. M. Lovata
punishment (Miller, 1978; Martin and Pear, 1983). From an operational perspective, managers are interested in achieving the best decisions in the least amount of time. Therefore, a performance reporting system that reduces the time spent making a good decision is valuable.
Experimental procedures First, a consent form and a short demographic questionnaire were completed. Next, the instructions were read and the simulation performed. Subjects were told to do their best when completing the simulation. While they were not specifically instructed to minimize decision time, they had the natural incentive to finish quickly. Finally, the subject was debriefed. A raw materials purchasing scenario was simulated in which subjects were told to buy peaches for a cannery from three suppliers. The first supplier had inexpensive peaches of low quality. The second had medium priced peaches with quality to match, while the third had high priced, high quality peaches. Quality was measured by the number of cans produced per bushel. The purchasing decision was repeated 24 times with the price and the quality of peaches varying each period though the general quality differences among the three suppliers remained constant. The subjects had to estimate trends in the quality of the suppliers' goods which indicated that the quality of supplier one's goods improved significantly over the duration of the experiment. Therefore, if the subject correctly estimated these trends, he or she would originally purchase mostly from suppliers two and three, then slowly move toward purchasing more from supplier one. The subject could detect these trends through past data presented with the instructions and from the results of his/her decisions. After each decision, a production report and a variance report (Figure 1) were presented on the CRT. The production report followed directly from the subject's purchase decision. If more was ordered than necessary, a $3.50 disposal charge was incurred on each unused bushel; if a shortage occurred, there was a 20% rush order charge for each additional bushel. On the production report, these two charges constitute the difference between the 'cost of materials ordered' and the 'total cost'. All groups received the production report every period. In this way, one group did not have an unfair advantage over the others. If this was the manipulated report, the group receiving it every period would have more information upon which to estimate the various distributions. Finally, the variance report was presented when applicable.3 A favourable or unfavourable variance report was issued dependent upon the subjects' performance and the cell to which they were assigned. Their performance relative to the standard dictated if the report was favourable or unfavourable. Favourable variance reports were accompanied by a statement indicating 'GOOD JOB' while unfavourable variance reports stated 'YOUR COSTS ARE TOO HIGH'. The variance report was shown to the subject only when appropriate given their cell assignment. One group of subjects never received the variance report, while another group received it after each decision, and a third group received it after every fourth decision. The exception reporting 3The variance was not divided into the price and efficiency components because in this case the two were inversely related and both were the responsibility of the purchasing agent. If poor quality goods were purchased, a favourable price variance and an unfavourable efficiency variance resulted. The purchasing agent was responsible for buying products that improved the overall variance so this was the only one presented.
The Use of Operant Theory PRODUCTION REPORT Quantity ordered Cans per bushel
Supplier
281
Price
Scheduled production = 65 000 cans Amount produced from materials ordered = 37 242 cans Cost of materials ordered = $16 910 Total cost = $32 108 VARIANCE REPORT The standard cost is $29 900 There is an unfavourable variance of: $2 208 YOUR COSTS ARE TOO HIGH
groups received the variance report if their performance on that decision resulted in the specified deviation from standard. Finally, a short debriefing session was held. The subject was asked to specify the decision rules used, and if he or she remembered receiving any variance reports. The subjects accurately recalled the reason they received a variance report and the number of variance reports received. This provides some assurance that the variance report was not overlooked. Lastly, the subject was free to ask questions concerning the experiment.
Pilot study A pilot study using 20 additional subjects was conducted to determine a proper standard and the number of decisions to be made. Table 2 depicts the number of reports that would have been issued in the pilot study given three possible standard costs per can. Since it is essential for each group to perceive their goal level to be equal, 46 cents per can was used in the actual study. Also, the three percent deviation from this standard was found to be difficult yet achievable. Finally, the pilot study determined the number of decisions to be made. Twenty-four iterations surpassed the learning curve while avoiding a fatigue factor. Table 2 Reports issued with diffmmt standards Standard
Number Number Number Number
of favourable reports 142 of unfavourable reports 338 70 surpassing 3% favourable surpassing 3% unfavourable 219 Total number of observations = 480
231 249 120 123
330 150 191 83
282
L. M. Lovata
6. Data analysis and results The mean cost for all groups was $31 482.90; the mean variance from standard cost was $182.34; and 55% of the variances were favourable while 45% were unfavourable. This nearly 50150 split supports the use of the pilot study for determining a standard. Within the exception reporting groups, each subject received an average of four negative and/or seven positive variance reports, therefore the 3% exception reporting level was sufficient to generate reports. The data analysis for this study is a three-part process. First, the number of iterations to be used for the analysis is determined. Second, two manipulation checks are performed, and finally, F-tests are used to determine the significant treatment variables.
Number of iterations During the first several decisions, a strong learning trend is observed. For decision time and cost effectiveness, the average of the iterations is used as the dependent ~ a r i a b l e If . ~ the first several trials are included in this average, the decision time and costs would be mis-stated since they may include differential learning produced by the treatments. Regression analysis is performed on the average decision time, measured in minutes, versus the iteration. The resultant power function (TIME = 55.84 ITERATION-^.^'^ with an R2 of 0.98) reveals that the majority of learning occurred in the first seven iterations (see Figure 2). These seven iterations, therefore, are excluded from the remainder of the analysis. As for the last iterations, there is not a dramatic drop in the tail of the time plot so there does not appear to be a fatigue factor. Subjects were told the total number of iterations to be simulated, however, so to control for possible end-of-game strategies, the 24th iteration is also excluded from further analysis. Of the 24 iterations performed by each subject, then, iterations eight through 23, inclusive, or 16 iterations, are averaged to determine each subject's costs and decision time. 7
-E
.=
E
F
.
1
1
1
,
,
l
l
l
l
l
l
,
I
l
l
l
l
l
l
l
l
l
l
l
'
6-
-
5-
-
4-
-
3-
-
2-
-
-
-
I
o
~
I
I
I
I
I
I
I
I
I
I
~
I
I
I
I
I
~
~
~
I
0 1 2 3 4 5 6 7 8 9 1011 I 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 Iteration
Figure 2. Learning curve. The hypotheses were also tested using repeated measures analysis with the same results. The averaging method is presented in the paper to facilitate the reporting of results.
I
I
~
The Use of Operant Theory
283
Decision time is inverted to depict decision speed in order to normalize that distrib~tion.~ The higher the number, then, the quicker the decision.
Manipulation checks One criticism of many OBM studies on the effects of feedback is that the information value of feedback is confounded with the reinforcement aspect (Duncan and Bruwelheide, 1985). It is difficult in many studies to determine whether performance is improved because of the information gained from feedback, or because feedback acts as a reinforcer for good performance. Therefore a control group is necessary to insure that the reinforcement quality and not the information content of the manipulated reports is triggering the responses. A group that always receives the report is contrasted to one never receiving it. If this test reveals differences between the groups, then the information transmitted in the performance report may have allowed users to make better decisions. Alternatively, simply knowing that a standard exists against which performance is being measured may elicit a reaction. In order for this experiment to test the reinforcing qualities of performance reporting, it is important for the report not to provide any incremental information which enables individuals to make better decisions. Ideally, the report only highlights performance deviations thereby causing the decision maker to better attend to the essential data. This comparison will test whether the information content or the existence of a standard are confounding factors in the analysis of the variance reports. The means of all groups are presented in Table 3. When contrasting those always receiving the variance report with the group never receiving the report, neither costs (F= 1.62; P = 0.209) nor the decision speed (F = 0.01; P = 0.932) is significantly different. Therefore there is no data essential to decision quality provided by the variance report nor does the mere existence of a standard appear to impact decisions. Any differences discovered between other groups are not due to these alternative hypotheses. Another manipulation check is performed to ensure that simply the intermittent receipt of a performance report does not cause the user to better attend to the data. Reports received intermittently based on the passage of time, not performance, should not improve decision making. Operant theory states that rewards and punishment must Table 3. Cell means
N o report Fourth Always Unfavourable Favourable Both
N
Costs
Speed (l/Time)
Minutes/Decision
10 10 10 10 10 10
31 628.0 32 252.0 31 122.1 31 612.9 31 354.3 31 274.8
481.3 720.6 472.4 464.5 597.8 661.2
2.08 1.39 2.12 2.15 1.67 1.51
Since there was a practical minimum and maximum time to complete the task, the distribution of time, and therefore the distribution of errors, was not normal (skewness of 1.5, kurtosis of 3.8). Several transformations were performed with the inverse resulting in the best distributional qualities (skewness of 0.6 and kurtosis of -0.05).
284
L. M. Lovata Table 4. Univariate F -tests
Costs* Speed*
Model df
F-Value
P-value
5 5
2.02 2.22
0.09 0.07
Significant at the 0.10 level.
be contingent on behaviour to favourably influence performance. If a performance report is received for other reasons, it should not be as effective as when the report is received due to performance deviations. Therefore, the group receiving the variance report every fourth period is contrasted to the both favourable and unfavourable group. Examining the means of the intermittent group on Table 3 reveals that overall the decision speed of the group receiving the report every fourth period is quicker than most other groups while costs are generally higher. Relative to the both favourable and unfavourable group, receipt of the report every fourth period does not elicit significant differences in the speed of the decisions (F = 0.33; P = 0.285) but does result in significantly higher costs (F = 6.03; P = 0.009). Therefore, the receipt of intermittent reports in and of itself does not improve performance. In fact, receiving the report based on this arbitrary passage of time encouraged subjects to make hasty, suboptimal decisions. This emphasizes the importance of making intermittent reports contingent upon behaviour .
Treatment variables The manipulation checks provide evidence that refutes several alternative hypotheses. This increases the credibility of the multivarite analysis of variance conducted to test the hypotheses. The Wilks' lambda produced by this test simultaneously examines both dependent variables (cost performance and decision speed) to determine if there are significant relationships among the dependent and independent variables. The test does not indicate which specific relationships are significant. To discern the significant relationships, the dependent variables must be examined independently through the use of univariate analysis of variance (Press, 1982, pp. 97-98). The overall Wilks' lambda is significant (lambda = 0.712; P = 0.04) so univariate tests on each dependent variable are performed and the results presented in Table 4. These results indicate that both decision speed and costs contributed marginally to the significant overall lambda. Contrasts are performed to determine the significant differences among these means (see Table 5). Discussion The results presented in Table 5 are now discussed in relation to the hypotheses presented earlier in this paper.
Hypotheses Hypothesis one: both versus non-contingent reporting. None of the groups achieved significantly lower costs than the both favourable and unfavourable group, and the
The Use of Operant Theory
285
Table 5. Dtffmmes among means t-values (P-values) Fourth No report
Costst Always Unfavourable
-624.0 2.46 (0.12)
505.9 1.62 (0.21) 1129.9** 8.06 (0.06)
-239.3" 5.28 (0.03)
8.9 0.01 (0.93)
Fourth
Favourable
Both*
Always
Unfavourable
Favourable
No report
Fourth
Always
Unfavourable
Favourable
t Negative sign indicates costs were lower or decision speed slower for the group with the heading in the left-hand column. $ Since a direction is hypothesized, the both favourable and unfavourable tests are one-tailed. * Significant at the 0.05 level. ** Significant at the 0.01 level. group receiving the report every fourth period incurred significantly higher costs. With respect to decision speed, the group receiving both favourable and unfavourable reports worked significantly faster than all but the fourth group. Examining costs in conjunction with speed reveals that the both favourable and unfavourable group performed significantly faster than the always and never groups without sacrificing cost performance. They also worked at the same pace as the fourth group while achieving
286
L. M. Lovata
significantly lower costs. Therefore, the both group's overall performance (low costs combined with quick decision speed) was superior to all of the fixed interval reporting systems.
Hypothesis two: both versus only favourable versus only unfavourable. Among the contingent reporting systems, the means of both cost performance and decision speed are in the hypothesized direction, but the significance levels are lacking. The means of both decision speed and costs are ranked as predicted, with the combination of favourable and unfavourable reporting yielding the quickest decision speed and the lowest costs. The mean cost of the favourable only group ranks second while the unfavourable group is third. Unfortunately, the only significant difference among these means is the comparison between the decision speed of the balance of favourable and unfavourable reports with the unfavourable only group. This particular contrast is especially important, however, since the majority of exception reporting systems take the negative reporting approach (Brownell, 1983).
7. Conclusions While the tenets of operant theory are only partially supported in this study, the limited results do suggest that this theory may be useful in the design of reporting systems. Given the number of cells and the sample size, it is apparent that the power of the tests is weak. Also, a limitation of this study is the time involved. In an actual work situation, decisions and the related performance reports occur weeks or even months apart. In this experiment the reports were received only minutes apart. Even within this short decision period, decision time is affected by the performance reports. Therefore, it is hypothesized that the effect could be strengthened by extending the reporting interval, but only continued research can reveal the actual affect of an increased time horizon. Also, in this experiment subjects were only instructed to do their best, while in other settings performance feedback is often associated with additional reinforcers such as praise, social interaction, or monetary rewards. Prior research has shown that providing additional rewards in conjunction with feedback enhances its effectiveness (Balcazar et al., 1985). Therefore, combining contingent feedback with other rewards, such as pay, should increase management's power to modify behaviour in the desired direction. The results presented here suggested that when performance reports are viewed as rewards or punishment, the most frequently used reporting systems do not utilize the power of feedback efficiently. First, most reporting systems issue reports based upon some arbitrary passage of time rather than making them contingent upon performance. Second, those systems that do relate reporting to behaviour generally emphasize the negative aspects of performance. Acknowledgements: Financial support for this research was generously provided by the IBM
Corporation. The author would like to thank Milton Jenkins, Mike Groomer, Les Heitger, Yaw Mensah, Mike Tiller, Sonia Goltz, Dave Cowan, Nancy Carter and two anonymous reviewers.
References Ashton, R. H. and Kramer, S. S., 1980. Students as surrogates in behavioral accounting research, Jounull of Accounting Research, Spring, 1-15.
The Use of Operant Theory
287
1973. At Emery Air Freight: Positive Reinforcement Boosts Performance, Organizational Dynamics, Winter, 41-50. Balcazar, F., Hopkins, B. L. and Suarez, Y., 1985. A critical objective review of performance feedback, Journal of Organizational Behavior Management, 7(3), 65-89. Beatty, R. W. and Schneier, C. E., 1975. A case for positive reinforcement, Business Horizons, April, 57-66. Becker, S. W., 1967. Discussion of the effect of frequency of feedback on attitudes and performance, J o u m l of Accounting Research (Suppl), 225-228. Benbasat, I. and Schroeder, R. G., 1977. An experimental investigation of some MIS design variables, M I S Quarterly, March, 37-48. Birnberg, J. G. and Nath, R., 1967. Behavioral science implications, The Accounting R e v i m , July, 478-479. Brownell, P., 1983. The motivational impact of management-by-exception in a budgetary context, Journal of Accounting Research, Autumn, 456-472. Bruns, W. J., 1980. The accounting period concept and its effect on management decisions, J o u m l of Accounting Research, November, 34-52. Cook, D. M., 1967. The effect of frequency of feedback on attitudes and performance, Journal of Accounting Research (Suppl), 213-224. Duncan, P. K. and Bruwelheide, L. R., 1985. Feedback: use and possible behavioral functions, Journal of Organizational Behavior Management, 7(3/4), 9 1- 114. Firth, M., 1980. The impact of some MIS design variables of managers' evaluations of subordinates' performance, M I S Quarterly, March, 45-53. Hirst, M. K. and Luckett, P. F., 1992. The relative effectiveness of different types of feedback in performance evaluation, Behavioral Research in Accounting, 1-22. Horngren, C. T . and Foster, G., 1987. Cost Accounting: A Managerial Emphasis, Englewood Cliffs, NJ, Prentice Hall. Ilgen, D. R., Fisher, C. D. and Taylor, S. M., 1979. Consequences of individual feedback on behavior in organizations, Journal of Applied Psychology, August, 349-371. Jensen, R. E., 1967. Discussion of the effect of frequency of feedback on attitudes and performance, Journal of Accounting Research (suppl) 229-234. Keithley, J. P., 1975. Improved use of exception reporting, Internal Auditor, MaylJune, 61-63. Ludwig, S., 1973. The power of praise, International Management, October, 32-34. Martin, G. and Pear, J., 1983. Behavior Modification: What It is and How to do I t , 2nd Edn, Englewood Cliffs, NJ, Prentice Hall. Miller, L. M., 1978. Behavior Management: The N m Science of Managing People at Work, New York, John Wiley. O'Hara, K., Johnson, C. M. and Beehr, T . A., 1985. Organizational behavior management in the private sector: a review of empirical research and recommendations for further investigation, Academy of Management Review, lO(4) 848-864. Potter, B., 1980. Changing Pe?$mance on the Job: Behavioral Techniques for Managers, New York, AMACOM. Press, S. J., 1982. Applied Multivariate Analysis: Using Bayesian and Frequentist Methods of Inference, Malabar, FL, Robert E. Kreiger. Prue, D. M. and Fairbank, J. A., 1981. Performance feedback in organizational behavior management: a review, Journal of Organizational Behavior Management, Spring, 1-16. Reynolds, G. S., 1968. A Primer of Operant Conditioning, New York, Appleton-Century-Crofts. Thompson, D. W., 1978. Managing People: Influencing Behavior, St Louis, MO, C. V. Mosby. Te'eni, D., 1991. Feedback in DSS as a source of control: experiments with the timing of feedback, Decision Sciences 22, 644-655.
Appendix A
Instructions f m the favourable and unfavourabb group You have just been hired as the purchasing agent for the Sunshine Canned Goods Company. It is your job to buy peaches to be canned. Your have three suppliers to buy from. The quality of the peaches sold by these suppliers differs considerably. Since you
288
L. M. Lovata
buy the peaches by the bushel and produce them by the can, the number of cans produced from one bushel will indicate the quality of the peaches sold by a particular supplier. The lower this ratio, the poorer the quality of the peaches since one bushel does not go as far. Attached is Table 1 which contains a listing of the canslbushel ratio of the last 25 purchases made from each supplier. You will be asked to purchase peaches from these suppliers. First, you will receive price quotes from the three suppliers. These prices change every period. Next you will receive a note from the production department indicating the number of cans they will produce the next period. You will then be asked to place your order. You can order any amount from one or more of the suppliers. To enter your order, wait for the question mark, then type in the number of bushels you want from each supplier, separated by commas. After you have done this, the computer will determine the total cost of production. There will be two components of this cost. (1) The first is the cost of the materials you just ordered. (2) Next you will be charged for ordering more or less than was necessary. (a) If you ordered more than was necessary, it will cost $3.50 to dispose of each extra bushel. The peaches cannot be stored for another period. (b) If the quantity ordered was not enough to cover the scheduled production, the production manager will special order what is needed. He has been instructed to Table A1 Number of cans per bushel
Supplier 1
Mean
24.0639 23.8502 26.0983 26.7084 24.0345 26.8824 25.3902 23.8208 26.0712 23.8004 29.2172 26.8983 26.445 8 30.2127 26.6862 25.5305 34.0208 27.9255 27.7872 31.5976 32.8529 25.2732 28.1515 27.6235 29.4663 27.2164
Supplier 2
Supplier 3
The Use of Operant Theory
289
order the peaches from suppliers in the same proportion you did. All suppliers charge 20% more than the stated price per bushel for this service. Whenever there is a favourable or unfavourable deviation from standard of more than 3%, a variance report will be issued. This report shows how you did compared to management's standards. Management determined this standard by observing the average materials cost per can over the past year. The average cost of the peaches has remained relatively constant, so this standard is still considered appropriate and achievable. An example of one iteration is shown below. The dark, underlined items are your responses. To type in numbers use the ten-key pad on the right. After you type them in, hit the return key. To type in 'yes' or 'no' and the carriage return, hit the appropriate key on the top of the keyboard. If while typing the quantity to be ordered, you enter a non-numeric character, the computer will respond 'Redo from start'. If this happens, simply re-enter the quantities you wanted to order from each supplier. Example of One Iteration Prices Der bushel are Supplier 1 $14.30 Supplier 2 $17.90 Supplier 3 $2 1.70 Production is 65 000 cans Quantity to be ordered (in bushels). (Separate the numbers with commas.)
? 200,300,400 The order is for 200 from Supplier 1 300 from Supplier 2 400 from Supplier 3 Is this order correct (yes or no)? yes -
The use of operant theory in the design of performance reporting systems Linda M. Lovata* Operant theory is a branch of psychology that focuses on reinforcement and punishment. In general, reinforcement is preferred over punishment and a combination of both should be optimal. Outcome feedback can provide either reinforcement or punishment depending on the cues transmitted. Therefore, the receipt of only positive feedback should elicit better task performance than the receipt of only negative feedback. Also, a combination of the two should be the best motivator. A laboratory experiment was designed in which subjects participated in a computer simulation of a raw materials purchasing scenario. Performance was measured by costs incurred and decision speed. The group receiving a variance report when costs were outside of a specified range, both positively and negatively, made the quickest decisions without sacrificing cost performance. Receiving only negative performance reports resulted in the worst task performance. Even though the results of the experiment provide very limited support of operant theory, they do suggest that this theory may be useful in the design of better reporting systems. Also, the results indicate that the most common reporting systems may not be the most effective. Key words: performance reporting; operant theory; outcome feedback.
1. Introduction Outcome feedback is an important systems design variable which has received very little attention in the literature. T h e timing of cognitive feedback has been shown to affect decision maker performance (Te'eni, 1991). The study reported here examines the timing of outcome feedback on decision makers. While some anecdotal and experimental evidence is available (At Emery. . . , 1973; Beatty and Schneier, 1975; Hirst and Luckett, 1992), no empirical work has explored the most effective use of outcome feedback. Perhaps the reason for this omission is that only with current technology are alternative systems viable. In the past, systems designers were constrained by the manual processes necessary for report preparation. It often took a week or month to input data prior to processing, so fdes were only updated periodically. The online processing allowed by present technology makes it feasible to implement more creative reporting systems because of the timeliness of the available information. Practitioners and academics alike have espoused the benefits of emphasizing outcome feedback that highlights positive *School of Business, Department of Accounting, Southern Illinois University at Edwardsville, IL 620261104, U.S.A. Received 23 July 1992; accepted 1 November 1992.
01992 Academic Press Limited
274
L. M. Lovata
performance (Birnberg and Nath, 1967; Beatty and Schneier, 1975; Brownell, 1983). This capability would have been expensive several years ago since it requires close traclung of performance, but currently is easily implemented. Therefore, it is now important to investigate the effect of the timing of outcome feedback on decision maker performance. Since the purpose of many internal reports is to motivate users to improve performance (Horngren and Foster, 1987), behaviour modification theory (also called operant theory) may provide insight into ways of increasing the effectiveness of reporting systems. Operant theory states that behaviour is modified based upon the stimulus that occurs immediately following a particular action. Behaviour which is followed by a reward will increase, while behaviour which is punished will be eliminated. This theory, then, may provide guidance as to how to use outcome feedback to maximize decision maker performance. The purpose of this study is to investigate within an operant theory framework the effect of outcome feedback on decision maker performance. The study reveals that the nature of the outcome feedback impacts performance and provides limited support of operant theory. Specifically, providing feedback arbitrarily can interfere with effective decision making, and providing a balance of favourable and unfavourable feedback can enhance performance over reporting only negative results. First, operant theory will be discussed. In the next section, related research is examined. Then operant theory is explained in relation to this particular study and the hypotheses developed. This is followed by a methodology section. The data analysis and results are then discussed, and a summary concludes the paper.
2. Operant theory Operant theory has been tested extensively in the areas of mental health and education often using animals, students or handicapped individuals as subjects. Its application in organizations is referred to as Organizational Behaviour Management (OBM). Originally it was feared that many of the tenets of operant theory only applied to very fundamental behaviours with naive subjects, but research in OBM has found these techniques to be valuable for a wide range of organizational behaviours (O'Hara et al., 1985). Several types of stimuli may occur subsequent to any behaviour. If the stimulus encourages the behaviour to recur, it is considered reinforcement. Rewards such as money or food are considered positive reinforcement if their presence increases the likelihood that the behaviour will recur. Negative reinforcement also increases the likelihood of the behaviour recurring, but this type of reinforcement occurs when an unpleasant stimulus is removed subsequent to the desired behaviour being performed. Any stimulus can be used as reinforcement. In fact, the definition of a reinforcer is simply a stimulus that increases the likelihood that the behaviour will recur. Therefore, the effectiveness of different stimuli will differ across individuals. While reinforcement will increase the likelihood of a behaviour recurring, there are two ways to reduce the likelihood: ignore or punish the behaviour. Behaviours that are ignored, i.e., neither reinforced nor punished, will be reduced. In order to eliminate these behaviours effectively, however, it is important to note that many different stimuli can serve as reinforcement. Even though tangible rewards are eliminated, if other
The Use of Operant Theory
275
s t i m u l ~such as social or intrinsic reinforcers continue, the behaviour will not be extinguished. Punishment is the more direct approach to eliminate behaviour. It consists of either the presentation of an undesirable stimulus or the removal of a pleasant stimuli subsequent to the behaviour. In both instances, the behaviour of interest should decrease. Operant theory warns of the excessive use of punishment. In general, the receipt of only punishment tends to result in behaviour which is, '. . . rigid, non-realisti~,and inappropriately strong' (Thompson, 1978, p. 67). This is due in part to recipients of only punishment receiving clear signals as to which behaviours are inappropriate without equally clear indications of the appropriate behaviours. Accordingly, they tend to overreact to the specific behaviours being punished. One key to using operant theory is to define appropriate rewards and punishment. Although a reward is often associated with money or food, and punishment may be electric shock or removal of privileges, outcome feedback can also serve as rewards or punishment (Duncan and Bruwelheide, 1985; Miller, 1978; Potter, 1980). In fact, within an organization, outcome feedback has the advantage over many rewards in that it is inexpensive and easy to administer (Miller, 1978, pp. 116- 117). When feedback is viewed in this manner, there are pervasive implications for the design of information systems. If feedback is favourable, it may reinforce behaviour; if it is negative, it is punishment (Ilgen et al., 1979). According to operant theory, then, the type of feedback received is critical to the subsequent behaviour of the decision maker. The majority of reporting systems currently used in organizations ignore this behavioural aspect of reporting, issuing reports based on an arbitrary passage of time, or if contingent on performance, emphasizing only negative performance deviations (Birnberg and Nath, 1967; Brownell, 1983; Keithley, 1975; Ludwig, 1973). The management report most directly related to rewards or punishment is a performance report. This report generally relates performance to a standard, budget, or goal.' If operant theory holds, task performance should improve when reports are issued based upon deviations from expected performance rather thsn the arbitrary passage of time. Also, highlighting favourable performance in addition to unfavourable performance should elicit better behaviour than emphasizing only unfavourable performance. Since prior research has not integrated operant theory and accounting reports, this study examines one specific report to determine its effect. While several different types of rewards can be made contingent on a behaviour, for this preliminary study of operant theory, only one potential stimulus will be isolated. Once tested, then subsequent studies can examine other reward structures to determine their incremental effects. The study reported here investigates variance reports within the context of operant theory. The results provide some general support for the use of operant theory in the design of variance reporting systems. 3. Prior research
Two areas of research relate to this investigation. The first studies deal with time-based reporting. Though many of these studies examine aggregation levels, contrasting, for example, weekly and monthly reports (Bruns, 1980; Firth, 1980), one study specifically 'Feedback is especially effective when it compares performance to a goal or standard (Potter, 1980). Therefore, a variance report was chosen as the mode of feedback.
276
L. M. Lovata
tests reporting intervals in combination with aggregation levels (Cook, 1967) and finds that more frequent reporting yields the best performance. Cook (1967) divided teams of students participating in a simulated business game into three groups. One group received the results of their past decisions prior to making subsequent ones. A second group received reports presenting the results of the four prior decisions every fourth period, while a third group received no feedback until the end of the game. The group receiving the most feedback performed the best. This study was criticized on several dimensions (Becker, 1967; Jensen, 1%7), two of which will be discussed here. First, the information withheld was critical to the successful completion of the business game. Second, the group participation over an entire semester made communication among groups likely. The study reported in this paper attempts to circumvent these problems in a similar decision environment. The manipulated report is not essential to the decision maker's analysis of the task. Also, the business simulation developed for this study can be completed in one session so as to diminish the probability of communication among participants. Two recent studies by Te'eni (1991) and Hirst and Luckett (1992) also examine the timing of outcome feedback. Hirst and Luckett (1992) find that when the task is perfectly predictable, outcome feedback is useful. Te'eni (1991) finds that the timing of cognitive feedback influences performance. Both studies manipulated reports based on periodicity, not performance. A second area of related research lies in the management by exception (MBE) literature. Two studies have systematically tested the effects of MBE and the results of both are equivocal. The first by Brownell (1983) examines MBE in an expectancy theory framework. By analysing 122 responses to a questionnaire, he tests both the use of favourable and unfavourable exception reporting and the interaction among participation, motivation, and MBE. The questionnaire measured participation through established, validated scales and motivation by eliciting intrinsic and extrinsic valences (utilities) for 17 scenarios. The use of MBE was determined by responses to questions dealing with when superiors investigate variances. Brownell finds that although negative exception reporting and motivation are inversely related as hypothesized, the relationship is not statistically significant. Also, the effect of MBE on motivation may differ given the level of budgetary participation. The finding most relevant to this study is that there is, in fact, a bias toward emphasizing unfavourable variances. Benbasat and Schroeder (1977), using an experimental computer simulation (business game), operationalize MBE by placing an asterisk beside reports revealing greater than a 10% change from the prior period. This manipulation did not provide significant changes in performance. Benbasat and Schroeder state: 'While this (lack of significance) may lead to the conclusion that exception reporting is ineffective, we feel that the size and nature of the reporting system used in the experiment were factors that reduced the potential effectiveness of exception reporting' ( P 46).
They then appeal for more research in the area. While Brownell focuses on the unfavourable-only type of exception reporting and Benbasat and Schroeder investigate the favourable and unfavourable condition, no study has contrasted these types of reporting systems. Also, prior accounting studies
The Use of Operant Theory
277
have not examined the differential impact due to the content of the performance reports. The management literature discusses the importance of this distinction. In the OBM literature, several authors have noted the need to better specify feedback content in order to assess the reinforcing nature of reports. As Ilgen et al. (1979) state: 'Theoretically, to assume that feedback per se is a positive reinforcer ignores the fact that feedback varies along some positive-to-negative continuum. It is well established in the operant conditioning literature that positive reinforcers have different effects on responses than does punishment (Reynolds, 1968). It is difficult to conceive of negative feedback serving as a positive reinforcer in many settings. Yet, much of the research on the frequency of feedback fails to deal with the sign of the feedback. . .' (p. 360).
More recently, several review articles have noted the lack of empirical research in this area (Prue and Fairbank, 1981; Balcazar et al., 1985; Duncan and Bruwelheide, 1985). The study reported in this paper attempts to address these omissions in OBM literature. The sign of the feedback is explicitly considered and analysed within the operant theory framework. In the next section, this framework is used to develop the specific hypotheses tested.
4. Operant theory and hypotheses development While periodic reporting transmits information regarding past performance, operant theory suggests that certain types of reports are more effective if they are tied directly to exceptional performance. The reports that would be appropriate for this type of schedule are those dealing with outcome feedback, which Prue and Fairbank (1981) define as being information concerning the quantity or quality of past performance. Limiting reports as suggested by operant theory is not applicable to reports that relay critical data concerning the decision environment. Therefore, the reporting systems for which operant theory may apply include performance evaluations whether they be quality control reports, personnel evaluations, or variance reports. These types of reports can exhibit reinforcing qualities if they are issued only when the desired behaviour is observed. While traditional reporting systems that issue reports at regular intervals, such as weekly or monthly, are effective, this theory suggests that the value of performance reports will increase if received based upon significant performance variations. Accountants refer to this type of system as exception reporting, in which performance reports are received only when deviations from standard are observed. Unfortunately, the most common type of exception reporting system is one where negative variances are scrutinized (Brownell, 1983; Birnberg and Nath, 1967). In operant theory terms, this provides only punishment to the decision maker. At the other extreme is an exception reporting system that reports only positive variances. With this system, rewarding reports are received for good performance while poor performance is ignored. A derivative of rhis type of system was implemented at Emery Air Freight and Michigan Bell with remarkably positive results (At Emery. . . , 1973; Beatty and Schneier, 1975). Emery Air Freight chose to exclude monetary incentives in their positive reinforcement program due primarily to practical problems in administering monetary rewards. They reported sales and productivity increases resulting in estimated benefits of over $3 million per year. Operant theory predicts that this type of system results in better task performance than the unfavourable only system.
278
L. M. Lovata
A practical problem with a strictly positive system is that cues are not provided when performance is out of control. Extremely poor performance may be allowed to continue for an extended period of time before an individual recognizes the problem and corrects it, which may result in frustration on the part of the decision maker (Martin and Pear, 1983, p. 53). Accordingly, a complete exception reporting system that generates reports when performance is exceptionally favourable as well as unfavourable should be optimal. This system balances rewards and punishment and prevents the situation where performance is extremely poor unbeknownst to the decision maker. This type of system should discourage an individual from continuing poor performance while rewarding appropriate responses. Therefore, the hypotheses to be tested require an exception reporting framework. In order to operationalize the exception reporting systems, a range of acceptable but not exceptional performance must be specified. If this is not done, then receiving both positive and negative feedback would be equivalent to receiving a report every period. Exception reporting systems that emphasize favourable performance should elicit better performance than those concentrating on negative performance, and a balance of favourable and unfavourable reporting should yield the highest level of performance. Two hypotheses are developed for this analysis. First, the best contingent reporting system (i. e., exception reporting system) should be superior to non-contingent reporting schemes. Stated in the alternative form: H1: Receipt of both favourable and unfavourable performance reports will increase task performance over receipt of reports on a non-contingent basis.
Second, receipt of both favourable and unfavourable reports should outperform the receipt of only favourable reports which should outperform the receipt of only negative reports. H2: Receipt of both favourable and unfavourable performance reports will increase task performance over receipt of only unfavourable or only favourable performance reports.
In order to provide strong support for operant theory, the null of both hypotheses should be rejected. While there is no explicit theory to predict how the less desirable contingent reporting systems compare with non-contingent systems, it is anticipated that positive reporting systems will increase task performance. Given the negative impact of receiving only punishment, the relation between the unfavourable only group and the others is unclear. Therefore, all groups will be compared in order to discern relationships. Table 1 presents a summary of the hypothesized results.
5. Method Control over the lag between the decision making activity and the receipt of the report is crucial because operant theory states that the reward or punishment must follow the behaviour as rapidly as possible. As a result, the experimental materials were presented to subjects on microcomputers as a business game. By using a computer simulated business game, the time lag is controlled precisely, and decision time and cost performance are measured unobtrusively.
The Use of Operant Theory
279
Table 1 Hypothesized dtfferences*
No report Fourth Always Unfavourable Favourable
Fourth
Always
Unfavourable
Favourable
Both
?
? ?
? ? ?
BETTER BETTER BETTER BETTER
BETTER BETTER BETTER BETTER BETTER
*BETTER indicates performance is hypothesized to be better for the group with the heading across the top of the table.
Subjects Sixty graduate business students were randomly assigned to one of six groups, resulting in 10 subjects per cell. The average participant was 26-years-old with three years of work experience. The experiment required no special knowledge or experience since the task was self-contained. Also, it was a decision making task not an attitudinal test. Ashton and Kramer (1980) provide an extensive review of the literature contrasting business people with students and conclude that student's performance is not significantly different in decision malung tasks. Since this is an experiment concerning basic human behaviour, not expert judgment, students were deemed a p p r ~ p r i a t e . ~ Each subject participated in the experiment individually, and the entire procedure was completed in one session. Variables The independent variable in this research is the timing of the variance report. The variance report is based only on the most current decision period and in no way accumulated the results of prior decisions. The timing of the report, then, not the lag factor or aggregation level, is manipulated. Six levels of this variable are tested: (1) (2) (3) (4)
Variance report never received (No Report); Variance report received every fourth period (Fourth); Variance report received every period (Always); Variance report received only when an unfavourable variance greater than 3% of standard is observed (Unfavourable); (5) Variance report received only when a favourable variance greater than 3% of standard is observed (Favourable); and (6) Variance report received when either a favourable or unfavourable variance greater than 3% of standard is observed (Both). Subjects were assigned to one of the six treatment conditions and received only that treatment for the duration of the experiment. The two performance measures used as dependent variables were cost performance and decision time. These were measured unobtrusively by the simulator, and the averages for each subject used. The time measure is examined because it has been shown that the lag between subsequent responses is often directly related to rewards or 2This was confirmed when analysis showed that age and experience did not affect decision maker performance.
280
L. M. Lovata
punishment (Miller, 1978; Martin and Pear, 1983). From an operational perspective, managers are interested in achieving the best decisions in the least amount of time. Therefore, a performance reporting system that reduces the time spent making a good decision is valuable.
Experimental procedures First, a consent form and a short demographic questionnaire were completed. Next, the instructions were read and the simulation performed. Subjects were told to do their best when completing the simulation. While they were not specifically instructed to minimize decision time, they had the natural incentive to finish quickly. Finally, the subject was debriefed. A raw materials purchasing scenario was simulated in which subjects were told to buy peaches for a cannery from three suppliers. The first supplier had inexpensive peaches of low quality. The second had medium priced peaches with quality to match, while the third had high priced, high quality peaches. Quality was measured by the number of cans produced per bushel. The purchasing decision was repeated 24 times with the price and the quality of peaches varying each period though the general quality differences among the three suppliers remained constant. The subjects had to estimate trends in the quality of the suppliers' goods which indicated that the quality of supplier one's goods improved significantly over the duration of the experiment. Therefore, if the subject correctly estimated these trends, he or she would originally purchase mostly from suppliers two and three, then slowly move toward purchasing more from supplier one. The subject could detect these trends through past data presented with the instructions and from the results of his/her decisions. After each decision, a production report and a variance report (Figure 1) were presented on the CRT. The production report followed directly from the subject's purchase decision. If more was ordered than necessary, a $3.50 disposal charge was incurred on each unused bushel; if a shortage occurred, there was a 20% rush order charge for each additional bushel. On the production report, these two charges constitute the difference between the 'cost of materials ordered' and the 'total cost'. All groups received the production report every period. In this way, one group did not have an unfair advantage over the others. If this was the manipulated report, the group receiving it every period would have more information upon which to estimate the various distributions. Finally, the variance report was presented when applicable.3 A favourable or unfavourable variance report was issued dependent upon the subjects' performance and the cell to which they were assigned. Their performance relative to the standard dictated if the report was favourable or unfavourable. Favourable variance reports were accompanied by a statement indicating 'GOOD JOB' while unfavourable variance reports stated 'YOUR COSTS ARE TOO HIGH'. The variance report was shown to the subject only when appropriate given their cell assignment. One group of subjects never received the variance report, while another group received it after each decision, and a third group received it after every fourth decision. The exception reporting 3The variance was not divided into the price and efficiency components because in this case the two were inversely related and both were the responsibility of the purchasing agent. If poor quality goods were purchased, a favourable price variance and an unfavourable efficiency variance resulted. The purchasing agent was responsible for buying products that improved the overall variance so this was the only one presented.
The Use of Operant Theory PRODUCTION REPORT Quantity ordered Cans per bushel
Supplier
281
Price
Scheduled production = 65 000 cans Amount produced from materials ordered = 37 242 cans Cost of materials ordered = $16 910 Total cost = $32 108 VARIANCE REPORT The standard cost is $29 900 There is an unfavourable variance of: $2 208 YOUR COSTS ARE TOO HIGH
groups received the variance report if their performance on that decision resulted in the specified deviation from standard. Finally, a short debriefing session was held. The subject was asked to specify the decision rules used, and if he or she remembered receiving any variance reports. The subjects accurately recalled the reason they received a variance report and the number of variance reports received. This provides some assurance that the variance report was not overlooked. Lastly, the subject was free to ask questions concerning the experiment.
Pilot study A pilot study using 20 additional subjects was conducted to determine a proper standard and the number of decisions to be made. Table 2 depicts the number of reports that would have been issued in the pilot study given three possible standard costs per can. Since it is essential for each group to perceive their goal level to be equal, 46 cents per can was used in the actual study. Also, the three percent deviation from this standard was found to be difficult yet achievable. Finally, the pilot study determined the number of decisions to be made. Twenty-four iterations surpassed the learning curve while avoiding a fatigue factor. Table 2 Reports issued with diffmmt standards Standard
Number Number Number Number
of favourable reports 142 of unfavourable reports 338 70 surpassing 3% favourable surpassing 3% unfavourable 219 Total number of observations = 480
231 249 120 123
330 150 191 83
282
L. M. Lovata
6. Data analysis and results The mean cost for all groups was $31 482.90; the mean variance from standard cost was $182.34; and 55% of the variances were favourable while 45% were unfavourable. This nearly 50150 split supports the use of the pilot study for determining a standard. Within the exception reporting groups, each subject received an average of four negative and/or seven positive variance reports, therefore the 3% exception reporting level was sufficient to generate reports. The data analysis for this study is a three-part process. First, the number of iterations to be used for the analysis is determined. Second, two manipulation checks are performed, and finally, F-tests are used to determine the significant treatment variables.
Number of iterations During the first several decisions, a strong learning trend is observed. For decision time and cost effectiveness, the average of the iterations is used as the dependent ~ a r i a b l e If . ~ the first several trials are included in this average, the decision time and costs would be mis-stated since they may include differential learning produced by the treatments. Regression analysis is performed on the average decision time, measured in minutes, versus the iteration. The resultant power function (TIME = 55.84 ITERATION-^.^'^ with an R2 of 0.98) reveals that the majority of learning occurred in the first seven iterations (see Figure 2). These seven iterations, therefore, are excluded from the remainder of the analysis. As for the last iterations, there is not a dramatic drop in the tail of the time plot so there does not appear to be a fatigue factor. Subjects were told the total number of iterations to be simulated, however, so to control for possible end-of-game strategies, the 24th iteration is also excluded from further analysis. Of the 24 iterations performed by each subject, then, iterations eight through 23, inclusive, or 16 iterations, are averaged to determine each subject's costs and decision time. 7
-E
.=
E
F
.
1
1
1
,
,
l
l
l
l
l
l
,
I
l
l
l
l
l
l
l
l
l
l
l
'
6-
-
5-
-
4-
-
3-
-
2-
-
-
-
I
o
~
I
I
I
I
I
I
I
I
I
I
~
I
I
I
I
I
~
~
~
I
0 1 2 3 4 5 6 7 8 9 1011 I 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 Iteration
Figure 2. Learning curve. The hypotheses were also tested using repeated measures analysis with the same results. The averaging method is presented in the paper to facilitate the reporting of results.
I
I
~
The Use of Operant Theory
283
Decision time is inverted to depict decision speed in order to normalize that distrib~tion.~ The higher the number, then, the quicker the decision.
Manipulation checks One criticism of many OBM studies on the effects of feedback is that the information value of feedback is confounded with the reinforcement aspect (Duncan and Bruwelheide, 1985). It is difficult in many studies to determine whether performance is improved because of the information gained from feedback, or because feedback acts as a reinforcer for good performance. Therefore a control group is necessary to insure that the reinforcement quality and not the information content of the manipulated reports is triggering the responses. A group that always receives the report is contrasted to one never receiving it. If this test reveals differences between the groups, then the information transmitted in the performance report may have allowed users to make better decisions. Alternatively, simply knowing that a standard exists against which performance is being measured may elicit a reaction. In order for this experiment to test the reinforcing qualities of performance reporting, it is important for the report not to provide any incremental information which enables individuals to make better decisions. Ideally, the report only highlights performance deviations thereby causing the decision maker to better attend to the essential data. This comparison will test whether the information content or the existence of a standard are confounding factors in the analysis of the variance reports. The means of all groups are presented in Table 3. When contrasting those always receiving the variance report with the group never receiving the report, neither costs (F= 1.62; P = 0.209) nor the decision speed (F = 0.01; P = 0.932) is significantly different. Therefore there is no data essential to decision quality provided by the variance report nor does the mere existence of a standard appear to impact decisions. Any differences discovered between other groups are not due to these alternative hypotheses. Another manipulation check is performed to ensure that simply the intermittent receipt of a performance report does not cause the user to better attend to the data. Reports received intermittently based on the passage of time, not performance, should not improve decision making. Operant theory states that rewards and punishment must Table 3. Cell means
N o report Fourth Always Unfavourable Favourable Both
N
Costs
Speed (l/Time)
Minutes/Decision
10 10 10 10 10 10
31 628.0 32 252.0 31 122.1 31 612.9 31 354.3 31 274.8
481.3 720.6 472.4 464.5 597.8 661.2
2.08 1.39 2.12 2.15 1.67 1.51
Since there was a practical minimum and maximum time to complete the task, the distribution of time, and therefore the distribution of errors, was not normal (skewness of 1.5, kurtosis of 3.8). Several transformations were performed with the inverse resulting in the best distributional qualities (skewness of 0.6 and kurtosis of -0.05).
284
L. M. Lovata Table 4. Univariate F -tests
Costs* Speed*
Model df
F-Value
P-value
5 5
2.02 2.22
0.09 0.07
Significant at the 0.10 level.
be contingent on behaviour to favourably influence performance. If a performance report is received for other reasons, it should not be as effective as when the report is received due to performance deviations. Therefore, the group receiving the variance report every fourth period is contrasted to the both favourable and unfavourable group. Examining the means of the intermittent group on Table 3 reveals that overall the decision speed of the group receiving the report every fourth period is quicker than most other groups while costs are generally higher. Relative to the both favourable and unfavourable group, receipt of the report every fourth period does not elicit significant differences in the speed of the decisions (F = 0.33; P = 0.285) but does result in significantly higher costs (F = 6.03; P = 0.009). Therefore, the receipt of intermittent reports in and of itself does not improve performance. In fact, receiving the report based on this arbitrary passage of time encouraged subjects to make hasty, suboptimal decisions. This emphasizes the importance of making intermittent reports contingent upon behaviour .
Treatment variables The manipulation checks provide evidence that refutes several alternative hypotheses. This increases the credibility of the multivarite analysis of variance conducted to test the hypotheses. The Wilks' lambda produced by this test simultaneously examines both dependent variables (cost performance and decision speed) to determine if there are significant relationships among the dependent and independent variables. The test does not indicate which specific relationships are significant. To discern the significant relationships, the dependent variables must be examined independently through the use of univariate analysis of variance (Press, 1982, pp. 97-98). The overall Wilks' lambda is significant (lambda = 0.712; P = 0.04) so univariate tests on each dependent variable are performed and the results presented in Table 4. These results indicate that both decision speed and costs contributed marginally to the significant overall lambda. Contrasts are performed to determine the significant differences among these means (see Table 5). Discussion The results presented in Table 5 are now discussed in relation to the hypotheses presented earlier in this paper.
Hypotheses Hypothesis one: both versus non-contingent reporting. None of the groups achieved significantly lower costs than the both favourable and unfavourable group, and the
The Use of Operant Theory
285
Table 5. Dtffmmes among means t-values (P-values) Fourth No report
Costst Always Unfavourable
-624.0 2.46 (0.12)
505.9 1.62 (0.21) 1129.9** 8.06 (0.06)
-239.3" 5.28 (0.03)
8.9 0.01 (0.93)
Fourth
Favourable
Both*
Always
Unfavourable
Favourable
No report
Fourth
Always
Unfavourable
Favourable
t Negative sign indicates costs were lower or decision speed slower for the group with the heading in the left-hand column. $ Since a direction is hypothesized, the both favourable and unfavourable tests are one-tailed. * Significant at the 0.05 level. ** Significant at the 0.01 level. group receiving the report every fourth period incurred significantly higher costs. With respect to decision speed, the group receiving both favourable and unfavourable reports worked significantly faster than all but the fourth group. Examining costs in conjunction with speed reveals that the both favourable and unfavourable group performed significantly faster than the always and never groups without sacrificing cost performance. They also worked at the same pace as the fourth group while achieving
286
L. M. Lovata
significantly lower costs. Therefore, the both group's overall performance (low costs combined with quick decision speed) was superior to all of the fixed interval reporting systems.
Hypothesis two: both versus only favourable versus only unfavourable. Among the contingent reporting systems, the means of both cost performance and decision speed are in the hypothesized direction, but the significance levels are lacking. The means of both decision speed and costs are ranked as predicted, with the combination of favourable and unfavourable reporting yielding the quickest decision speed and the lowest costs. The mean cost of the favourable only group ranks second while the unfavourable group is third. Unfortunately, the only significant difference among these means is the comparison between the decision speed of the balance of favourable and unfavourable reports with the unfavourable only group. This particular contrast is especially important, however, since the majority of exception reporting systems take the negative reporting approach (Brownell, 1983).
7. Conclusions While the tenets of operant theory are only partially supported in this study, the limited results do suggest that this theory may be useful in the design of reporting systems. Given the number of cells and the sample size, it is apparent that the power of the tests is weak. Also, a limitation of this study is the time involved. In an actual work situation, decisions and the related performance reports occur weeks or even months apart. In this experiment the reports were received only minutes apart. Even within this short decision period, decision time is affected by the performance reports. Therefore, it is hypothesized that the effect could be strengthened by extending the reporting interval, but only continued research can reveal the actual affect of an increased time horizon. Also, in this experiment subjects were only instructed to do their best, while in other settings performance feedback is often associated with additional reinforcers such as praise, social interaction, or monetary rewards. Prior research has shown that providing additional rewards in conjunction with feedback enhances its effectiveness (Balcazar et al., 1985). Therefore, combining contingent feedback with other rewards, such as pay, should increase management's power to modify behaviour in the desired direction. The results presented here suggested that when performance reports are viewed as rewards or punishment, the most frequently used reporting systems do not utilize the power of feedback efficiently. First, most reporting systems issue reports based upon some arbitrary passage of time rather than making them contingent upon performance. Second, those systems that do relate reporting to behaviour generally emphasize the negative aspects of performance. Acknowledgements: Financial support for this research was generously provided by the IBM
Corporation. The author would like to thank Milton Jenkins, Mike Groomer, Les Heitger, Yaw Mensah, Mike Tiller, Sonia Goltz, Dave Cowan, Nancy Carter and two anonymous reviewers.
References Ashton, R. H. and Kramer, S. S., 1980. Students as surrogates in behavioral accounting research, Jounull of Accounting Research, Spring, 1-15.
The Use of Operant Theory
287
1973. At Emery Air Freight: Positive Reinforcement Boosts Performance, Organizational Dynamics, Winter, 41-50. Balcazar, F., Hopkins, B. L. and Suarez, Y., 1985. A critical objective review of performance feedback, Journal of Organizational Behavior Management, 7(3), 65-89. Beatty, R. W. and Schneier, C. E., 1975. A case for positive reinforcement, Business Horizons, April, 57-66. Becker, S. W., 1967. Discussion of the effect of frequency of feedback on attitudes and performance, J o u m l of Accounting Research (Suppl), 225-228. Benbasat, I. and Schroeder, R. G., 1977. An experimental investigation of some MIS design variables, M I S Quarterly, March, 37-48. Birnberg, J. G. and Nath, R., 1967. Behavioral science implications, The Accounting R e v i m , July, 478-479. Brownell, P., 1983. The motivational impact of management-by-exception in a budgetary context, Journal of Accounting Research, Autumn, 456-472. Bruns, W. J., 1980. The accounting period concept and its effect on management decisions, J o u m l of Accounting Research, November, 34-52. Cook, D. M., 1967. The effect of frequency of feedback on attitudes and performance, Journal of Accounting Research (Suppl), 213-224. Duncan, P. K. and Bruwelheide, L. R., 1985. Feedback: use and possible behavioral functions, Journal of Organizational Behavior Management, 7(3/4), 9 1- 114. Firth, M., 1980. The impact of some MIS design variables of managers' evaluations of subordinates' performance, M I S Quarterly, March, 45-53. Hirst, M. K. and Luckett, P. F., 1992. The relative effectiveness of different types of feedback in performance evaluation, Behavioral Research in Accounting, 1-22. Horngren, C. T . and Foster, G., 1987. Cost Accounting: A Managerial Emphasis, Englewood Cliffs, NJ, Prentice Hall. Ilgen, D. R., Fisher, C. D. and Taylor, S. M., 1979. Consequences of individual feedback on behavior in organizations, Journal of Applied Psychology, August, 349-371. Jensen, R. E., 1967. Discussion of the effect of frequency of feedback on attitudes and performance, Journal of Accounting Research (suppl) 229-234. Keithley, J. P., 1975. Improved use of exception reporting, Internal Auditor, MaylJune, 61-63. Ludwig, S., 1973. The power of praise, International Management, October, 32-34. Martin, G. and Pear, J., 1983. Behavior Modification: What It is and How to do I t , 2nd Edn, Englewood Cliffs, NJ, Prentice Hall. Miller, L. M., 1978. Behavior Management: The N m Science of Managing People at Work, New York, John Wiley. O'Hara, K., Johnson, C. M. and Beehr, T . A., 1985. Organizational behavior management in the private sector: a review of empirical research and recommendations for further investigation, Academy of Management Review, lO(4) 848-864. Potter, B., 1980. Changing Pe?$mance on the Job: Behavioral Techniques for Managers, New York, AMACOM. Press, S. J., 1982. Applied Multivariate Analysis: Using Bayesian and Frequentist Methods of Inference, Malabar, FL, Robert E. Kreiger. Prue, D. M. and Fairbank, J. A., 1981. Performance feedback in organizational behavior management: a review, Journal of Organizational Behavior Management, Spring, 1-16. Reynolds, G. S., 1968. A Primer of Operant Conditioning, New York, Appleton-Century-Crofts. Thompson, D. W., 1978. Managing People: Influencing Behavior, St Louis, MO, C. V. Mosby. Te'eni, D., 1991. Feedback in DSS as a source of control: experiments with the timing of feedback, Decision Sciences 22, 644-655.
Appendix A
Instructions f m the favourable and unfavourabb group You have just been hired as the purchasing agent for the Sunshine Canned Goods Company. It is your job to buy peaches to be canned. Your have three suppliers to buy from. The quality of the peaches sold by these suppliers differs considerably. Since you
288
L. M. Lovata
buy the peaches by the bushel and produce them by the can, the number of cans produced from one bushel will indicate the quality of the peaches sold by a particular supplier. The lower this ratio, the poorer the quality of the peaches since one bushel does not go as far. Attached is Table 1 which contains a listing of the canslbushel ratio of the last 25 purchases made from each supplier. You will be asked to purchase peaches from these suppliers. First, you will receive price quotes from the three suppliers. These prices change every period. Next you will receive a note from the production department indicating the number of cans they will produce the next period. You will then be asked to place your order. You can order any amount from one or more of the suppliers. To enter your order, wait for the question mark, then type in the number of bushels you want from each supplier, separated by commas. After you have done this, the computer will determine the total cost of production. There will be two components of this cost. (1) The first is the cost of the materials you just ordered. (2) Next you will be charged for ordering more or less than was necessary. (a) If you ordered more than was necessary, it will cost $3.50 to dispose of each extra bushel. The peaches cannot be stored for another period. (b) If the quantity ordered was not enough to cover the scheduled production, the production manager will special order what is needed. He has been instructed to Table A1 Number of cans per bushel
Supplier 1
Mean
24.0639 23.8502 26.0983 26.7084 24.0345 26.8824 25.3902 23.8208 26.0712 23.8004 29.2172 26.8983 26.445 8 30.2127 26.6862 25.5305 34.0208 27.9255 27.7872 31.5976 32.8529 25.2732 28.1515 27.6235 29.4663 27.2164
Supplier 2
Supplier 3
The Use of Operant Theory
289
order the peaches from suppliers in the same proportion you did. All suppliers charge 20% more than the stated price per bushel for this service. Whenever there is a favourable or unfavourable deviation from standard of more than 3%, a variance report will be issued. This report shows how you did compared to management's standards. Management determined this standard by observing the average materials cost per can over the past year. The average cost of the peaches has remained relatively constant, so this standard is still considered appropriate and achievable. An example of one iteration is shown below. The dark, underlined items are your responses. To type in numbers use the ten-key pad on the right. After you type them in, hit the return key. To type in 'yes' or 'no' and the carriage return, hit the appropriate key on the top of the keyboard. If while typing the quantity to be ordered, you enter a non-numeric character, the computer will respond 'Redo from start'. If this happens, simply re-enter the quantities you wanted to order from each supplier. Example of One Iteration Prices Der bushel are Supplier 1 $14.30 Supplier 2 $17.90 Supplier 3 $2 1.70 Production is 65 000 cans Quantity to be ordered (in bushels). (Separate the numbers with commas.)
? 200,300,400 The order is for 200 from Supplier 1 300 from Supplier 2 400 from Supplier 3 Is this order correct (yes or no)? yes -