Chapter 5 Cognitive Feedback

HUMAN JUDGMENT: The SJT View Berndt Brehmer & C.R.B. Joyce (editors) @ Elsevier Science Publiahers B. V. (North-Holland). 1988

163

CHAPTER 5

COGNITIVE FEEDBACK Michael E. Doherty and William K. Balzer Bowling Green S t a t e University

"Our understanding of the way things are interweaves great recurring themes, plays upon certain fundamental and all-informing ideas. The number of the themes of science is surprisingly small.... feedback is the infant among these few great themes" (Judson, 1980, p. 88-89 )

.

In spite of the centrality of the principle of feedback in twentieth century attempts to comprehend nature, it was first articulated as a general principle in this century, and the term "feedback" first appeared in print, according to Judson (1980), during the lifetime of the person honored by this volume. The d e f i n i t i o n of feedback. While the term feedback (FB) has been used in a variety of ways in different disciplines, by definition it involves an environment that returns some measure of the output of a system back to the system which produced that output. The FB then allows the system to compare its present state with an ideal state, to adjust itself in light of that comparison, and bring itself closer to that ideal state. For our purposes, the system is normally a person, but may sometimes be two or more people. This chapter deals with cognitive feedback (CFB), or the return of some measure of the output of a person's cognitive processes, to help that person come to terms with the environment. Before describing CFB in detail, it will be useful to mention three major currents in contemporary cognitive psychology that are directly relevant to the present

164

M. E. Doherty and W. K. Balzer

chapter. These themes have implications for the understanding of the current work on CFB and f o r our speculations about its future roles. Limitations on cognitive abilities. One general implication of research on cognitive processes, from James (1890) to Kahneman, Slovic and Tversky (1982), is that humans have limited cognitive abilities. These limitations are not confined to the well-known capacity-related attributes of cognition such as attention span and short term memory; they also extend to our abilities to deal with environmental uncertainty, learn function forms and weights, assess covariation, abandon falsified hypotheses (or perhaps know when our hypotheses have been falsified), understand ourselves, etc. The implication is that we need help. Cognitive aids. Several investigators, Hammond and Edwards to name two, have decried the tendency of psychologists to study what might be called "Man the Intuitive Barbarian". That is, in spite of the fact that human beings are tool-users pat excellence, cognitive psychologists typically study people in situations in which they are not permitted to use those tools for thought normally used in everyday commerce with the environment. Such investigations may be of interest for what they tell us about unaided cognition, but they do not tell us about cognition in the tool-using environment in which we have evolved and to which we have adapted. Edwards (1978; von Winterfeldt & Edwards, 1986) has championed "decision analysis" as a cognitive aid; Hammond (1975; Hammond C Adelman, 1976), on the other hand, has pursued the use of "cognitive feedback". Both forms of cognitive aid involve the decomposition and externalization of a complex decision problem. Knowledge representation. In the work explored in this chapter, the knowledge to be represented is a complex set of relations called a "policy". Much of the research on knowledge representation has been concerned with issues such as whether images or list structures are a more parsimonious explanation of experimental data, the nature of scripts and schemas, and artificial intelligence (Gardner, 1985). One of

Cognitive feedback

165

the assumptions underlying much of that research is that "people in general" can be described in a particular way: that is, much of cognitive psychology has been guided by an underlying nomothetic bias. In contradistinction to the nomothetic approach, we assume not only that different people may use different mental representations of knowledge, but that a given person can effectively use a variety of representations. In Brunswik's terminology, this locates a form of vicarious functioning in the central region rather than in the proximal/peripheral region (see Hammond, 1981, p. 4). Considering that multiple environmental representations of the same object are now commonplace (e.g., an automobile may be represented visually by drawings, black and white photographs, color photographs, toys, models, moving two-dimensional representations in films or on television, etc. with all of these varying in size, orientation, and other attributes), our assumption that any individual may routinely use multiple mental representations of the same object is almost required by Brunswik's (1952) principle of parallel concepts.

What is cognitive feedback? A fundamental assumption of Brunswik's probabilistic func-

tionalism is that the basic unit of cognition, or knowing, is the relationship. This refers to relationships among cues, between cues and distal objects (or criteria), between cues and cognitions, and between cognitions and distal objects. This assumption is epitomized in what previous investigators have called CFB, which provide the person with information describing the relationships: (a) between cues and the criterion, i.e., information about the task (Task Information: TI): (b) between cues and the person's inferences, i.e., information about the person's cognitive state (Cognitive Information: CI): and/or (c) between cognitions and the distal objects. This third category comprises indices of "functional validity"

166

M. E. Doherty and W. K. Balzer

information ( M I ; Brunswik, 1956, p. 30). This category system (to be used in this chapter) is an elaboration of that provided by Hammond, McClelland & Mumpower (1980; p. 22 8 ), with the third category added to reflect the lens model more faithfully and to allow the system to incorporate all lens model indices. TI includes the indices on the ecological side of the lens (note that TI may be information about another person, as in interpersonal learning and conflict reduction research); CI includes all the information on the subject's side. FVI will be used somewhat more broadly than Brunswik's term "functional validity" (see also Nystedt Ei Magnusson, 1973), and will refer to all three measures (ra, G, and C) of the wide-arching dependencies so central to Brunswik's conceptual system. CFB operations, which often conjoin TI, CI, and FVI using a graphic display, epitomize Brunswik's lifelong call for a psychology that is not encapsulated within the organism but one that is at least equally concerned with an understanding of the environment. Furthermore, and as described below, presenting TI and CI in the same forms and joining them in a single display is literally a graphic embodiment of Brunswik's (1952) principle of parallel concepts. Consider a person trying to come to terms with an uncertain environment. A fundamental task of such a person is to obtain valid knowledge of objects in that environment. Generally, knowledge Is of some object in depth, which is inferred from a set of cues to, or environmental effects of, the thing to be known. This endeavor Is often represented experimentally by multiple cue probability learning (MCPL) studies. In a typical MCPL investigation, the person is presented with an array of cues that are related probabllistically to a criterion, using a sufficient number of trials to permit idiographic/statistical analyses of each person. In the description of the lens model by Hammond and Summers (1972), the cues are denoted Xi, the criterion Ye, and the jud$ment Ys. Values of the criterion and the judgment predicted from best fitting linear models are denoted Y', and

Cognitive feedback

167

YVs, respectively. Metrics commonly used to indicate the relations between cues and criterion and between cues and judgments include correlation coefficients and beta weights. These are transformed in various ways to communicate information to the person to whom TI and/or CI is being provided (Schmitt & Levine, 1977). Some basic terminological distinctions must be drawn. In earlier descriptions of CFB, investigators have routinely considered under the heading of CFB the three components that we are separating. We will be consistent with the conceptual definition of CFB and maintain a distinction between TI on the one hand, and information with a cognitive component, either CI, FVI, or both, on the other. That is, for us to label some operation CFB, either CI, FVI, or both must be presented (see Hoffman, Earle C Slovic, 1981; Nystedt & Magnusson, 1973). The major contrast we will draw is between TI and CFB, but in specific cases we will specify what component(s) of CFB we mean. We will also maintain the usual distinctions between CFB and both outcome feedback (OFB) and feedforward (FF). The procedure of informing the subject in an experiment of the value of Ye immediately after that subject produces Ys defines "OFB". Its effects have been investigated extensively in MCPL studies (Brehmer, 1980) and it has often been employed in one or more conditions during CFB investigations. We have chosen to enclose "OFB" in quotation marks since outcome "feedback" is, if one adheres to the definition of FB, not FB at all. It is in no sense a measure of some output of the person returned to the person generating it. The presentation of Ye after the person produces a Ys is simply a very limited form of task information. It is not TI in the sense that we have defined TI above; that is, as representing relationships in the environment. The reason that we use the term "OFB" at all is that it has acquired the status of a technical term in the social judgment literature. The distinction between FB and FF is temporal (Bjarkman, 1972). FB refers to information about past per-

168

M. E .

Doherty a d W . K . B a l z e r

formance and FF to information about the future. Assuming a stable task environment, what is FB for a preceding block of trials in an experiment is also FF for the succeeding block(s). Since FF refers to a future-oriented procedure, FF concerning TI may be implemented before any response at all from the subjects Investigations described as CFB studies have been implemented in various ways, but what most have labelled as CFB have actually been composites of TI and CFB, which we will call TI+CFB. They typically include cue weights and function forms (often in a graphic format) of both the environment and the subject's consistency (Rs). Other indices (see, e.g., Brehmer & Svensson, 1976; Hammond & Boyle, 1971) have been included, though less often. Values on the subject side of the lens are based on the regression analysis of the subject's responses on a preceding block of trials, while FVI indices are bivariate correlations linking the two sides of the lens. The TI+CFB composite may be given once in a MCPL study or after each of many blocks of trials. CFB has been employed in policy capturing, or what is referred to as the "single-system case" (Hammond, Stewart, Brehmer C Steinmann, 1975), as a cognitive aid to enhance insight into one's own system of values as it applies to a given environment. Of course, only CI can be provided in a single system case; the concepts of TI, FVI, FF, and "OFB" are irrelevant. Note also that CFB, or TI+CFB, has the potential for being a powerful tool for the enhancement of interpersonal learning (Deane, 1979) and for conflict reduction (Balke, Hammond C Meyer, 1973; Hammond & Brehmer, 1973). The impact of CFB on conflict has already produced the deep insight that much of the conventional wisdom about it, expressed in the conceptual system of game theory, is simply wrong. Other usages of the term feedback in p s y c h o l o g y . The CQnimon usage of the word "feedback" in psychology seems to encompass any event that occurs after a response has been made. A pat on the back and a hearty "well done" is "positive feedback". A course grade is "feedback". A meeting in

.

Cognitive feedback

169

which a supervisor rates the performance of an employee is called "feedback". The sound of a pellet falling into the food tray of a Skinner box and the sight and taste of the pellet are said to provide "feedback" as well as reinforcement to the rat or pigeon. But none of these events has the essential, defining character of FB. None returns to the behaving system some measure of its output which it can then compare to an ideal state. What all do is return information about the effect of the output of that system on the environment, but none provides means of monitoring the output that has mediated the environmental effect. This may seem like a quibble, but such indiscriminate usage of the term threatens to deny us a potentially important idea. The distinction between reinforcement effects and CFB effects may be especially important. If you have ever observed subjects in an interactive MCPL task, you may have noted the elation on the part of many subjects when they "get it exactly right". That is a reinforcement effect: putting it idiomatically, for many subjects it seems that "a miss is as good as a mile". To the extent that MCPL subjects are seeking reinforcement, the reinforcing effects of "OFB" will work against the cognitive effects of CFB. In the phrase often used to describe the deleterious effects of "OFB", the subjects will chase error. This is consistent, of course, with Brehmer's (1980) analysis that subjects treat the fundamentally probabilistic MCPL task as a deterministic one. If this conclusion is correct, the loss function inherent in a least squares statistical analysis may be misrepresenting the psychological facts. Recall that Residual Mean Square Error (RMSE) penalizes subjects as the square of the difference between Ye and Ys. If "a miss is as good as a mile" analysis of their subjects' performance in terms of RMSE, and the design of CFB based on that analysis, may be misleading to subjects and experimenter alike. One other point is too important to ignore. Reinforcement is often described as having high "informational" value. It is said (correctly, we believe) to select one ac-

170

M. E. Doherty and W. K. Balzer

tion out of many and to enhance the probability of it being repeated. On the other hand, CFB is a means to knowledge, or pattern matching (Campbell, 1966). Perhaps we may have arrived at a stage of cultural evolution in which reinforcement, which so powerfully narrows the focus of the organism and which is so crucial to survival in a jungle ecology, has become a massive impediment to survival in a policy ecology.

Modes of policy representation Our cognitive apparatus is wonderfully adapted for many interactions with the environment but woefully limited and inadequate for others. There ia consensus that new tools are needed to extend our ability to represent complex environments, much as telescopes and microscopes extended our sensory powers. We do have some such cognitive aids: maps, computers, formal logic, flow charts, and diagrams of many kinds, to name but a few. To the extent that we now live in a policy ecology, we need cognitive aids for representing policy knowledge. Both TI and CI involve constructing policy models for the person to use: TI presents a model of the environment, CI a model of the person. The difference is not so consequential as it might seem at first glance, since the development of a usable model of the environment requires an understanding of what sort of representation the subject is able to use. The principle of parallel concepts (in fact, the whole tradition of the lens model approach) suggests that a usable model of the environment and a usable mental model would be similar, if not identical, in form. Most often, when considering the research on mental models (Gentner C Stevens, 1983), one thinks of a representation of a structure, perhaps of a molecule. Brunswik and Hammond became engaged in mental modelling long before it was fashionable. They became engaged at a very difficult level, and developed a mental Rodel to represent a policy, i.e., an abstract set of relations.

Cognitive feedback

171

How is a person's judgment policy represented? How good is that representation? Whether the representation is veridical requires an omniscience we do not possess: instead we will try to assess whether the representations change behavior in a predictable way. First, let us see what information has been presented. As suggested above, the indices presented in a double-system case include some or all of the following: TI: (a) correlational indices of predictability of the environment (Re or R 2e); (b) the ecological validities (e.g., riel; (a) the function forms relating the criterion to the cues; CI: (a) correlational indices of predictability of the 2 subject (Rs or R s): (b) the utilization coefficients (e.g., rig); (a) the function forms relating the judgment to the cues: FVI: ra, and, in theory at least, G and C. In a single-system case, the subject can receive only CI There are other variations, such as plotting the subject's last block of judgments on the same graph as the true function form (Brehrner 6. Svensson, 1976), including the means of the criterion and/or judgment values (Todd 6. H m mond, 1965), and presenting the central tendencies and ranges of various indices (Galbraith, 1984; Stang, 1985). Presentation format in TI and CFB studies has varied greatly from study to study, and on occasion has been studied as an independent variable. In the first CFB study (Newton, 1965), actual correlation coefficients were used to present TI, CI, and FVI. In one condition, verbal rules were also presented to guide the subjects' judgment behavior. Newton provided subjects with a brief (6 to 8 minute) explanation of the meaning of such coefficients. Clover (1979) also presented actual numerical coefficients, but only after extensive training and the provision of a 116 page manual as an aid to subjects' judgments. Certainly one of the more

.

172

M. E . Doherty and W. K. Balzer

common indices is some transformation of Beta*r (Scbmitt & Levine, 1977). Purely verbal descriptions of weights and function forms have been used (e.g., mane, Hammond & Summers, 1972; Hammond, 1971; Summers, Taliaferro & Fletcher, 1970), as have combinations of verbal and graphic descriptions of function forms (e.g., Brehmer, 1975). The most frequent method of presenting TI and CFB has, of course, been graphic. Graphic representation. Hammond (1971) stressed the importance of providing a "picture" of the task properties and of the subject's cognitive system, and described Cognograph, a software system for the application of computer graphics as an aid to judgment in uncertain environments. An example of the displays produced by Cognograph is shown in Figure 1. Figure 1A shows the relationship between task outcomes and a subject's judgments, 1B bar graphs representing the ecological validities and the subject's utilization coefficients, and 1C the best fit curve representing the function form and variability around that estimate. The representation of the subject's policy can be visually compared to that of the task (Figure 1D). An excellent example of computer graphic policy representation is the conflict reduction study with labor and management negotiators by Balke et al. (1972). Graphic representation using a schematic lens model permitted the participants to compare their own cue weights (utilization coefficients) and function forms directly to those of the counterparts with whom they had negotiated (i.e., a modification of Figure 1D where the task side of the lens model is replaced by the counterpart). CFB has also been returned to the subject by the use of paper and pencil graphs drawn after computer analysis of a block of data. For example, Fero (1975) provided CFB to eight clinical practitioners working on the drug rehabilitation ward of a Veteran's Administration Hospital, who were asked to predict, from either five or ten cues, whether released patients would be able to stay free of drugs and be

Cognitive feedback

173

--

RELATIVE CUE WEIGHTS OUTCOME JUDQMENTS

SUBJECT

::N:

346

I

a

B

A TEST RECORD

5 TRIALS

FUNCTION FORM

10

WiN;

CUE 8

VARIANCE ACCOUNTED FOR

( R'

= ,895 1

LENS MODEL

I

TOTAL ACHIEVEMENT LINEAR ACHIEVEMENT NONLINEAR ACHIEVEMENT DEPENDENCY MATCHING TASK LINEARITY SUBJECT LlNLARlTY

.a68 .954 I20

.

,083

.a99 695

Figure 1 . Examples of displays produced by Cognograph. (A) Graph of task outcomes and a subject's judgments over 10 test trials. (B) Representation of the variance of a subject's judgments in a test trial set that is attributable to cues A, B and C; "unknown" represents variance unaccounted f o r in the subject's cognitive system by the regression model. (C) A graphic representation of the functional relation between a cue and the subject's judgment. (D) A representation of the function allowing weight comparisons between a subject's policy and the task; the thickness of the line between two variables indicates the degree of their covariance. (Adapted from Hammond, K. R. C Boyle, P. J. R. (1971), Bulletin of the B r i t i s h Psychological Society, 24, p. 108, with permission of the authors and the British Psychological Society.)

174

M. E. Doherty and W. K. Balzer

able to hold a job. Excellent criterion data were available. About a week later, practitioners were provided with bar graphs depicting their own weights and the ecological weights: the bar graphs were extensively labeled with cue names at the top, signed numerical values of the weights on the abscissa, and, if the sign of the subject's weight differed from that of the environment, the signs were highlighted in color. Research comparing formats of policy representation. There has been relatively little research comparing formats. The details of the format of the TI and CFB e.g., bar are unlikely to be of much graphs or schematic lens models importance. There is already evidence that this is the case with respect to verbal vs. pictorial representations of weights and function forms (Deane, Hammond & Summers, 1972). Human beings are well adapted to handling either sort of input, but there may be individual differences in what works "best". If the purpose is to develop a cognitive aid, it seems that the best strategy would be to build in as much redundancy as is reasonable, using verbal, numerical, and graphic representations of the ecological and cognitive systems. If the goal is theoretical, then one may wish to determine the necessary and sufficient conditions for a meaningful representation, being careful to maintain an idiographic perspective. If statistical variations of the formal characteristics are considered (i.e., r vs. B vs. 0*r) as format issues, there is more relevant research (Brehmer & Qvarnstrom, 1976; Schmitt h Levine, 19771, but the question of the most appropriate statistical index of the cue weights on which to base feedback is still open.

-

-

Methods for measuring change in policy Until now, when the lens model indices have been mentioned, they have been described as providing CFB in the forms of TI or CI. T h e y are also key measures for the investigator, especially the FVI indices, for assessing whether CFB affects

Cognitive feedback

175

subjects' policy judgments. The investigator of human judgment has the same sorts of cognitive limitations as the subject, and needs tools for thought as much as or more, given his or her purposes, than does the person being studied. The comparisons directly relevant to an assessment of differences in performance have been ra, Rs, and G (or in the earlier papers, Ed). Less often, C has also been of interest. The utilization coefficients may be directly compared with their corresponding ecological validities, although G does this in summary form. The subjects' function forms may also be inspected to determine the degree to which they have the same shape as those of the environment, and to determine if different function forms may be responsible for low values of ra. There are also some indirect means of assessing policy change. For example, researchers have used self-report scales of satisfaction with CFB procedures or confidence in the policy based on CFB (e.g., Deane, 1979). Testimonials from CFB recipients have also been used to support the conclusion that CFB procedures improve judgment (e.g., Balke et al., 1972; Hammond & Adelman, 1976). In an effort to integrate the criteria used to evaluate the efficacy of CFB, we have adopted, with some minor modification, Kirkpatrick's (1976) classification criteria for evaluating training programs. The first set, Reaction Criteria, refers to self-report measures, either informal testimonials or formal scale responses, by CFB recipients. The second set, Behavioral Criteria, refers to lens model indices which can be computed in either single- or double-system cases. The final set, Results Criteria, includes measures of whether individuals exposed to CFB and techniques of social judgment theory seek out and use the methods to solve real problems.

Does CFB work? Behavioral criteria. In general, what has been called CFB has been found to be markedly superior to "OFB". After a

176

M. E . Doherty and

W. K. B a l Z e r

brief review of the evidence for that assertion, we will turn our attention in the light of the distinctions drawn, to the components of CFB that seem to be responsible f o r its superiority. Todd and Hammond (1965) gave CFB by providing subjects with their ra, ecological validities and utilization coefficients (rie and ris values), and means of the criterion and response for each of eight blocks of 25 trials. They found that CFB (TI, CI, and FVI) led to significantly higher achievement than did "OFB". The superiority of relational information, or what has loosely been called CFB, over "OFB" has been confirmed many times (e.g., Adelman, 1981; Gillis, Stewart & Gritz, 1975; Hammond, 1971; Hoffman et al., 1981; Lindell, 1976; Neville, 1977; Nystedt & Magnusson, 1973; Schmitt, Coyle 6 King, 1976; Schmitt, Coyle & Saari, 1977; Steinmann, 1974). Many studies also demonstrate the superiority of CFB to no information, as evidenced by subjects' policy change from trial block 1 to trial block 2 after CFB. Hammond, Summers and Deane (1973) showed dramatic improvement in ra, G, and Rs from block 1 to block 2, as did Hammond and Boyle (1971). The same effect was observed in a non-laboratory situation by Fero (1975), who reported significant increases in ra, G, and Rs when clinical practitioners were provided with TI and CI. The superiority of CFB over no information is also shown, for example, in a study by Schmitt et al. (1976), where G was significantly and substantially higher in block 1 for a group given TI as FF than for a group given no information (see also Summers et al., 1970). Thus, it appears that CFB can influence subjects' behavior. Occasional reports of "OFB" providing better performance than CFB turn out to be only tangentially relevant to the issue at hand: for example, Brehmer and Svensson (1975) reported the superiority of "OFB", but in a single cue learning task with Re = 1.0. The more difficult question is to discover why CFB works. What is it about CFB that influences people to change their policies? The original conception advanced by Todd and

Cognitive feedback

177

Hammond (1965) was that "feedback which directly informs the S about the relation of his cue utilizations .... (to the should make it possible for the S to cue validities) adjust his cue utilizations in the direction of the cue validities" (p. 430). But are both TI and CI necessary? Is FVI useful? These questions cannot be answered fully because the number of studies directly addressing this issue is limited, but there are sound investigations which bear on them. Most studies have used a form of CFB which confounds TI and CI (Adelman, 1981; Clover, 1979; Fero, 1975; Gillis et al., 1975, Experiment 2; Hammond & Boyle, 1971, with respect to the Cognograph data; Hammond & Summers, 1972; Hammond et al., 1973; Hoffman et al., 1981; Holzworth C Doherty, 1976; Lindell, 1976; Neville, 1977, 1978; Steinmann, 1974, 1976; Todd C Hammond, 1965). The use of the term "confounding" is in no way a criticism of the investigators, who were not trying to decompose CFB into its components. Others have presented only TI (e.g., Gillis et al., 1975, Experiment 1; Deane, Hammond & Summers, 1972; Hammond et al., 1973; Summers et al., 1970; Weichelsbaum, 1975). Other studies have compared TI without CI versus TI+CI (e-g., Gillis et al., 1975, Experiment 3: Nystedt & Magnusson, 1973). Only two studies were found which compared TI without CI versus CI without TI (Newton, 1965; Schmitt et al., 1976). The weight of the evidence in the double-system case clearly suggests that TI alone is sufficient to mediate change. This is the essential conclusion of both Newton (1965) and Schmitt et al. (1976), and is supported by those studies which found that TI+CI was not significantly better than TI alone (Gillis et al., 1975, Experiment 3; Nystedt & Magnusson, 1973). The many studies that deliberately confounded TI and CI are also consistent with this conclusion, since the typical finding is that TI+CI feedback, which is consciously built into both Cognograph and POLICY PC (Rohrbaugh, 1986), brings about change in behavior. On the other hand, there is no evidence that CI, in the absence of TI, influences behavior in the double-system case. What evidence there is, in fact, is to the contrary (Newton, 1965;

....

178

M. E. Doherty and W. K. Balzer

Schmitt et al., 1976): CI alone does not help. In one study which found no effect of TI+CI (Clover, 1979), subjects were highly committed to their own original policies. The above review of TI and CI has been with reference to weights and function forms. We have not addressed the possible influence of providing Re and Rs, nor have we considered the role of FVI at all. There is simply not enough research which has manipulated these as factors to make any sensible statements on their effects. We have not reported details of the dependent variables that were influenced by the TI or TI+CFB. Again, the data base is not adequate for a full review of differential effects. However, in general, the major lens model indices (ra, R,, and G) all seem to be influenced positively by TI and TI+CI, with Rs being especially sensitive (Brehmer, 1976; Hammond & Summers, 1972). The reported effects are by no means uniform, but the nonuniformity may be due to differential lack of power among the studies. CFB applications in the single-system case generally capture an individual's policy with respect to some set of values, and then provide information about his or her judgment policy. A review of single-system studies, however, reveals that the effect of CFB on judgment policies has scarcely been investigated with respect to behavioral criteria. Reaction and results criteria. The great bulk of double-system research is laboratory-oriented: hence, results criteria are simply irrelevant. Some double-system studies, especially those of conflict reduction, do report reaction criteria. Balke et al. (1973) and Tucker (1982) each reenacted labor-management disputes. In both, the negotiators were highly laudatory of the process by which they felt they had come to know themselves and one another better. In both studies, however, the subjects indicated they would not recommend the process to future negotiators, expressing concerns for the secrecy of their negotiating positions. There are many extraordinarily interesting applications of the single-system case. The best known of these is the

Cognitive feedback

I79

"Denver bullet study" (Hammond 6 Adelman, 1976). In that study, not only did social policy makers agree that the policies, in terms of weights and function forms, satisfactorily described their value systems, but there were practical results as well. After capturing the policies and applying the weights to cue values supplied by experts on the subject matter, the Denver City Council adopted the action recommended by the policy consultants. The faculty salary study by Roose and Doherty (1978) is a second example of a results-oriented single-system study. A list of faculty members (coded by numbers only) who were identified as underpaid was turned over to the appropriate vice-president, who indicated that he would pursue the issue with the departments involved. (For obvious reasons, the results of his actions were not made public.) Other single-system studies that have used only reaction criteria such as whether the subjects modified weights or function forms provided (e.g., Hammond C Marvin, 1981; Stewart, West, Hammond C Kreith, 1975) or had subjects perform the judgment task a second time (Flack h Summers, 1971).

Relation t o Cognitive Continuum Theory In recent years Hammond has devoted a great deal of effort to developing a theory of cognition which would be of significance not only to investigators in judgment and decision processes but to those in problem solving as well (Hammond, 1980, 1981, 1986). The fundamental concept of this endeavor is that of the cognitive continuum. Cognitive Continuum Theory posits that cognitive tasks can be ordered along a continuum anchored at one pole by fully analytical, rational thinking and at the other pole by intuition. Mixtures of analysis and intuition, called quasirationality, characterize the vast majority of cognitive tasks. Task characteristics influence the sort of cognition that will be employed: Some are much more likely to induce a person to think analytically, while other task characteristics are likely to induce an intuitive mode of cognition.

180

M . E . Doherty and W . K . Balzer

Cryptarithmetic or "cannibals and missionaries" problems, for example, would tend to elicit predominantly analytical, step-by-step cognition, while the task of sorting through a number of employment applications would tend to elicit the intuitive, holistic mode. In a quasirational mode people are prone to move back and forth along the continuum as understanding of the task changes, as new strategies or organizing principles become available to them, etc. What is the relevance of Cognitive continuum Theory t o CFB? Drawing inferences from CFB is itself a cognitive task. Recognizing the difficulties created for the perceiving organism by the uncertain relations between surface and depth, and by the inherently (from the organism's point of view) probabilistic character of the natural environment, Brunswik insisted that a crucial task for psychology was an understanding of the natural environment. But CFB is an artificial environment created to represent the natural ecology, not in its original, entangled form but in a new and more "user-friendly" one. Can we turn to Cognitive Continuum Theory for insight into how to develop truly "perceiverfriendly" CFB or for insight into those model of the environment that have been developed? Table 1 (from Hammond, 1986, p. 8 0 ) is part of a larger table which relates task properties to the induced properties of the cognitive system. Consider the displays that provide the TI+CFB composite in the light of Table 1. The TI component provides a priori task decomposition for judgments yet to come and a posteriori for judgments already made. The CFB component does the same for cognitive decomposition. The type of cue data depends upon the problem, but TI+CFB (as in POLICY PC, for example) handles either equally well. The cue definition, in POLICY FC, Cognograph, and in various paper and pencil renderings of the composite, is both pictorial and quantitative, and perhaps includes a verbal component as well. The mingling of verbal, pictorial, and quantitative representations of the weights allows the person drawing inferences from the display to use both objective and subjective mea-

Cognitive feedback

181

surement. Response time to the TI+CFB displays is usually determined by the subject. Hence, whether by careful design or by some unreportable intuitive insight, the original conceptualizers of T a b l e 1.

Forms of task presentation

Task Decomposition:

A posteriori

a priori

Cognitive Decomposition:

A posteriori

a priori

Type of Cue Data:

Continuous

Dichotomous

Type of Cue Definition:

Pictorially

Quantitative

Subjectively measured

Objectively measured

CFB systems designed displays that were laced with redundancy (cue intersubstitutability, or vicarious mediation) appropriate to any locus on the cognitive continuum. The displays are ideally suited to what Hammond (1981) referred to as "vicarious functioning of organizing principles." FUtWe

research

Dependent Variables. Clearly, more work is needed to clarify what we mean when we say CFB "works." Are all of the dependent measures discussed above appropriate indices by which to measure the efficacy of CFB? Are they validly measuring the impact of CFB on an individual's understanding of the task and of his or her policy, or do they simply tap a global evaluation of CFB procedures? What inference should we draw when two measures of the effectiveness of CFB pro-

182

M. E. Doherty and W. K. Balzer

vide inconsistent evidence (as, for example, when Steinmann (1974) found that CFB led to improvements in Rs but not in ra)? We need to clarify the dimensions along which CFB strategies might be evaluated and to decide, if possible, which measures provide, for the purpose at hand, appropriate indices of the usefulness of CFB. Self-insight. Early in the process of reviewing the Ute-rature for this chapter, we were somewhat troubled by the possibility that it was the TI component of the TI+CFB compound that mediated change. The source of our discomfort was the apparent contradiction between the data and the original conception of why CFB was needed, i.e., to compare the parameters of the individual's cognitive system with those of the task. But this conception assumes that there is a lack of self-insight concerning the parameters of the cognitive system, particularly when statistical cue weights are compared to self-reported cue weights (Hoffman, 1960; Slovic C Lichtenstein, 1971). We have several reservations about this conclusion. First, anecdotal evidence suggests that people have a substantial degree of insight. Recently, for example, one of our colleagues provided CFB (including TI, CI, and FVI) to a number of professional individuals on their decision golicies. They strongly disavowed the CFB provided, confirming our colleague's expectation that the participants would demonstrate poor self-insight. Much to his chagrin, however, he discovered that a clerical error had led him to confuse different participants' policies! Thus, individuals may in fact have at least sufficient self-insight into their decision policy to recognize when they are given incorrect feedback. Second, most comparisons of subjective and statistical weights (or comparisons of their predictions) suggest that insight is imperfect, not totally absent. Third, fundamental measurement issues regarding subjective weights have not been addressed, complicating the interpretation that people lack policy knowledge (Cook C Stewart, 1975; Schmitt C Levine, 1977). All the research on insight into one's own policy, in fact, has asked people to produce their subjec-

Cognitive feedback

183

tive weights on some scale designed by the investigator (e.g., distributing 100 points among the attributes). To our knowledge, none of these scales has been developed with an eye toward its psychometric properties, and construct validity has been assumed rather than assessed. An equally plausible hypothesis is that people know their policy weights reasonably well, and can use them, but simply cannot express them. If so, they would be able to compare, albeit imperfectly, TI with their own cognitive system: thus, presenting CI to individuals would provide little advantage over TI alone. An experimentally verifiable consequence of this hypothesis is that people should be able to recognize their own policies. In a study just completed, Reilly (1987) had 40 accounting students rate 160 hypothetical job offers, each consisting of nineteen attributes. Eleven of the 40 returned for feedback, and each was asked to identify his or her own policy from a 19 x 40 array of UIs computed for all students. Seven did so, and two others vacillated between their own and another before they selected the other: two picked dissimilar policies. Given the null hypothesis that p = . 0 2 5 , the probability of 7 or more hits in 11 tries is .00000000184! This suggests that people may have better self-knowledge than hitherto suspected. Overall, our tentatively supported hypothesis has important implications for judgment analysis. Perhaps TI has thus far been the source of significant change in the TI+CFB research is that individuals do not need externally generated CFB to supplement that self-insight. While the research has pointed to the mismatch between what people do (when forced to make tradeoffs) and what they say, the hypothesis advanced here suggests that it is primarily deficiencies in the expression of one's knowledge rather than deficiencies in self knowledge that are the wellsprings of cognitive conflict. (This result also supports the idea that a regression model is more than the purely mathematical function that some have interpreted Hoffman's (1960) argument about paramorphic representation to mean.)

184

M. E. Doherty and W. K. Balzer

Whether there are conditions under which both task and personal policy information are necessary needs further study. Considering the expense associated with computing and providing detailed CFB to real-world decision makers, studies ought to be designed to separate TI and CI experimentally, and to investigate their separate and joint effects. Although this call for separation of TI and CFB may at first glance appear to be a retreat to systematic design, in fact the question is how best to design an artificial environment, not observe experimentally how a person functions in a natural one. Training. One neglected area is the training of decision makers. While it appears that methods of CFB training has differed among studies, most failed to report how individuals were trained to interpret CFB. Where training is mentioned at all, it ranged from brief description, such as Newton's (1965) 6-8 minute presentation on how to interpret r, to perhaps overly detailed instructions (Clover, 1979). To our knowledge, there has been no empirical investigation of the impact of the training. Given that CFB is often presented in correlational terms (ra, R,, etc.), we need to determine how best to teach decision makers to understand and use CFB. Similar concerns also apply to the understanding of function forms, utilization coefficients, etc. Complexity. While some researchers have examined the impact of such task factors as the number (e.g., Fero, 1975) and complexity of the cues (e.g., cue reliability and validity, linear vs. nonlinear; Brehmer, Hagafors & Johansson, 1980), they have done so at the low ends of the continua involved. Clearly, considerably more research is needed on the complexity of the judgment task: tasks in the real world are often far more complex than those that have been studied. We know little about how to present CFB for complex tasks with varying degrees of cue intercorrelations. We might also experimentally manipulate and study the complexity of the organizing principles underlying the integration of task information (in effect, using CFB at different loci on the cognitive continuum). While simplified cue structures and

Cognitive feedback

185

tasks have contributed to our understanding of what people can do, complex cue relationships and complex integration tasks, representing the complex decision environments in which decision makers work, may prove to be the arena where CI will enhance performance over and above the effect of TI. Two further issues relevant to CFB that are completely unresearched. Given that vicarious mediation is at the very core of Brunswik's conceptualization of the nature of both the person and the environment, it is amazing that no investigators have attempted to develop a means of directly representing cue intercorrelation. Investigators have provided subjects with correlated cues (e.g., Lindell, 1976; Newton, 1965; Nystedt & Magnusson, 1973; Roose & Doherty, 1978; U11man 6 Doherty, 1984), but none has dealt directly with the representation of cue intersubstitutability! The second, and equally surprising gap, given Brunswik's emphasis on the probabilistic nature of the environment, is the failure of investigators to develop means of representing uncertainty for the subjects. None has presented, for example, error bands around estimates of parameters. Point estimates of overall error have been represented (e.g., Re and Re or trans-formations thereof) as well as the scatter about least square function forms. But no investigation has represented the error associated with estimates of ecological validities, utilization coefficients, Rs, ra, etc. It seems to us that veridical means of representing uncertainty might have a significant impact on people, especially given the widespread assumption that a major source of difficulty in learning is the subjects' implicit assumption that probabilistic environments are deterministic.

CFB and other psychological theories CFB may help us understand psychological theories of learning, perception, information processing, and value development and change. For example, if, as we suspect, it turns out that radical differences in TI or CFB format have no effect, and each may change behavior in certain situations, we

186

M. E .

Doherty and W. K. Balzer

will have learned something about flexibility ( o r vicarious functioning) in knowledge representation. We have already learned that in complex environments learning is facilitated by the TI+CFB composite but inhibited by "OFB". This statement reflects something profound about the nature of cognition; as Brunswik insisted, we learn relations, not arrays of observations. The impact of TI+CFB or CFB on conflict reduction, as Rohrbaugh (see Chapter 6) explores, deflates many of the motivational hypotheses, and undercuts the belief that conflict must be understood from the vantage of game-theory. Theories of learning, perception, information processing, and value formation may further CFB theory and application. Given the relatively brief, and often only single, presentation of CFB information during a typical study, limitations on a person's ability to encode and remember information would serve to undermine the influence of CFB. In addition, the CFB literature may benefit from research that suggests that the memorability and salience of information can be enhanced by increasing its vividness, e.g., by making it more concrete, more emotional, or more personally relevant (Fiske & Taylor, 1984; Nisbett & Ross, 1980). For example, it is widely accepted that people have difficulty learning nonlinear relation's. Yet, as Coombs (1983) argued, we carry around with us all sorts of single-peaked (i.e., inverted U shaped) utility functions. Perhaps it would facilitate the learning of such relations if we were to tell subjects of the analogy between the relation of Cue A to the criterion, and that of "the number of spoons of sugar in your coffee and how much you like it. The use of causal schemata to organize events by causeeffect relations may also facilitate the acceptance of CFB. Fero (1975) investigated the cues eight clinical practitioners and drug counsellors used to predict future drug use by 160 individuals from a V.A. drug dependency unit. Ecological analyses showed that marital status was a significantly negatively weighted predictor of recidivism (i.e., married individuals were more likely to return to drug use), contrary

Cognitive feedback

187

to Fero's and his subjects' "schema" that married individuals should have a more stable home life, thereby minimizing their contact with the drug culture. The resistance to this schema-inconsistent piece of CFB disappeared, however, when one participant suggested that being married to a member of the drug culture might increase the client's contact with drug users. Given this causal scenario, judges were able to accept and integrate this TI+CFB into their policies. One of the most active areas in cognitive psychology, and in cognitive social psychology, is schema theory (Rumelhart, 1984). Consider the relation between TI+CFB and a schema: as suggested above, a p o l i c y is a schema. A policy allows one to systematically organize what might otherwise be chaotic incoming information. Commerce with the environment without a policy, or schema, is unthinkable. Schema theorists have not investigated several issues that are highly relevant to social judgment research. For example, does a person's ability to apply a schema reliably diminish (Hammond & Summers, 1972) when the environment forces a modification of that schema? Can schema conflict be investigated via those strategies used in cognitive conflict studies? And what can CFB researchers learn from the rich and burgeoning literature in schema theory? The literature on commitment (Janis & Mann, 1977; Staw, 1976) may suggest some boundary conditions for the efficacy of CFB. For example, a decision maker who is committed to his or her policy may be resistant to changing it for many reasons, including a mistaken perception that he or she is an infallible decision maker, lowered self-esteem or loss of face from admitting to decision errors, or an intractable commitment to an earlier judgment policy. Cloverls (1979) interviewer subjects had conducted as many as 35,000 interviews prior to the intervention; in his study, neither TI nor CFB had an effect on interviewer policy. Similarly, the literature on confirmation bias suggests that individuals may nonconsciously distort or ignore CFB. Selective attention to those aspects of CFB which support the present state of affairs, and inattention to or distor-

188

M. E . Doherty and W. R. B a l Z e r

tion of those aspects which challenge the current decision policy, may serve to confirm rather than disconfirm a judge's conception of his or her policy, or of environmental matching. Current CFB studies have not examined whether decision makers fully digest the CFB they receive (i.e., there are no formal manipulation checks). Investigations to determine whether all components of CFB are assimilated by the decision maker, or only those related to personal biases or implicit theories, may clarify the relationship between CFB and prior biases.

Applications of CFB Given the increasing availability and affordability of personal computers and CFB software such as POLICY PC, we expect that the use of CFB will grow in the future. CFB may be useful for many major personal decisions, including choosing an automobile, house, or career. Feedback can be used to identify what an individual personally values (i.e., single system CFB) or to understand areas of agreement and disagreement with spouse, family members, neighbors, etc. (i.e., double system CFB: see Chapter 6). Of course, individuals will need additional training to identify important decision components, interpret CFB, and discuss policies, but communication and interpersonal understanding could be greatly enhanced. The use of CFB technology could also enhance business decisions, leading to a better understanding of decision strategies used by managers, improving selection and promotion decisions, and highlighting areas of disagreement between supervisors and subordinates. CFB has been applied in academic counselling (Johnson & Doherty, 19831, and may also be applied in particular occupations such as marriage an6 career counselling, financial/consumer counselling, or law. It may also be a useful tool for education and training programs, making professionals aware of their policies for employee selection and appraisal, medical or clinical diagnosis (see Chapter 7 ) , security trading decisions, and so on.

Cognitive feedback

189

The provision of CFB, however, does not guarantee that people will use and benefit from it. Even though CFB may improve a decision maker's understanding of his or her policy and the decision environment, research has yet to identify those situations where the decision makers unconsciously, or even consciously, distort or ignore CFB. They may sometimes benefit from keeping their policies covert. Although Balke et al. (1973) reported that management and union negotiators thought CFB would be valuable early in the negotiation process, negotiators in a similar study (Tucker, 1982) said that while they were impressed with insights provided by CFB, they would not recommend the procedure for future use. In their eyes, they saw a weakened negotiating position if the other party knew precisely what they were seeking for their constituencies; without the unimportant, superfluous issues to concede in the give-and-take bargaining process, they would have nothing to trade. In this situation, there appears to be some (mistaken?) advantage in keeping secret information about what you are really interested in obtaining. Would any amount of CFB change such a situation? Similarly, Flack and Summers (1971) reported higher levels of disagreement between engineers evaluating water resource planning projects after CFB, perhaps because they may have deemed it politically wiser to adopt more extreme policy positions after CFB, in order to allow more room for concessions. There may in fact be a number of domains where CFB will not work, not because of the limitations of CFB but because of personal or organizational concerns. We believe that the future growth of CFB applications will be in the measurement of utilities rather than in modeling environments (see Hammond, Rohrbaugh, Mumpower & Adelman, 1977). We see more people benefiting from CFB, specifically the CI component, by an increased understanding of, and ability to communicate, what they personally value (e.g., what they value in a house: what they look for in expanding a stock portfolio, etc.) rather than from an improved prediction of an uncertain environment (e.g., what housing characteristics lead to housing costs, predicting

190

M. E. Doherty and W. K. Balzer

stock performance, etc.). If the promise of wide availability of CFB software packages for general purposes is fulfilled, there may be another benefit to users. Since the user will have to decide upon the dimensions (or cues) and their levels, the user will not only have the benefit of having to make trade-offs, receiving CFB, etc., but will also have the benefit of the insights gained from the decomposition of the problem common to the early stages of decision analysis (see Gardiner & Edwards, 1975). The ecological aspect of CFB may be severely limited by difficulties in obtaining outcomes in important environments or by their unstable nature which may make it difficult to provide valid TI. Still more importantly, many day-to-day decisions often have no optimizing referent (the so-called "gold standard") with which we can compare our personal decisions: instead, we may only wish to learn more about what we value and to learn how to communicate those values.

Acknowledgement The authors would like to thank Raymond O'Connor Jr. for his help with the literature review and for his helpful comments on earlier versions of this paper.

References Adelman, L. (1981). The influence of formal, substantive and contextual task properties on the relative effectiveness of different forms of feedback in multiple-cue probability learning tasks. Organizational Behavior and Human Performance, 27, 423-442. Balke, W. M., Hammond, K. R. & Meyer, G. D. (1973). An alternate approach to labor-management relations. Administrative Science Quarterly, 18, 311-327. BjOrkman, M. (1972). Feedforward and feedback as determiners of knowledge and policy: Notes on a neglected issue. Scandinavian Journal of Psychology, 13, 152-158.

M. E. Doherty and W. K. Balzer

191

Coombs, C. H. (1983). Psychology and mathematics: An essay on theory. Ann Arbor, MI: The University of Michigan Press. Deane, D. H. (1979). Effects of judgment feedback and discussion in decision-making groups: A comparison of social judgment analysis and the nominal group technique. Dissertation Abstracts International, 39B, 4100-4101. Deane, D. H., Hammond, K. R. & Summers, D. A. (1972). Acquisition and application of knowledge in complex inference tasks. Journal of Experimental Psychology, 92, 2026. Edwards, W. (1978). Technology for director dubious: Evaluation and decision in public contexts. In K. R. Hammond (Ed.), Judgment and decision in public policy formation. Boulder, CO: Westview Press. Fero, D. D. (1975). A lens model analysis of the effects of

amount of information and mechanical decision making aid on clinical judgment and confidence. Unpublished doctoral dissertation, Bowling Green State University, Bowling Green, OH. Fiske, S. T. & Taylor, S. E. (1984). Social cognition. Reading, MA: Addison-Wesley Publishing Company. Flack, J. E. & Summers, D. A. (1971). Computer aided conflict resolution in water resource planning: An illustration. Water Resources Research, 7, 1410-1414. Galbraith, J. T. (1984). Training assessment center assessors: Applying principles of human judgment. Unpublished doctoral dissertation, Bowling Green State Unlversity, Bowling Green, OH. Gardiner, P. J. & Edwards, W. (1975). Public values: Multiattribute-utility measurement for social decision making. In M. F. Kaplan & S. Schwartz (Eds.), Human judgment and decision processes. New York: Academic Press. Gardner, H. (1985). The mind's new science. New York: Basic Books. Gentner, D. h Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Lawrence Erlbaum Associates.

192

Cognitive feedback

Brehmer, B. (1975). Learning to use functional rules in inference tasks. Umeb Psychological Reports No. 86. University of Wmeb, Sweden. Brehmer, B. (1976). Note on clinical judgment and the formal characteristics of clinical tasks. Psychological Bulletin, 83, 778-782. Brehmer, B. (1980). In one word: Not from experience. Acta Psychologica, 45, 223-241. Brehmer, B., Hagafors, R. & Johansson, R. (1980). Cognitive skills in judgment: Subjects' ability to use infor-mation about weights, function forms, and organizing principles. Organizational Behavior and Human Performance, 26, 373-385. Brehmer, B. & Qvarnstrbn, G. Information integration and subjective weights in multiple-cue judgments. Organizational Behavior and Human Performance, 17, 118-126. Brehmer, B. & Svsnsson, C. (1976). Learning to use function rules in inference tasks. Scandinavian Journal of Psychology, 17, 313-319. Brunswik, E. (1952). The conceptual framework of psychology. Chicago: University of Chicago Press. Brunswik, E. (1956). Perception and the representative design of psychological experiments. Berkeley, CA: University of California Press. Campbell, D. T. (1966). Pattern matching as an essential in distal knowing. In K. R. Hammond (Ed.), The psychology of Egon Brunswik. New York: Holt, Rinehart, & Winston, Inc. Clover, W. (1979). Cognitive feedback in the selection interview: Applying social judgment theory in the field. Unpublished doctoral dissertation, Bowling Green State University, Bowling Green, OH. Cook, R., & Stewart, T. R. (1975). A comparison of seven methods for obtaining subjective descriptions of judgment policy. Organizational Behavior and Human Performance, 13, 31-45.

Cognitive feedback

193

Gillis, J. S., Stewart, T. R. & Gritz, E. R. (1975). New procedures: Use of interactive computer graphics terminals with psychiatric patients. In K. R. Hammond & C. R. B. Joyce (Eds.), Psychoactive drugs and social judgment: Theory and research. New Y o r k : John Wiley & Sons. Hammond, K. R. (1971). Computer graphics as an aid to learning. Science, 172, 903-908. Hammond, K. R. (1975). Externalizing the parameters of quasi-rational thought (Center for Research on Judgment and Policy Report No. 186). Boulder: University of Colorado. Hammond, K. R. (1980). The integration of research in judgment and decision theory (Center for Research on Judgment and Policy Report No. 226). Boulder: Univer-sity of Colorado. Hammond, K. (1981). Principles of organization in intuitive and analytical cognition (Center for Research on Judgment and Policy Report No. 231). Boulder: University of Colorado. Hammond, K. R. (1986). A theoretically based review of the-

ory and research in judgment and decision making (Center for Research on Judgment and Policy Report No. 260). Boulder: University of Colorado. Hammond, K. R. & Adelman, L. (1976). Science, values and human judgment. Science, 194, 389-396. Hammond, K. R. & Boyle, P. J. R. (1971). Quasi-rationality, quarrels, and new conceptions of feedback. Bulletin of the British Psychological Society, 24, 103-113. Hammond, K. R. & Brehmer, B. (1973). Quasi-rationality and distrust: Implications for international conflict. In L. Rappoport & D. Summers (Eds.), Human judgment and social interaction. New Y o r k : Holt, Rinehart & Winston. Hammond, K. R. & Marvin, B. A. (1981). Report to the Rocky

Flats monitoring committee concerning scientists’ judgments of cancer risks. Boulder: University of Colorado, Center for Research on Judgment and Policy. Hammond, K. R., McClelland, G. H. & Mumpower, J. (1980). Human judgment and decision making. New Y o r k : Praeger.

194

M.

E. Doherty and W. K. Balzer

Hammond, K. R., Rohrbaugh, J. R., Mumpower, J. & Adelman, L. (1977). Social judgment theory: Applications in policy formation. In M. Kaplan & S. Schwartz (Eds.), Human judgment and decision processes in applied settings. New York: Academic Press. Hammond, K. R., Stewart, T. R., Brehmer, B. & Steinmann, D. 0. (1975). Social judgment theory. In M. Kaplan C S. Schwartz (Eds.), Human judgment and decision processes. New York: Academic Press. Hammond, K. R. & Summers, D. A. (1972). Cognitive control. Psychological Review, 79, 58-67. Hammond, K. R., Summers, D. A. & Deane, D. H. (1973). Negative effects of outcome-feedback in multiple-cue probability learning. Organizational Behavior and H u m a n Performance, 9, 30-34. Hoffman, P. J. (1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57, 116-131. Hoffman, P. J., Earle, T. C. & Slovic, P. (1981). Multi-dimensional functional learning (MFL) and some new conceptions of feedback. Organizational Behavior and H u m a n Performance, 27, 75-102. Holzworth, R. J. & Doherty, M. E. (1976). Feedback effects in a metric multiple-cue probability learning task. Bulletin of the Psychonomic Society, 8, 1-3. James, W. (1890). Principles of psychology. New York: Henry Holt Janis, I. L. & Mann, L. (1977). Decision making: A psychological analysis. New York: The Free Press. Johnson, W. R. & Doherty, M. E. (1983). Social judgment theory and academic advisement. JOWnal of Counselling PSyChOlOgy, 30, 271-274. Judson, H. F. (1980). The search for solutions. New York: Holt, Rinehart 6 Winston. Kahneman, D., Slovic, P. 6 Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. London: Cambridge University Press.

.

Cognitive feedback

195

Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development. New York: McGraw-Hill. Lindell, M. K. (1976). Cognitive and outcome feedback in multiple-cue probability learning tasks. Journal of Ex-

perimental Psychology: H u m a n Learning and Memory, 2 , 739-745. Neville, K. G. (1977). Self-instruction and cognitive feedback with a complex cognitive task. Bowling Green, OH: Bowling Green State University, Department of Psychology, Unpublished manuscript. Neville, K. G. (1978). Self-instruction and cognitive feedback: A study of training and generalization with complex cognitive tasks. Unpublished doctoral dissertation, Bowling Green State University, Bowling Green, OH. Newton, J. R. (1965). Judgment and feedback in a quasi-clinical situation. Journal of Personality and Social Psychology, 1, 336-342. Nisbett, R. & Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice-Hall, Inc. Nystedt, L. & Magnusson, D. (1973). Cue relevance and feedback in clinical prediction task. Organizational Behavior and Human Performance, 9, 100-109. Postman, L. & Tolman, E. C. (1959). Brunswik's probabilistic functionalism. In S. Koch (Ed.), Psychology: A study of science (Vol. 1. Sensory, perceptual, and physiological formulations). New York: McGraw-Hill. Reilly, B. A. (1987). The importance of job attributes in the recruitment of accountants: A policy capturing approach. Unpublished masters thesis, Bowling Green State University, Bowling Green, OH. Rohrbaugh, J. R. (1986). Policy PC: Software for judgment analysis. Albany, NY: Executive Decision Services.

196

M. E .

Doherty and W. K. BalZer

Roose, J. E. & Doherty, M. E. (1978). A social judgment theoretic approach to sex discrimination in faculty salaries. Organizational Behavior and Human Performance, 22, 193-215. Rumelhart, D. E. (1984). Schemata and the cognitive system. In R. S. Wyer, Jr. & T. K. Srull (Eds.), Handbook of social cognition (Volume 1). Hillsdale, NJ: Lawrence Erlbaum Associates. Schmitt, N., Coyle, B. W. & King, L. (1976). Feedback and task predictability as determinants of performance in multiple-cue probability learning tasks. Organizational Behavior and Human Performance, 16, 388-402. Schmitt, N., Coyle, B. W. & Saari, B. B. (1977). Types of task information feedback in multiple-cue probability learning. Organizational Behavior and Human Performance, 1 8 , 316-328. Schmitt, N. & Levine, R. (1977). Statistical and subjective weights: Some problems and proposals. Organizational Behavior and Human Performance, 20, 15-30. Slovic, P. & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational Behavior and Human Performance, 6, 649-744. Stang, S. w. (1985). An interactive judgment analysis of job worth. Unpublished doctoral dissertation, Bowling Green State University, Bowling Green, OH. Staw, 8. (1976). Knee deep in the big muddy: A study of escalating commitment to a chosen course of action. Organizational Behavior and Human Performance, 23, 27-44. Steinmann, D. 0. (1974). Transfer of lens model training. Organizational Behavior and Human Performance, 12, 116. Steinmann, D. 0. (1976). The effects of cognitive feedback and task complexity in multiple-cue probability learning. Organizational Behavior and Human Performance, 1 5 , 168-179.

Cognitive feedback

197

Stewart, T. R., West, R. E., Hammond, K. R. & Kreith, F. (1975). Improving human judgment in technology assessment. Journal of the International Society for Technology Assessment, June, 37-43. Summers, D. A., Taliaferro, J. D. & Fletcher, D. J. (1970). Judgment policy and interpersonal learning. Behavioral Science, 15, 514-521. Tucker, S. (1982). A method to enhance mutual policy formation in negotiators. Unpublished masters thesis, Bowling Green State University, Bowling Green, OH. Todd, F. J. 6 Hammond, K. R. (1965). Differential effects in two multiple-cue probability learning tasks. Behavioral Science, 10, 429-435. Ullman, D. G. & Doherty, M. E. (1984). Two determinants of the diagnosis of hyperactivity: The child and the clinician. In M. Wolraich & D. K. Routh (Eds.), Advances in behavioral pedriatics. Greenwich, CT: JAI Press. von Winterfeldt, D. & Edwards, E. (1986). Decision analysis and behavioral research. New York: Cambridge University Press. Weichelsbaum, H. F. (1975). New concepts: Effects of methadone maintenance on cognitive control. In K. R. Hammond & C. R. B. Joyce (Eds.), Psychoactive drugs and social judgment. New York: John Wiley 6 Sons.