Multiple-stage probabilistic information processing

ORGANIZATIONALBEHAVIOR AND I~UI~A~ PERFORIVIANCE 10, 374-387 (1973)

Multiple-Stage Probabilistic Information Processing 1'2 CHARLES GETTYS, CH~LES !V[ICHEL, A~CDJAMES m. STEIG~,R University o] Oklahoma AND CLINTON W . KELLY AND CAMERON t~. PETERSON

Decisions and Designs, Inc. Human performance on hierarchical inferences may be suboptimum. As a means of improving this performance, it is suggested that a hierarchical inference task be decomposed into a series of simpler intuitive inferences linking each stage in the hierarchy and that assessments of these inferences be formally combined using the optimum hierarchical model. The results of laboratory experiments with this procedure show that it can improve performance over a completely intuitive approach. A case study in which the procedure was applied to a real world inference problem is also described and some practical difficulties are identified. Decision analysis requires probabilities as p r i m a r y inputs, and in a d y n a m i c world the probabilities m u s t be revised in the light of new information. However, because most events of interest are unique, the process of assessment usually requires h u m a n judgment, r a t h e r t h a n some objective measure such as relative frequencies. I n addition to assessing probabilities, people often rely on their own j u d g m e n t in the revision of their probability assessments. Substantial psychological research has shown, however, t h a t people tend to be sub o p t i m u m when t h e y revise probabilities; t h a t is, t h e y exhibit biases. M u c h of the research suggests t h a t the s u b o p t i m a l i t y occurs in the process of aggregating the various pieces of information, r a t h e r t h a n in the process of assessing the i m p a c t of each individual datum. Accordingly, a class of 1 This paper was supported by the National Aeronautics and Space Administration Grant No. NGL-23-005-171 and the Advanced Research Projects Agency and the Office of Naval Research Grant Nos. NONR-N-0014-73C-0149 and NR-197-023. 2 Experiment 1 was conducted at the Engineering Psychology Laboratory, University of Michigan by C. Gettys, C. W. Kelly, and C. Peterson. Experiment 2 was conducted at the University of Oklahoma by C. Michel, C. Gettys, and J. Steiger. The case study was the result of. consulting work done by C. W. Kelly and C. Peterson, who also wrote the bulk of this article. 374 Copyright O 1973 by Academic Press, Inc. All rights of reproduction in any form reserved.

PROBABILISTIC INFORMATION

PROCESSING

375

procedures called "probabilistic information processing" systems has been developed to counter such biases. These systems work by decomposing the diagnostic task in such a manner that an expert assesses the impact of each datum, and the assessments are combined for all the data using the optimum rule, Bayes' theorem. Even though people seem to be able to accurately assess t.he diagnostic impact of a single datum in a single-stage inference, there is substantial evidence that people exhibit biases in inferences involving multiple stages (inferences in which the data are only indirectly linked to the hypotheses of interest). When, for example, data provide information about which of the intermediate level events is true, and that event in turn provides information about which of the hypotheses is true, there seems to be a tendency to ignore all but the most dominant, or salient, chain through the hierarchy linking the data to the hypotheses. These biases suggest, the need for a system or procedure which will permit the decomposition of the inference task into its components so that an expert may assess the probabilistic connection between one stage or level and the next and then these assessments may be combined in an optimum manner. The theoretical work for this kind of decomposition has already been done (Dodson, 1961; Gettys & Willke, 1969; Kelly, 1972; Kelly & Barclay, 1973; Schum & Kelly, 1973). For example, the following equation provides a means of calculating the posterior probability that any particular hypothesis H is true after observing a datum D: P(H[D) -

P(H) P(D)

P(DiEi)P(EilH)' i

where E i denotes a particular event in the intermediate level variable linking the datum to the hypothesis. Note that this equation requires assessments of P(DiEi ) as measures of the link between the datum and the states of the intermediate level variable and assessments of P(Ej IH) as measures of the link between the states of the intermediate level variable and the hypothesis. It is not necessary to assess the llkelihood of E~, however, since the amount of revision in the probability of the hypothesis depends only on the degree to which the intermediate level variable is associated with both the hypothesis and the datum. This equation, sometimes referred to as the "modified Bayes' theorem," suggests the manner in which the multiple-stage or hierarchical inference may be decomposed into a series of single-stage inferences; each single-stage inference is represented by assessed conditional probabilities relating adjacent stages in the hierarchy. The experiments reported be-

37,6

GL~rTYS et al.

low were designed to test the validity of this procedure as a means of improving unaided human inference. EXPERIMENT

1

Method Experimental design. The subjects were asked to revise odds estimates in a two-stage hierarchical inference; that is, there was one intermediate level variable intervening between the data and the hypotheses. Given data about a sample of college students, the subjects were asked to infer which of two departments the students were majoring in. Specifically, based upon the height of a student and his score on a test which had been designed to discriminate between males and females, the subject in the experiment could infer probabilities about the sex of the student, and from these probabilities he could infer the likelihood that the student was majoring in either speech or zoology. Probability distributions related the adjacent variables in the hierarchy. For example, the zoology department majors consisted of 75,% men and 25,% women students; whereas, the speech department majors were 25% men and 75% women (these percentages were close to the actual percentages at the University of Michigan, where the experiment was conducted). For the intermediate level variable, normal probability distributions described the distributions of heights and M - F scores~ for each sex. These distributions were constructed so as to be similar to census data: the population of men students contained more taller students, and also more students with a higher M - F score, than was the case for the population of women students. Apparatus and procedure. The experimental apparatus consisted of two displays of the conditional probabilities relating the adjacent levels in the hierarchy. The first display contained four histograms: two of these showed the distributions of heights of men and women in the range of 58-74~ in.; the other two showed the distribution of M - F scores for men and women in the range of 20-36 yr. Each of the histograms showed a sample of 100 college students. The second display showed the relative percentage of men and women in the zoology and speech departments. The subjects were given instruction in the meaning of the histograms, and also about the meaning of their quantitative responses. They were then given training by observing samples of heights and M - F scores for individuals selected at random, by inferring whether the individual 3A masculinity-femininity (M-F) scale was contrived by the experimenters. Subjects were told tha~ values along this scale were based upon subsets of items taken from a well known personMity inventory.

PROBABILISTIC INFORMATION PROCESSING

377

was more likely to be a man or a woman, and by considering the odds in favor of the more likely hypothesis. In order to instruct them about random processes, the subjects were asked to imagine that each of the two hypotheses, zoology and speech majors, were represented by a bag full of cards. Then they were to imagine that a fair coin had been tossed to select one of the two imaginary bags, and a card containing the height and M - F score of one individual was drawn at random from that bag. The subjects would be first told the M - F score on the card and asked to indicate on the response form which bag was being sampled from and how sure they were in odds. After they had made the first assessment, they would be told the height on the card and asked to revise their odds on the basis of the additional information. A second variation of this procedure was also used. The subjects were asked to imagine the same procedure as described above, but rather than reading the height from the first card, the experimenter would drop the card back in the bag, draw a second card randomly, and read the height from it. During the actual running of the experiment the subjects were told that to save time the actual drawing had been done prior to the experiment and the observations recorded. This procedure permitted each subjeer to receive the same sequence of observations. There were 27 trials, the two variations of the procedure occurring in random order. In addition to the odds assessment for the multiple-stage inference, once every six or seven trials the subieets were asked for the odds relevant to each single-stage inference in the hierarchy: the odds for sex given M - F score alone, given height alone, or given height, and M - F score together, and the odds for the hypotheses given sex. Subjects. The subjects were 25 male undergraduates at the University of Michigan. They were paid 81.75 an hour, and were run in groups of three to five each.

Results Figure 1 shews the degree of correspondence between the posterior odds as assessed and aggregated by the subjects, and the posterior odds calculated from the probabilities given in the displays and aggregated using modified Bayes' theorem, the optimum model. Each data point represents the :median across subjects for the assessments based on the observation of both pieces of data, height and M - F score. The circles represent those trials in which each type of data was associated with a different individual, and the triangles represent the trials in which each type of data came from the same individual.

378

et al.

GETTYS

0.6

-t~AJOR IHT, MF, DIFFERENT INDPVIDUALS <

0.5

Z~ -MAJORI H]~, MF, SAME INDIVIDUALS

0.4

(~

Z50

0.3 0.2

o

°3:% ~

i

-06-os-0.4

I

-

i

i

~0.3-0.2-o.i ~

0

i

o

o.J

0,2

i

0.3

,

014

0[5

0[6

-0.I

-0.2

g, ~

~

o/ /o

--

-0.S

-- 0o. 5s -0,6 ~

LOG MBT ODDS FOR MAJOR

Fro. 1. Degree of correspondence between posterior odds of subjects and posterior odds obtained using modified Bayes' theorem. Performance appears to be essentially the same regardless of whether the lower level data are samples of the same individual or samples of different individuals. Both circles and triangles tend to be slightly excessive with respect to the identity line, except, where the data is very undiagnostic (log odds are less than .2). In Fig. 2 aided inference is compared to the optimum model. Aided inference means that the single-stage inferences have been combined using the optimum model, modified Bayes' theorem. Here the slight excessive bias has been virtually eliminated, suggesting that those biases which do exist, are probably a result of the subject's aggregation process, rather than his assessment of impact of one stage on another. At any rate, the use of modified Bayes' theorem to aid the subject represents only a slight improvement since the unaided multiple-stage inferences were not markedly suboptimum. This result suggests that. an evaluation of the usefulness of the decomposition procedure should be made in a situation that. leads to poorer intuitive performance. EXPERIMENT

2

Because the relativeiy good performance in the unaided condition of Experiment. 1 left relatively little room for improvement, a second experiment was conducted in which there was, a priori evidence that the subjects would perform poorly in making unaided judgments. The

379

PROBABILISTIC INFORMATION PROCESSING 0605-TYPE I SAMSIS USPNG P(SEXIMF}' PISEXiHT)

04-

0 -TYPE 2 SAMSIS USING P{SEX[MP,HT} e - T Y P E SAMSIS FOR "DIFFERENT" INDIVIDUALS

3

03 020.1-

-& 45 -;4 43 -o2 -o,ff o e / ~ 2. o • -01

/

/

/

/

{

o, 12 0'3 o14 o'5 o].6

/ ii

-0.4

LOG MaT POSTERIOR ODDS FOR MAJOR

FIG. 2. A comparison of aided inference to the o p t i m u m model.

experimental paradigm chosen was adopted from Steiger and Gettys (1972). Su.bjects in this task tended to single out and react to only the intermediate event which was most favored by the datum that had been observed; they essentially ignored implications of other possible intermediate events (see Gettys, Kelly, & Peterson, 1973).

Method Experimental desig7~. For each problem, the subjects were shown two bags, each of which contained different proportions of red, blue, and orange envelopes. Each envelope in turn contained pieces of black construction paper cut into circles, squares, and triangles, the proportions of the various shapes depended on the color of the envelope. The subjeers' task was to infer, on the basis of the shape which had been drawn, the probabilities that the shape had come from each of the envelopes, and from these probabilities, infer which was the more likely bag to. have been chosen. Apparatus and procedure. The apparatus consisted of two displays. The first showed two rectangles, each of which was divided into three colored sections, representing the proportions of the three colors of envelopes in the bags (see Fig. 3). The second display was similar; it contained three circles, one for each color of envelope, divided into segments according to the proportion of circles, triangles, and squares contained in the envelopes.

380

GETTYS et al. Bag

Bag 2

E o

i ,&.

Red envelopes

Blue envelopes

Orange envelopes

FIG. 3. Rectangular and circle displays.

The subjects were told that the experimenter had tossed a fair coin to choose one of the two bags, then had reached into the bag and randomly drawn an envelope without looking at its color. Then the experimenter had drawn a shape out of the envelope, for example, a triangle. Knowing that a triangle had been drawn, knowing the proportion of triangles in each of the three kinds of envelopes, and knowing the proportion of each color of envelope in the two bags, the subjects were to infer which was the more likely bag, and assign odds to indicate how sure they were. Consider the proportions given in Fig. 3. Assuming that the experimenter has drawn a triangle, it is most likely that the triangle was drawn from a red envelope. However, there is also a reasonably high chance that it came from a blue envelope, and a very small chance that it came from the orange envelope. If the triangle had been drawn from the red envelope, then it is reasonably likely that it came from bag 1. But, on the other hand, if it came from a blue envelope, then it is extremely likely that it came from bag 2. In this case, even though the triangle is less likely to have been drawn from the blue envelope than the red, this is outweighed by the fact that blue envelopes are much stronger evidence for bag 2 than red envelopes are for bag 1. In fact, the probability is .562' that the second bag was the one chosen by the experimenter. Problems like this, in which the more likely bag is not the bag favored by the most likely envelope, are called disagree problems. Problems in which the more likely bag is also favored by the most likely envelope are called agree problems.

PROBABILISTIC

INFORMATION

381

PROCESSING

Three sets of rectangular displays and two set, of circle displays were used together with different shapes as the beginning point for inference. In this way eight problems were presented to the subjects; five "agree" problems and three "disagree" problems. Subjects. Thirty male undergraduates were volunteers from a University of Oklahoma introductory psychology course. Participation in the experiment earned them points which could be used to raise their course grade. They were run in groups of six to eight.

Results Figure 4 displays the relationship between the optimum posterior odds and the median posterior odds produced by the subjects. The squares represent unaided assessments of the posterior odds and the circles repre0.8

I

I

I

l

l

l

l - - I

I

[3

]

0

0.7 0.6

{3 0,5

[3

0,4 0 0.3 0.2 0.1

0

o d

-OJ o

-0.2

O-SAMSIS

~ O'

t

-0.3

Q-S's UNAIDED JUDGEMENTS

-0.4

d-DISAGREE PROBLEMS

-0.5 -0.6 -0.7 -0.8

0 I

-C,5 -0.4

T

I

l

I

T

I

-Q3 -0.2 -0.1 0.1 0.2 03 LOG GOD's ODDS {IN FAVOR OF BBI}

t

l

I

0.4

0.5

0,6

OY

FIe. 4. Relationship between optimum posterior odds and the median posterior odds produced by subjects.

382

aETTVS et al.

sent odds which were produced by combining the subjects' stage-by-stage assessments using the optimum procedure. First consider the unaided responses. In all eases the "disagree" problems led to posterior odds which favored the incorrect bag, as would be expected from the results obtained by Steiger and Gettys (1972). For the "agree" problems the unaided assessments are in the correct direction but are consistently excessive when compared to the optimum odds. On the other hand, all of the aided judgments are fairly close to the optimum. That is, when combined by the optimum procedure, most of the suboptimality in the subjects' assessments seems to disappear. (The mean squared discrepancy between the subjects' responses and the optimum responses was .115 for the unaided judgments, and only .009 for the aided judgments, a ratio of about 12 to 1 in favor of the aided posteriors.) Furthermore, posterior odds for the aided judgments were invariably in the correct direction; whereas, in the "disagree" problems, the unaided posterior odds favored the incorrect hypothesis. A CASE. STUDY In addition to the experimental investigation of the applicability of the multiple-stage hierarchical procedure, two of the authors have tested the procedure in several ease studies, with generally positive results. One such ease study concerned the possibility that a certain country was developing a particular weapons system (this research is described in more detail in Kelly, 1972). Figure 5 provides a schematic illustration

Theweaponssystemis: HH2= 1=Bei as ghpdorily priorityproj projeect, ct, Beinnggdevel develooped ped asoohilow Hs:Notbeingdeveloped I

Hypothesis

....1,, [:;:!!!
~ ~ r 2 : r : 2 m ~

FIG. 5. Schematic

of the first hierarehica! problem.

...... l

PROBABILISTIC

INFORMATION

PROCESSING

383

of how this case study was structured. The upper level of the hierarchy describes the hypotheses of central interest: the weapons system is being developed as a high priority project; it is being developed as a low priority project; or it is not being developed. In order that Shese three hypotheses form an exhaustive list of mutually exclusive events, the phrase "arLd a basic research effort on . . . is also ongoing" was added to each hypothesis. That is, similar observables eould be generated both as a consequence of weapons development and because the country was carrying out a basic research program for purely scientific reasons in a given area. It was necessary to work with three hypotheses since the analyst stated that the level of priority assigned to the development effort would affect, the status of supporting research and development activities and eventually the observables. The development of a weapons system would require certain kinds of supporting research and development activity. The second level of Fig. 5 refers to different research and development programs and related activities that would be expected to occur if the weapons system were being developed. In each of the illustrated eases, the research and development program is either in an advanced stage, an early stage, or has not begun. This distinction in time is necessary to reflect the different priorities in the level above. The third level of the hierarchy is comprised of sets of indicators of the research and development programs and activities. A good indicator has the property that it is very likely given a particular program and very unlikely otherwise. Usually, a separate set of indicators will be relevant to each program. It is important that indicators and activities be included which would be expected in the absence of a development program as well as in the presence of such a program. Otherwise, there is a danger that information gathering will be biased in such a way that all the data collected will support the conclusion that a weapons system is being developed. The fourth level describes events which are relevant to the indicators but whose oectrrence has only been repol%ed; they have not been directly observed by the analyst. Finally, the fifth and lowest level in the hierarchy is composed of observed events or dat~a such as research described in a scientific journal .or a defector's report describing research being carried out at a particular institute. Notice that it is the defector's report that w'~s unambiguously observed and not the event, described by his report. The fifth level of Fig. 5 also points out, that. occasionally the absence of data confirming the presence of an indicator may be an important datum and must be included to avoid biasing the results. The inference tree. After obtaining the basic structure of the problem

384

Gsrrvs et aI.

from the analysts, the next step in applying the hierarchical model was to construct an inference tree, displayed in Fig. 5. The procedure used to develop the tree made explicit use of subjective judgments of causality. The analysts began by assuming that. hi was true. They then generated a list of research and development programs and supporting activities given t.hat hi was true. Similarly, a list of indicators was generated for each program by assuming that particular states of level two were true. The process was then repeated for h2 and h3. Next, available data were linked to levels one, two, and three as appropriate. Where the data were in the form of reports of events, level four variables describing these events were constructed. It was found that some data which the analysts felt were relevant, to the problem had no point of entry to the tree. When this occurred, they were asked to hypothesize explanations for these data. These explanations then provided the basis for constructing additional descriptions of research and development programs and indicators. Thus, both "top down" and "bottom up" approaches were used to structure the tree. In general, the "top down" approach seemed easiest for the analysts, perhaps because it was deductive. Two analysts participated in the construction of this particular tree. One of them, the primary analyst, prepared an initial tree as a vehicle for discussion and debate. Both analysts then discussed this initial structure and modified it as appropriate. Throughout these discussions, the implications of the dependency constraints of the model were interpreted for the analysts. Because causal relations were used to construct the tree, the conditional independence requirements were almost always satisfied, however, and little restructuring was required. Elicitation of probabilities. Two kinds of probability estimates were required: conditional probabilities to link the first four levels and likelihood ratios linking the data to their respective levels. Before the conditional probabilities were estimated, the analysts were extensively questioned to determine if an assumption .of conditional independence was justified. The analysts' responses, describing the relations between variables in a path, indicated that. the assumption was valid. Since there were typically more research and development programs than hypotheses, probability estimates were made across hypotheses, completing all the estimates for one research and development program before starting the next. The analysts said that they preferred to work across hypotheses because there was more to keep in mind with a given research and development program than with the hypotheses--this implies that they perceived less change in set across hypotheses. Even so, it may be that this procedure should be reversed when the number

PROBABILISTIC INFORMATION PROCESSING

385

of hypotheses is greater than the, number of indicators. A similar proeedure was adopted for estimates of indicators given research and development programs. After all the estimates had been obtained, inconsistencies were identified for the analysts and they were directed to resolve them. It was quite obvious from the beginning that different estimates almost always implied different knowledge rather than a lack of understanding about the significance of probabilities. The discussion among the analysts was very revealing in this regard. Interestingly, the analysts with the greater knowledge were identified by their ability to generate a larger number of credible scenarios accounting for the presence of a datum and work with all of them rather than selecting one preferred explanation and adamantly defending it. When differences existed between estimates, it was usually because the less knowledgeable analyst had only considered a t!ew of the credible scenarios available. Typically, this affected estimates of P(Di[e2) not. P(D~Iel ) where el implied the occurrence of an event and e2 was its negation. Estimates of likelihood ratios linking the data to the appropriate level variables were obtained easily except in those eases where a datum was the nonoccurrence of an event. The difficulty with this kind of datum seemed to arise in describing precisely what. the nonoccurrence of an event was. For example, if a datum, Di, is the. statement, "the dog barked last. night," then does D~ mean, "the dog did not bark last night," or "the dog did not bark at 11:58 p.m."? That is, the dog did nothing or the dog did something at another time. The issue here is that an event, can occur only one way (usually) but it cannot occur in many, possibly nondenumerable, ways. Therefore, when the datum was the nonoccurrence of an event, the specific conditions or details of observation become crucial. One other kind of probability is needed within the hierarchy, that is the prier probability distribution over the upper level hypotheses. These estmates were based upon a logical rationale, derived from strategic and poliey considerations, describing why a given weapons system might or might not be developed. After .obtaining the necessary estimates, a posterior probability distribution was ealculated. In addition, likelihood ratios were calculated independently for each datum given the hypotheses of interest. Fittering. Computation of the independent likelihood ratios was quite revealing. They were almost all substantially smaller than the analysts believed that they should be, i.e., much of the impact of a datum was apparently filtered out in the hierarchy. After determining, that this was not an artifact induced by subjective suboptimality, two reasons were

386

~m'TYS et al.

identified as being responsible for the filtering. First, a datum might actually be composed of several data, only one of which was related to a particular indicator. For example, one datum was a technical report describing some ongoing research. This research per se was related to a specific indicator so the technical report datum was shown linked to that indicator. Questioning revealed, however, that in evaluating the impact of this datum the analyst included the author of the report, his institutional affiliation, and the list of references as additional data. The indicator, however, was only relevant to report content. This problem was solved by decomposing the datum into four data and constructing additional indicators. Second, many indicators contained much less detail than the data. That is~ they were so diffuse that they were about equally likely given any state of the researeh and development program to which they were related. Therefore, even if the indicator was known to be true, it would have little impact on the hypotheses. To counter this problem, the descriptions of the indicators and of the research and development programs were tightened up. The intent was to make each level more detailed than the level below it. This is another instance where the requirements of a hierarchical structure negate the likelihood principle. No negative data. The act of computing the posterior probability distribution called attention to what had been apparent all along, viz., that there were no diseonfirming data. To remedy this, the analysts were asked to hypothesize events which should occur given the truth of the indicators. The lower-level data were then reports that these events had not been observed. In most. eases, the reports were equally likely given that the event in question had or had not occurred; hence, they carried little diagnostic impact. Another approach was to take as a datum the fact that an indicator had not been observed. Here, by eliminating the event variable, more diagnostieity was preserved and some of these data were included in the inference tree. It is felt that the problem of a lack of diseonfirming data is a serious one which deserves much further study. It may be that the procedure used to construct the inference tree biases the result. The analysts found it difficult to describe negative indicators, for example, the indicator that a research and development program is not being carried out. Similarly, they found it difficult to hypothesize research and development programs which would be a eons.equenee of h3 vs hi and h~. It may also be true that the sampling or data collection process is biased toward collecting reports of occurrences of events versus nonoeeurrenees. Again, perhaps because the report of a nonoccurrence is so difficult to define.

PROBABILISTIC INFORMATION PROCESSII~'G

3~7'

CONCLUSION The two experiments presented here demonstrate that, although human intuitive inference in some multiple-stage situations m a y be close to optimum, in other cases it is quite suboptimum and is greatly improved by decomposing the inference into component assessments which can be combined formally using the optimum rule. With respect to the ease study, we believe that, independent of the degree to which the decomposition procedure can eliminate excessive biases, it has three definite advantages over the unaided procedure: (1) the expert m~king the assessments has a great deal more confidence in the final result, because he realizes t h a t all factors have been explicitly considered; (2) the hierarchical structure containing the expert's assessments is a justification for the final result which is open to scrutiny; and (3) areas of disagreement between experts can be quickly localized to specific areas for dialogue and discussion. REFERENCES DObSOn, J. D. Simulation system design for a TEAS simulation research facility. AFCRL 1112, PRC R-194, Planning Research Corporation, Los Angeles, CA, 1961. GETTYS, C., KELLY, C. W., & P~TERSON, C. R. The best-guess hypothesis in multistage inference. Organizational behavior and human per]ormance, 1973, 10, 364-373. GETTYS, C., & WILLKE, T. A. The application of Bayes' theorem when the true data state is uncertain. Organizational behavior and human per]ormance, 1969, 4, 125-141. KELLY, C. W. Application of Bayesian procedures to hierarchical inferences. Unpublished doctoral dissertation, University of Michigan, 1972. KELLY, C. W., & BAaCLAY,S. A general Bayesian model for hierarchical inference. Organizational behavior and human per]ormance, 1973, 10, 388-403. SCHUM, D. A., & KELLY, C. W. A problem in cascaded inference: Determining the

inferential i:mpaot of confirming and conflicting reports from several unreliable sources. Organizational behavior and human per]ormance, 1973, 10, 404-423. STEIGBR,J. H., & GETTYS, C. Best-guess errors in multi-stage inference. Journal o] experimental psychology, 1972, 9~, 1-7.