Understanding and using linguistic uncertainties

Acta Psychologica 68 (1988) 39-52 North-Holland

39

UNDERSTANDING AND USING LINGUISTIC UNCERTAINTIES * Thomas S. WALLSTEN University of North Carolina at Chapel Hill, USA

David V. BUDESCU University of Haifa, Israel

Ido EREV University of North Carolina at Chapel Hill, USA

In many real-world situations requiring choices or judgments, the available evidence is sparse, indirect, or imprecise, resulting in uncertainty that is more easily expressed verbally than numerically. This paper describes a series of studies on how people understand and use linguistic uncertainties. Our research on measuring the meanings of linguistic probabilities and on comparing bids for gambles described verbally or numerically is reviewed briefly. The two sets of studies appear to contrast with each other, in that the measurement studies document the vagueness of the linguistic expressions while the bidding studies show relatively small differences in response to the verbal and numerical descriptors. A theory that may reconcile the differences is outlined and supported by data from a choice study. Improvements to the theory may also account for the effects of context on phrase meaning.

Introduction This paper summarizes some of our recent research on how people understand linguistic probability expressions, such as doubtful, probable, or good chance, and introduces a theory of how decisions are made on the basis of such expressions. The issues are important for at least three reasons. First and most simply, people frequently use probability expressions. * This research was supported in part by Grant BNS 8608692 from the U.S. National Science Foundation and in part by Grant 8203394 from the US-Israel Binational Science Foundation. We thank the referees and Amnon Rapoport for their comments on earlier drafts. Requests for reprints should be sent to T.S. Wallsten, Dept. of Psychology, Davie Hall. CB 3270, University of North Carolina, Chapel Hill, NC 27599-3270. USA. OOOl-6918/88/$3.50

0 1988, Elsevier Science Publishers B.V. (North-Holland)

40

T.S. Wallsten et al. / Linguistic

uncertainties

Second, there is a controversy about whether such expressions should be used in risk assessment and forecasting, particularly when the uncertain events are related to the data only indirectly or by inference, or when the evidence itself is imprecise or linguistic in nature. For example, on the one hand, it is argued that quantitative risk assessment ‘may be illusory or misleading if the use of precise numbers is unjustified’, (U. S. National Research Council Governing Board Committee on the Assessment of Risk 1981: 15), whereas on the other, it is suggested that ‘For a decision maker, ambiguous probability statements are useless. They simply do not provide the information necessary to analyze a decision’ (Behn and Vaupel 1982: 78). Our own current recommendation is that forecasters should ‘estimate the uncertainty of future events as accurately and precisely as possible, but not . . . give the illusion of more precision than justified by the interpretation of the data’ (Budescu and Wallsten 1987: 67). It remains to be determined whether this recommendation can be satisfied with the appropriate use of linguistic probability phrases (see, e.g., Erev and Cohen, in press). Third, it is conceivable that information is processed differently when the final output is to be linguistic rather than numeric. For example, Zimmer (1983) has suggested that people generally handle uncertainty verbally and according to rules of language, rather than numerically and according to the rules of probability, and presented some data suggesting that linguistic information is actually processed more optimally than is numerical.

Probability phrase meanings Because the linguistic expression of uncertainty is so common, and in order to investigate more fully the issues discussed above, it is necessary to study how people understand and use verbal probability phrases. Our colleagues and we (Wallsten, Budescu, Rapoport, Zwick and Forsyth 1986b; Rapoport, Wallsten and Cox 1987) have developed and validated techniques for representing the vague meanings to individuals of linguistic probability expressions in specific contexts. These representations are in the form of functions over the [OJ] interval, as illustrated generically for four phrases in fig. 1. A particular function assumes its minimum value, generally zero, for probabilities not at all

T.S. Wallsten et al. / Linguistic uncertainties

41

Fig. 1. Four generic membership functions.

denoted by the phrase it represents; it assumes its maximum value, generally one, for probabilities definitely denoted by the phrase; and intermediate values for probabilities that are denoted by the phrase to some degree. Within fuzzy set theory such functions are called membership functions (e.g., Zadeh 1973, but it is not necessary to relate them specifically to that theory in order to justify their use in the present context. One method for establishing the membership functions makes use of graded pair-comparisons. A probability phrase is presented on each trial, below which are two probabilities (generally in the form of spinners) on either side of the computer screen, with a response line extending the width of the screen below the probabilities. The subject’s task is to move a cursor on the line to indicate which probability is better described by the phrase, and how much better it is described. By factorially manipulating the probabilities on the two sides of the screen for a particular phrase, conjoint-measurement and metric scaling techniques can be employed to establish and validate the function for that phrase. Wallsten et al. (1986b) established that the conjoint-measurement axioms were well satisfied with this procedure, goodness-of-fit measures for metric scaling procedures were high, individual differences in functions were large but stable, and finally that functions derived in one task successfully predicted those obtained in another.

42

T.S. Wahten et al. / Linguistic uncertainties

Because the pair-comparison procedures are so arduous, and in order to obtain converging evidence on the meaningfulness of the functions, Rapoport et al. (1987) compared the pair-comparison to a direct estimation technique. On each trial of the latter type, a probability phrase was paired with a probability value, with a response line just below them on the screen. The subject was to indicate how well the phrase described the probability by moving a cursor on the line, which was anchored at ‘not at all’ on the left and ‘absolutely’ on the right. A function was established for a given phrase by presenting it with numerous probability values. The two procedures generally yielded similarly shaped functions for a given person-phrase combination, but the directly estimated functions tended to indicate greater vagueness than did (i.e., lay above) the functions derived from pair-comparisons. Where the functions differed, certain analyses suggest that the latter may have been the more accurate. It was concluded, among other things, that although the resulting functions are not identical, the less taxing direct estimation method may generally yield sufficiently good results for most purposes. Fig. 1 shows the three basic forms of membership functions that have been established for probability phrases by Wallsten et al. (1986b), Rapoport et al. (1987), and Fillenbaum, Wallsten, Cohen, and Cox (1987). Phrases denoting relatively central probabilities, such as possible or rather unlikely, tend to be represented by single peaked functions such as shown for w2 or wg. Phrases denoting relatively low probabilities, such as almost impossible or doubtful tend to be represented by monotonic decreasing functions such as shown by wl. And finally, phrases denoting higher probabilities, such as very probable or almost certain, tend to be represented by monotonic increasing functions such as shown for w4. Meaning appears to depend on context, despite the fact that the vague meanings of probability expressions as represented by these functions are consistent to individuals within a context. Specifically, it appears that for a given individual the function representing a phrase is located more centrally and covers a broader interval when the phrase is received from, rather than selected for communication to, another person (Fillenbaum et al. 1987). In addition, the meanings of probability phrases (or at least of the high and middle ones) depend on the base rate of the events they are modifying (Wallsten, Fillenbaum and Cox

T.S. Wallsten et al. / Linguistic uncertainties

43

1986a, and references contained therein), and also on the desirability of the events they are describing (Cohen 1986).

Bidding studies While the work described above has focused on measuring the vague meanings of linguistic probabilities and on factors that affect such meanings, our more recent research has been concerned with how people actually make choices or judgments when faced with vague linguistic uncertainties. In a two-part study, Budescu, Weinberg, and Wallsten (1988) compared decisions based on numerically and verbally expressed probabilities. In stage 1, subjects selected and approximately equated numerical and verbal descriptors to 11 different graphical displays (spinners). Both sets of judgments were reliable, internally consistent, and monotonically related to the displays, but the numerical judgments were much less variable than the verbal, both within- and between-subjects. In stage 2, the subjects bid (using the procedure of Becker, De Groot and Marshak 1964) for simple two-outcome gambles, with one outcome always 0. The uncertain event was drawn from the previously equated numerical, verbal, and spinner descriptors. The gambles varied according to domain (gain-possible $1 win, or loss-possible $1 loss), original probability (11 values), and mode of presentation (numerical, verbal, or graphical). If verbal probabilities are vague and if vague uncertainties are generally avoided (e.g., Ellsberg 1961), then it can be predicted that in the positive domain subjects will bid less to obtain the verbal than the other gambles, and in the negative domain will pay more to give up the verbal than the other gambles, even at the sacrifice of expected value. The opposite predictions can be derived from Zimmer’s (1983, 1984) suggestions that communication is more natural in the verbal than in the numerical mode. That is, subjects will prefer and bid more optimally for the verbal gambles than for the others. Neither set of predictions was sustained. There was a small but significant effect of presentation mode on the absolute value of the bids, which are shown in table 1 averaged over all probabilities, opposite Budescu, Weinberg and Wallsten (1988). It can be seen that the subjects bid more to obtain the verbal than the other gambles in the domain of gains and offered more to avoid them in the domain of

44

T.S. WaIfsten et al. / Linguistic uncertainties

Table 1 Mean absolute adjusted bids in three experiments. Experiment

Stake

Domain

Mode Numeric

a

Verbal

Budescu, Weinberg and Wallsten (1988)

$1

Gains Losses

0.51 0.56

0.53 0.58

Budescu and Wallsten (1988) Experiment 1

$1

Gains Losses

0.55 0.49

0.55 0.49

$1

Gains Losses

0.53 0.47

0.52 0.48

$5

Gains b Losses b

0.57 0.51

0.56 0.56

Experiment 2

’ Entries for the Budescu, Weinberg and Wallsten (1988) study are averaged over numerical and graphic modes, which did not differ from each other. b Entries are mean bid divided by 5, in order to be comparable to the other entries.

losses. Nevertheless, they made significantly less money with the verbal than the other gambles in both domains (1.2% less for gains, 4.7% less for losses, but 24.4% less combined over both). The significant effects of presentation mode on bids were surprisingly small in light of the stage 1 differences in assigning numbers and phrases to displays, which themselves were consistent with the previous research establishing that probability phrases are vague to individuals. In order to determine whether the similarity in bids was artifactually caused by the stage 1 procedures, Budescu and Wallsten (1988) conducted two studies with dyads, in which one person was the forecaster and the other the decision maker. In the first experiment, the forecaster in one cubicle saw one of 11 spinners on a monitor and then communicated via the computer to the decision maker in another cubicle the chances of the spinner landing on white. Depending on the trial, the forecaster communicated either numerically or verbally. The decision maker was required to utilize the forecast in order to bid for a gamble in which the dyad would win or lose $1 if the spinner landed on white and would win $0 otherwise. Following each verbal forecast and associated bid, both members of the dyad indicated the numerical probability he or she considered the

T.S. Wallsten et al. / Linguistic uncertainties

45

forecast to have conveyed. Type of forecast varied randomly from trial to trial, and in addition, forecasters and decision makers exchanged roles from session 1 to sessions 2. Absolute bids averaged over displayed probabilities are shown in the second section of table 1. It is evident that, unlike in the previous study, there was no main effect of display mode. Display mode, however, did interact significantly with probability level and domain, but the effects were small and will not be discussed here. A second experiment extended the first and also controlled for certain possible artifacts that may have weakened the effects due to presentation mode. The non-zero outcomes within the gambles were gains or losses of $1 or of $5, and also follo&ing the bidding sessions, membership functions for the verbal forecasts were determined using the direct estimation method of Rapoport et al. (1987). Despite the various controls, the results are virtually identical to those of the previous study-no main effect of presentation mode, although numerous small, but significant interactions. The data are summarized in the remainder of table 1, which shows absolute adjusted bids for gains and losses and the two levels of stake, averaged over displayed probabilities and over various between-subject control factors. Note that for purposes of comparability, entries for the $5 gambles are the absolute bids divided by 5. However, an effect of display mode in experiment 2 does emerge when the variability of the bids is investigated. Specifically, for most subjects, the pooled variance of the adjusted bids to individual verbal forecasts exceeded the pooled variance of the adjusted bids to classes of numerical forecasts formed by rounding to the nearest 0.05. The ratio of verbal to numerical variances exceeded 1.0 for 25 of the 36 subjects, and overall had a mean value of 2.44 and a median value 1.67. A test of the log of the ratios was significant.

A theory and a choice experiment Considering the well-documented vagueness of probability phrases, and the commonly found effects of vagueness on preference (e.g., Einhorn and Hogarth 1985; Ellsberg 1961), the results of the three bidding experiments are surprising. Despite some main effects of presentation mode and interactions of presentation mode with other

46

T.S. Wallsten et al. / Linguistic uncertainties

factors, the mean bids are on the average very similar for gambles based on numerically or verbally described events referenced back to the same probability displays. Bids to specific verbal forecasts are, however, somewhat more variable than are bids to a 0.05 range of numerical forecasts. It is necessary to develop a theory that, among other desiderata, allows for the overriding similarity in decisions made to numerical and verbal descriptors of the same events. We describe a simple form of a theory of judgment given vague linguistic information, and then a choice experiment to test it. Finally, we relate the theory back to the bidding data and also indicate the direction in which we believe the theory must be enriched. For the simplest version of the theory, we assume that the vague meaning of a linguistic probability expression to an individual is represented by his or her membership function for that phrase. When required to act on the basis of a linguistic probability, the individual behaves in accordance with a specific probability value whose membership is sufficiently high, i.e., above a threshold that may depend on the task or on the importance of the decision. The probability value employed is randomly selected according to a weighting function that depends on the membership values above threshold, a notion similar to the concept underlying many choice models, that choice probabilities depend on response strengths (e.g., Lute 1977). This theory can be modeled by first assuming that probabilities are only considered if their membership value p is greater than or equal to a threshold Y. Hence, we call this the v-p model. Three possible thresholds are illustrated in fig. 1. A new function &,(p) is formed from pL,( p) for phrase w in conjunction with the threshold v, according to

PI,(P) =

0, if k(P)
I%(PI,

if j.tcL,(p) 2 v.

The density function f,(p) governing the selection of a specific probability value is then obtained by normalizing the membership function above the threshold: fJp)

=

f:(p)

/o P:(P) dp.

T.S. Walkten et al. / Linguistic uncertainties

-188 points 0 points

if white if red

ii.,t/I.~I1”LY to land on white

Do

you prefer to qlar the phrase gamble.

Press

S for

spinner

the

spinner

OF P for

or

phrase.

Fig. 2. Sample trial for the choice study.

The threshold v is the sole free parameter in the model, to be determined uniquely for each subject and each condition in a given task. The model was evaluated in two experiments in which subjects made binary choices between linguistic and precise gambles for identical outcomes (Wallsten and Erev, in prep.). The two yielded identical results, and only one of them will be outlined here. A typical trial is illustrated in fig. 2. In this example, the subject loses 100 points if a spinner lands on white and wins zero points if it lands on red. He or she can choose to gamble with the spinner on the right, for which the chances of landing on white can be estimated relatively precisely, or with the spinner on the left, for which the subject knows only that the chances of landing on white have been described as 1ikeZy by another person. The subject’s task was simply to indicate whether he or she preferred to play the spinner or the phrase gamble. In order to test for an effect on the threshold, outcomes for landing on white changed from trial to trial according to a two domain (gain or loss) by two stake (100 or 1,000 points) design. In all cases the subject won zero if the spinner landed on red. The linguistic gamble was represented by one of five phrases, doubtful, improbable, possible, good chance, and likely. Each phrase was paired with six different visible spinner probabilities of landing on white, selected to generally span the range of meanings for the phrase (0.03, 0.08,. . . ,0.28 for doubtfuI and improbable; 0.23, 0.33,. . . , 0.73 for possible; and 0.33, 0.43 ,..., 0.83 for good chance and likely). Over a series of three sessions, each specific

48

T.S. Wdsten

et al. / Linguistic uncertainties

choice situation was presented nine times. The selected gamble was actually played, but without feedback, and the subject’s point total was adjusted accordingly; subjects were told their scores at the end of each session. Following the choice sessions, membership functions were established for each phrase using the direct estimation method discussed by Rapoport et al. (1987), obtaining three replications per point. Ten subjects were paid for their time and in addition competed for bonuses based on point totals. On the basis of both an analysis of variance (ANOVA) on the choice frequencies and the model analysis, it was concluded that the outcome manipulations had no effect on behavior, and therefore, of course, none on the threshold (details are provided by Wallsten and Erev, in prep.). These manipulations will therefore be ignored in what follows. The P-P model was applied to the choice data with the addition of one more assumption, namely that following selection of a specific probability for the phrase meaning, the subject made an error (i.e., pressed the wrong key) with probability q in choosing which gamble to play (what Levine (1966) called an ‘oops error’). Thus for a choice between spinner probability p* and phrase w, the probability that the phrase is translated as having a value less than the spinner is, from eq. (2)P F,(P*)

=~p*fwb)

dp-

The probability that the phrase is translated as having a probability greater than the spinner is G,(P*)

=

1 -F,(P*).

Finally, invoking the error probability q, the probability that the subject actually chooses as if the phrase represents the higher value, Pr(w BP*), is f’+>~*)

=

(1- q)G,h*) + qF,(p*).

(3)

Based on Levine’s (1966) work, q was set equal to 0.02. Other values from 0.01 to 0.05 were tried, but with virtually no effect on the fit of the model.

T.S. Walisten et al. / Linguistic uncertainties

49

The primary data analysis involved first fitting a modified third order polynomial equation to each subject’s membership function judgments. Subsequently, a single threshold value, Y, to use in eq. (1) was determined for each subject such that the PI-( w > p * ) in eq. (3) for each ( W, p * ) combination were maximum likelihood estimates of the subject’s actual choice probabilities in the choice task. Membership functions and choice probabilities for each phrase varied considerably over subjects. Nevertheless, the model was reasonably successful for nine of the ten subjects, failing completely for one. The threshold values, V, are roughly uniformly distributed from 0.10 and 0;90 over the nine subjects. One descriptive measure of the fit of the model is the square of the correlation between observed and predicted choice probabilities, according to which the model handles from 70% to 95% of the variance in the observed values for the nine subjects, with a median of 85%. Linear functions were fit to the plot of observed vs. predicted choice proportions. For only two of the nine subjects were the regression lines significantly different from the expected (zero intercept and unit slope), although admittedly significance tests are inappropriate due to the manner in which the predicted values were obtained. Despite these favorable results, according to the most powerful and conservative maximum likelihood ratio test, the model deviates significantly from the data for all 10 subjects. The obtained x2 values were greater than 49.5 (p -C 0.01, df = 29) in all cases, ranging for the nine subjects from 70.5 to 334.6, with a median of 180.3. However, the deviations between observed and predicted choice probabilities were small (mean = 0.01 over the nine well described subjects) and relatively consistent. Specifically, the choice probabilities were generally predicted to be slightly too high for possible, good chance, and likely, although they were very accurate for the low phrases doubtfur and improbable.

General discussion The choice studies provide relatively strong support for the V-/A model, particularly in view of the fact that the model was tested on individual, not group data. The lack of a perfect fit between model and data is due in part to the already noted bias in the model predictions, but other less interesting factors probably also contributed. For exam-

50

T.S. Wailsten et al. / Linguistic uncertainties

ple, as described by Wallsten and Erev (in prep.), the membership functions were estimated with less precision than they should have been. Because the membership functions were estimated independently of the data they were used to predict, the model involved but a single free parameter per subject, namely the threshold, v. The threshold concept is clearly important, in that it allowed widely varying choice patterns to be related to equally diverse membership functions by means of a single model. Optimal threshold values varied across subjects over the entire allowable range, although they did not vary with outcome domain or value, as we had thought they might. It remains to be determined whether more extreme manipulations would affect threshold levels within a single task. Finally, it must be emphasized that it is the threshold that allows a vague probability phrase to be acted upon in a relatively precise manner, and therefore provides the conceptual link to the results of the bidding studies. The model cannot be applied quantitatively to the bidding data described earlier without invoking the questionable assumptions that subjects were actually bidding the subjectively expected utility of the gambles and that utility is linear in money. Nevertheless, if we assume that the phrases have relatively high membership values in the neighborhood of the probabilities they were used to describe, and also that decision makers in that task operated with relatively high thresholds, then it follows that bids to the phrases should be similar to bids to the numerical probabilities, as they were. The generally greater variance in the bids to verbal forecast occurred for those decision makers whose thresholds were less than 1. Our notion, yet to be explored, for reducing the bias in the v-p model predictions, is to assume that the individual has an opinion about the uncertain event prior to receiving the linguistic information. This opinion itself is represented as a function over [OJ], and may be uniform or have a particular shape to represent a particular judgment or bias. This idea is similar to that of the scope functions proposed by Zimmer (1984). The prior and linguistic membership functions are combined by a suitable operator, yielding a revised function to which the threshold is applied. Generally, when the two functions are combined, the outcome will be a function located more centrally in [OJ] and broader than the membership function. Also, the tighter is the membership function, the

T.S. Wallsten et al. / Linguistic uncertainties

51

less will be the relative effect of the combination. These results in conjunction with the general finding that low phrases are more precise than neutral or high ones (Wallsten et al. 1986a) are consistent with the pattern of deviations reported above between the predicted and observed choice probabilities. That is, the present model treated the phrases possible, good chance, and 1ikeZy as denoting slightly higher probabilities than the subjects apparently treated them. Doubtful and improbable, however, were handled quite accurately. If it turns out that the context function systematically improves the predictions of the model, it may also provide an account of the systematic effects on meaning noted earlier. For example, the base rate and desirability effects occur because the interpretation of a phrase in a given context depends on the combination of the modal phrase meaning with the prior judgment for that situation, with the former represented by a context-free membership function and the latter by a context function. The theory and model discussed here are preliminary. However, we believe that the threshold concept is an important one in relating vague meanings to precise actions, and are encouraged that with additional refinements a wide range of results can be brought together.

References Becker, G.M., M.H. DeGroot and J. Marshak, 1964. Measuring utility by a single-response sequential method. Behavorial Science 9,226-232. Bebn, R.D. and J.W. Vaupel, 1982. Quick analysis for busy decision makers. New York: Basic Books. Budescu, D.V. and T.S. Wahsten, 1987. ‘Subjective estimation of vague and precise uncertainties’. In: G. Wright and P. Ayton (eds.), Judgmental forecasting. Cbichester: Wiley. Budescu, D.V. and T.S. Wahsten, 1988. Dyadic decisions with numerical and verbal probabilities. (Submitted for publication.) Budescu, D.V., S. Weinberg and T.S. Wallsten, 1988. Decisions based on numericahy and verbally expressed uncertainties. Journal of ExperimentaI Psychology: Human Perception and Performance 14, 281-294. Cohen, B.L., 1986. The effect of outcome desirability on comparisons of hnguistic and numerical probabilities. Unpublished MA thesis, University of North Carolina at Chapel Hill Eþ, H.J. and R.M. Hogarth, 1985. Ambiguity and uncertainty in probabilistic inference. Psychological Review 92,433~461. B&berg, D., 1961. Risk, ambiguity, and the Savage axioms. Quarterly Journal of Economics 75, 643-669. Erev, I. and B.L. Cohen, in press. Verbal versus numerical probabilities: efficiency, biases and the preference paradox. Organizational Behavior and Human Decision Processes.

52

T.S. Wdsten

et al. / Linguistic uncertainties

Fillenbaum, S., T.S. Wallsten, B.L. Cohen and J.A. Cox, (1987. Some effects of available vocabulary and communication task on the understanding and use of non-numerical probability expressions. L.L. Thurstone Psychometric Laboratory Report No. 177. Chapel Hill, NC: University of North Carolina. Levine, M., 1966. Hypothesis behavior by humans during discrimination learning. Journal of Experimental Psychology 71, 331-338. Lute, R.D., 1977. The choice axiom after twenty years. Journal of Mathematical Psychology 15, 215-233. Rapoport, A., T.S. Wallsten and J.A. Cox, 1987. Direct and indirect scaling of membership functions of probability phrases. Mathematical Modelling 9, 397-417. U.S. National Research Council Governing Board Committee on the Assessment of Risk, 1981. The handling of risk assessments in NRC Reports. Washington, DC: U.S. National Research council. Wallsten, T.S. and I. Erev, in preparation. Choosing between linguistic and precise gambles: a test of a theory of choice given vague information. Wallsten, T.S., S. Fillenbaum and J.A. Cox, 1986a. Base rate effects on the interpretations of probability and frequency expressions. Journal of Memory and Language 25, 571-587. Wallsten, T.S., D.V. Budescu, A. Rapoport, R. Zwick and B. Forsyth, 1986b. Measuring the vague meanings of probability terms. Journal of Experimental Psychology 115, 348-365. Zadeh, L.A., 1975. The concept of a linguistic variable and its application to approximate reasoning. Parts 1, 2, 3. Information Sciences 8, 199-249; 8, 301-357; 9,43-98. Zimmer, A.C., 1983. ‘Verbal vs. numerical processing of subjective probabilities’. In: R.W. Scholz (ed.), Decision making under uncertainty. Amsterdam: North-Holland. Zimmer, AC., 1984. A model for the interpretation of verbal predictions. International Journal of Man-Machine Studies 20, 121-134.