- Email: [email protected]
Mental models as an explanation of belief bias effects in syllogistic reasoning
Cognition,
46 (1993)
93-97
Discussion Mental models as an explanation of belief bias effects in syllogistic reasoning* Stephen
E. Newstead
Department
of Psychology,
Newstead, syllogistic
and
Jonathan
St.B.T.
Evans
University of Plymouth, Plymouth PL4 8AA,
S.E., and Evans, J.St.B.T., 1993. Mental reasoning. Cognition, 46: 93-97.
models
UK
as an explanation
of belief bias effects
in
During the past decade or so-the period in which we have been conducting research into belief bias effects in syllogistic reasoning - there have, to the best of our knowledge, been only two explanations put forward for the interaction between validity and belief. These are the theories that have been termed the selective scrutiny account and the misinterpreted necessity account (Evans, 1989). The research that is reported in our paper (Newstead, Pollard, Evans, & Allen, 1992) was motivated at the outset by a desire to test between these alternative explanations. It was only when we found data that seemed inconsistent with both these models that we looked for alternative theoretical accounts, and started to develop an explanation based on the theory of mental models. Our paper (Newstead et al., 1992) is the first published attempt to explain the interaction in terms of mental models; up until this time, this approach had been used only to explain the existence of belief bias. It is useful to establish this chronology of theoretical development in order to put some of the comments of Oakhill and Garnham (1993) into perspective. They claim on two separate occasions that both the selective scrutiny and the misinterpreted necessity accounts had been disproved by the results of studies by Oakhill, Johnson-Laird, and Garnham (1989) and should have been rejected because of these findings. More specifically, they point out that their failure to Correspondence to: Stephen E. Newstead, Plymouth PL4 8AA, UK. *A rejoinder to Oakhill, J. and Garnham, reasoning. Cognition, $6, 87-92. OOlO-0277/93/$05.00
0
1993 - Elsevier
Department A. (1993).
Science
of Psychology, On theories
Publishers
University of belief
of Plymouth,
bias in syllogistic
B.V. All rights reserved.
94
S. E. Newstead
find any effects
and J.St. B. T. Evans
of believability
on multiple-model,
determinate
syllogisms
runs
counter to the predictions of both theories. In the case of the misinterpreted necessity model this is simply not true. Oakhill and Garnham (1993) assert that this model claims that subjects are guided by believability whenever a conclusion is neither falsified nor determined by the premises, and that this applies to both determinate and indeterminate syllogisms. This is a misrepresentation says is that subjects fail to distinguish
of what the model claims. What it actually between conclusions which are definitely
false and those which are possibly false, and that when they encounter conclusions which are possibly false, they base their response on the believability of the conclusion. This theory says nothing about determinately valid syllogisms. The theory was put forward primarily to explain the interaction between logic and belief, not to explain belief bias effects in general, and hence it is unaffected by any finding that belief bias does not occur with valid syllogisms. The claim by Oakhill and Garnham that their failure to find belief bias effects on determinate (i.e., valid) multiple-model syllogisms goes against the predictions of the misinterpreted necessity model is simply incorrect. The selective scrutiny account, on the other hand, would predict belief bias effects on all syllogisms whether they are valid or not, and any evidence that the effects do not exist is clearly difficult to encompass within such a theory. However, the same applies to the mental models account. In fact, according to the mental models account, there are two separate sources of belief bias effects on these problems: since these are multiple model syllogisms, subjects should fail to construct alternative models when the first model yields a believable conclusion; and there should also be some conclusion filtering taking place to screen out unbelievable conclusions. We know of no account of belief bias effects which can explain the failure to obtain effects of believability on these multiple-model syllogisms, and hence it seems rather strange for Oakhill and Garnham to use this finding solely to criticize the selective scrutiny and misinterpreted necessity accounts. We were fully aware of these difficulties in writing our paper, and indeed we comment upon them. In our Experiment 4, we too failed to obtain an effect of believability, and drew attention to the fact that Oakhill et al. (1989) had obtained a similar result. However, we also pointed out that this finding is far from universal. Earlier research in our laboratory (Evans, Barston, & Pollard, 1983) found significant effects of believability in three separate studies using precisely this type of syllogism, and we have also found such effects in recent, unpublished studies (Evans, Allen, Newstead, & Pollard, in preparation). There is almost certainly something unusual about these syllogisms. One feature that they share in common, as we pointed out in our paper, is that they all though why this should lead to idiosyncratic lead to “some . . . not” conclusions, response patterns is not clear. Inspection of any database of syllogistic reasoning
Belief bias in syllogistic reasoning
behaviour syllogisms
(e.g., Dickstein, 1978) reveals that the predominant is that no valid conclusion follows. Perhaps these
95
response on such syllogisms are so
complex that subjects cannot construct any coherent representation of them and hence respond that nothing follows. This tendency to say that no conclusion follows whenever the representation becomes too complicated may be a strong one and may overpower the effects of belief bias. Although the results of our experiments provided support for the mental models theory, it must be pointed out there there are some findings which it does not explain particularly well. One continuing weakness of the approach is the way in which it explains belief bias effects on single-model problems. It was originally predicted that these problems would not yield any belief bias effects, but studies have repeatedly shown that they do. Mental models theorists explain this by a rather arbitrary, add-on mechanism - that of conclusion filtering. There is nothing in the theory of mental models that would lead to the postulation of such a mechanism, and it seems to have been developed simply to explain the findings. In fact, it has much more in common with response bias approaches to reasoning (such as selective scrutiny) than to mental models, a point we repeatedly stressed in our paper. Oakhill and Garnham (1993) are uncomfortable with this and attack the general idea of selective scrutiny. This seems a little strange given that conclusion filtering is essentially the same as selective scrutiny; the main difference is in when the filtering (or selection) occurs, not in the type of mechanism involved. It should also be pointed out that the instructional effects that we obtained in our Experiment 5, and which produced such convincing support for the mental models approach, may not be as robust as we first thought. Evans et al. (in preparation) have conducted three further experiments on the effects of instructions, and instructions stressing the importance of logical processing have not been uniformly effective in reducing or eliminating the interaction between belief and validity. Logic instructions do seem to inhibit both belief bias and the interaction between logic and belief, but they do not always eliminate these effects, nor is the inhibition consistently significant. Oakhill and Garnham attach some importance to the finding of Evans and Pollard (1990) that belief bias was independent of the complexity of the problems. Evans and Pollard interpreted this as evidence for the selective scrutiny account, or more precisely as evidence for the view that belief exerts its influence prior to reasoning taking place. Otherwise we might expect that there would be more belief bias on more complex problems, instead of just more random error as actually occurs. Oakhill and Garnham argue that this cannot be taken as unequivocal support for the selective scrutiny account since this latter fails to specify the process of reasoning. In particular, subjects might come up with false positive rather than non-propositional conclusions. They further claim that “there is no way of determining the relative proportions of simple and complex problems
96
S. E. Newstead
on which interest
and J.2.
believability
is the overall
B. T. Evans
can
have
an effect”.
rate of acceptance
Why not? Surely the measure of conclusions in each condition?
of
In the first experiment reported by Evans and Pollard, conditional inferences were incorporated into a complex condition known, from previous research, to lead to “no valid conclusion” responses. Inspection of their Table 5 shows that the mean acceptance ratings were considerably lower for the complex condition but that the relative effect of belief was not significantly different. It is not clear what would be predicted by Oakhill and Garnham in this situation since, as we have already seen, belief can have its effects at two different locations. One might argue that complexity should increase the difficulty of searching for counterexamples, thus leading to acceptance of spuriously believable conclusions. On the other hand, if fewer putative conclusions can be derived from the complex problems - as appears to be the case - then post-conclusion filtering of unbelievable conclusions cannot occur so often, thus leading to less belief bias. It might even be argued that the two effects should cancel each other out, thus leading to no overall difference between simple and complex problems. In their second experiment, Evans and Pollard manipulated complexity in a different way, by the use of additional premises. Inspection of their Table 7 shows - especially for definitional material - that the effect on conclusion acceptability was opposite to that obtained in their first experiment, that is, subjects rated conclusions as more acceptable in the complex condition, despite a large decline in their logical accuracy. This, then, fits Oakhill and Garnham’s prediction that complex problems might lead to false positive conclusions. Why, however, is the increased acceptance of believable and unbelievable conclusions equal? Since selective scrutiny - according to Oakhill and Garnham - occurs after the production of putative conclusions, then surely these authors must predict more belief bias with availability
these complex problems ? The effect should be magnified by of more putative but unbelievable conclusions being available
the for
post-conclusion filtering. While the study reported by Newstead et al. produced results which are highly encouraging for the mental models approach, it would be wrong to argue that the matter is closed. In this rejoinder we have pointed out a number of problems with the theory, perhaps the most telling of which is that it needs to be supplemented by a number of response biases (conclusion filtering, producing “no valid conclusion” responses when problems are excessively difficult) if it is to explain all the data. Mental models theory is the best that we currently have available, but it is unlikely to be the final word in research in this area.
References Dickstein, L. (1978). The effect of figure on syllogistic reasoning. Memory & Cognition, h, 76-83. Evans, J.St.B.T. (1989). Bias in humun reasoning: Causes and consequences. Brighton: Erlbaum.
Belief bias in syllogistic
Evans,
reasoning
97
J.St.B.T.. Allen, J.L., Newstead, S.E., & Pollard, P. (in preparation). Logical necessity and belief bias in reasoning: The effect of instructions. Evans. J.St.B.T., Barston, J.L. & Pollard, P. (1983) On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295-306. Evans, J.Q.B.T., & Pollard, P. (1990). Belief bias and problem complexity in deductive reasoning. In J.P. Caverni, J.M. Fabre & M. Gonzales (Eds.), Cognitive biases. Amsterdam: North-Holland. Newstead, S.E., Pollard, P., Evans, J.St.B.T., & Allen, J.L. (1992). The source of belief bias effects in syllogistic reasoning. Cognition, 45, 257-284. Oakhill. J., & Garnham, A. (1993). On theories of belief bias in syllogistic reasoning. Cognition, 46, 87-92. Oakhill, J., Johnson-Laird, P.N.. & Garnham, A. (1989). Believability and syllogistic reasoning. Cognifion, 31, 117-140.
46 (1993)
93-97
Discussion Mental models as an explanation of belief bias effects in syllogistic reasoning* Stephen
E. Newstead
Department
of Psychology,
Newstead, syllogistic
and
Jonathan
St.B.T.
Evans
University of Plymouth, Plymouth PL4 8AA,
S.E., and Evans, J.St.B.T., 1993. Mental reasoning. Cognition, 46: 93-97.
models
UK
as an explanation
of belief bias effects
in
During the past decade or so-the period in which we have been conducting research into belief bias effects in syllogistic reasoning - there have, to the best of our knowledge, been only two explanations put forward for the interaction between validity and belief. These are the theories that have been termed the selective scrutiny account and the misinterpreted necessity account (Evans, 1989). The research that is reported in our paper (Newstead, Pollard, Evans, & Allen, 1992) was motivated at the outset by a desire to test between these alternative explanations. It was only when we found data that seemed inconsistent with both these models that we looked for alternative theoretical accounts, and started to develop an explanation based on the theory of mental models. Our paper (Newstead et al., 1992) is the first published attempt to explain the interaction in terms of mental models; up until this time, this approach had been used only to explain the existence of belief bias. It is useful to establish this chronology of theoretical development in order to put some of the comments of Oakhill and Garnham (1993) into perspective. They claim on two separate occasions that both the selective scrutiny and the misinterpreted necessity accounts had been disproved by the results of studies by Oakhill, Johnson-Laird, and Garnham (1989) and should have been rejected because of these findings. More specifically, they point out that their failure to Correspondence to: Stephen E. Newstead, Plymouth PL4 8AA, UK. *A rejoinder to Oakhill, J. and Garnham, reasoning. Cognition, $6, 87-92. OOlO-0277/93/$05.00
0
1993 - Elsevier
Department A. (1993).
Science
of Psychology, On theories
Publishers
University of belief
of Plymouth,
bias in syllogistic
B.V. All rights reserved.
94
S. E. Newstead
find any effects
and J.St. B. T. Evans
of believability
on multiple-model,
determinate
syllogisms
runs
counter to the predictions of both theories. In the case of the misinterpreted necessity model this is simply not true. Oakhill and Garnham (1993) assert that this model claims that subjects are guided by believability whenever a conclusion is neither falsified nor determined by the premises, and that this applies to both determinate and indeterminate syllogisms. This is a misrepresentation says is that subjects fail to distinguish
of what the model claims. What it actually between conclusions which are definitely
false and those which are possibly false, and that when they encounter conclusions which are possibly false, they base their response on the believability of the conclusion. This theory says nothing about determinately valid syllogisms. The theory was put forward primarily to explain the interaction between logic and belief, not to explain belief bias effects in general, and hence it is unaffected by any finding that belief bias does not occur with valid syllogisms. The claim by Oakhill and Garnham that their failure to find belief bias effects on determinate (i.e., valid) multiple-model syllogisms goes against the predictions of the misinterpreted necessity model is simply incorrect. The selective scrutiny account, on the other hand, would predict belief bias effects on all syllogisms whether they are valid or not, and any evidence that the effects do not exist is clearly difficult to encompass within such a theory. However, the same applies to the mental models account. In fact, according to the mental models account, there are two separate sources of belief bias effects on these problems: since these are multiple model syllogisms, subjects should fail to construct alternative models when the first model yields a believable conclusion; and there should also be some conclusion filtering taking place to screen out unbelievable conclusions. We know of no account of belief bias effects which can explain the failure to obtain effects of believability on these multiple-model syllogisms, and hence it seems rather strange for Oakhill and Garnham to use this finding solely to criticize the selective scrutiny and misinterpreted necessity accounts. We were fully aware of these difficulties in writing our paper, and indeed we comment upon them. In our Experiment 4, we too failed to obtain an effect of believability, and drew attention to the fact that Oakhill et al. (1989) had obtained a similar result. However, we also pointed out that this finding is far from universal. Earlier research in our laboratory (Evans, Barston, & Pollard, 1983) found significant effects of believability in three separate studies using precisely this type of syllogism, and we have also found such effects in recent, unpublished studies (Evans, Allen, Newstead, & Pollard, in preparation). There is almost certainly something unusual about these syllogisms. One feature that they share in common, as we pointed out in our paper, is that they all though why this should lead to idiosyncratic lead to “some . . . not” conclusions, response patterns is not clear. Inspection of any database of syllogistic reasoning
Belief bias in syllogistic reasoning
behaviour syllogisms
(e.g., Dickstein, 1978) reveals that the predominant is that no valid conclusion follows. Perhaps these
95
response on such syllogisms are so
complex that subjects cannot construct any coherent representation of them and hence respond that nothing follows. This tendency to say that no conclusion follows whenever the representation becomes too complicated may be a strong one and may overpower the effects of belief bias. Although the results of our experiments provided support for the mental models theory, it must be pointed out there there are some findings which it does not explain particularly well. One continuing weakness of the approach is the way in which it explains belief bias effects on single-model problems. It was originally predicted that these problems would not yield any belief bias effects, but studies have repeatedly shown that they do. Mental models theorists explain this by a rather arbitrary, add-on mechanism - that of conclusion filtering. There is nothing in the theory of mental models that would lead to the postulation of such a mechanism, and it seems to have been developed simply to explain the findings. In fact, it has much more in common with response bias approaches to reasoning (such as selective scrutiny) than to mental models, a point we repeatedly stressed in our paper. Oakhill and Garnham (1993) are uncomfortable with this and attack the general idea of selective scrutiny. This seems a little strange given that conclusion filtering is essentially the same as selective scrutiny; the main difference is in when the filtering (or selection) occurs, not in the type of mechanism involved. It should also be pointed out that the instructional effects that we obtained in our Experiment 5, and which produced such convincing support for the mental models approach, may not be as robust as we first thought. Evans et al. (in preparation) have conducted three further experiments on the effects of instructions, and instructions stressing the importance of logical processing have not been uniformly effective in reducing or eliminating the interaction between belief and validity. Logic instructions do seem to inhibit both belief bias and the interaction between logic and belief, but they do not always eliminate these effects, nor is the inhibition consistently significant. Oakhill and Garnham attach some importance to the finding of Evans and Pollard (1990) that belief bias was independent of the complexity of the problems. Evans and Pollard interpreted this as evidence for the selective scrutiny account, or more precisely as evidence for the view that belief exerts its influence prior to reasoning taking place. Otherwise we might expect that there would be more belief bias on more complex problems, instead of just more random error as actually occurs. Oakhill and Garnham argue that this cannot be taken as unequivocal support for the selective scrutiny account since this latter fails to specify the process of reasoning. In particular, subjects might come up with false positive rather than non-propositional conclusions. They further claim that “there is no way of determining the relative proportions of simple and complex problems
96
S. E. Newstead
on which interest
and J.2.
believability
is the overall
B. T. Evans
can
have
an effect”.
rate of acceptance
Why not? Surely the measure of conclusions in each condition?
of
In the first experiment reported by Evans and Pollard, conditional inferences were incorporated into a complex condition known, from previous research, to lead to “no valid conclusion” responses. Inspection of their Table 5 shows that the mean acceptance ratings were considerably lower for the complex condition but that the relative effect of belief was not significantly different. It is not clear what would be predicted by Oakhill and Garnham in this situation since, as we have already seen, belief can have its effects at two different locations. One might argue that complexity should increase the difficulty of searching for counterexamples, thus leading to acceptance of spuriously believable conclusions. On the other hand, if fewer putative conclusions can be derived from the complex problems - as appears to be the case - then post-conclusion filtering of unbelievable conclusions cannot occur so often, thus leading to less belief bias. It might even be argued that the two effects should cancel each other out, thus leading to no overall difference between simple and complex problems. In their second experiment, Evans and Pollard manipulated complexity in a different way, by the use of additional premises. Inspection of their Table 7 shows - especially for definitional material - that the effect on conclusion acceptability was opposite to that obtained in their first experiment, that is, subjects rated conclusions as more acceptable in the complex condition, despite a large decline in their logical accuracy. This, then, fits Oakhill and Garnham’s prediction that complex problems might lead to false positive conclusions. Why, however, is the increased acceptance of believable and unbelievable conclusions equal? Since selective scrutiny - according to Oakhill and Garnham - occurs after the production of putative conclusions, then surely these authors must predict more belief bias with availability
these complex problems ? The effect should be magnified by of more putative but unbelievable conclusions being available
the for
post-conclusion filtering. While the study reported by Newstead et al. produced results which are highly encouraging for the mental models approach, it would be wrong to argue that the matter is closed. In this rejoinder we have pointed out a number of problems with the theory, perhaps the most telling of which is that it needs to be supplemented by a number of response biases (conclusion filtering, producing “no valid conclusion” responses when problems are excessively difficult) if it is to explain all the data. Mental models theory is the best that we currently have available, but it is unlikely to be the final word in research in this area.
References Dickstein, L. (1978). The effect of figure on syllogistic reasoning. Memory & Cognition, h, 76-83. Evans, J.St.B.T. (1989). Bias in humun reasoning: Causes and consequences. Brighton: Erlbaum.
Belief bias in syllogistic
Evans,
reasoning
97
J.St.B.T.. Allen, J.L., Newstead, S.E., & Pollard, P. (in preparation). Logical necessity and belief bias in reasoning: The effect of instructions. Evans. J.St.B.T., Barston, J.L. & Pollard, P. (1983) On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295-306. Evans, J.Q.B.T., & Pollard, P. (1990). Belief bias and problem complexity in deductive reasoning. In J.P. Caverni, J.M. Fabre & M. Gonzales (Eds.), Cognitive biases. Amsterdam: North-Holland. Newstead, S.E., Pollard, P., Evans, J.St.B.T., & Allen, J.L. (1992). The source of belief bias effects in syllogistic reasoning. Cognition, 45, 257-284. Oakhill. J., & Garnham, A. (1993). On theories of belief bias in syllogistic reasoning. Cognition, 46, 87-92. Oakhill, J., Johnson-Laird, P.N.. & Garnham, A. (1989). Believability and syllogistic reasoning. Cognifion, 31, 117-140.