Selection and use of propositional knowledge in statistical problem solving

Learning and Instruction 12 (2002) 323–344 www.elsevier.com/locate/learninstruc

Selection and use of propositional knowledge in statistical problem solving Nick J. Broers

*

Department of Methodology and Statistics, Maastricht University, P.O. Box 616, 6200 MD Maastricht, Netherlands

Abstract Central to this study is the question of why subjects who possess the necessary factual or propositional knowledge needed to solve a particular statistical problem, often fail to find the solution to that problem. Ten undergraduate psychology students were trained so as to possess all the relevant knowledge needed to solve five multiple choice problems on descriptive regression analysis. They were asked to think aloud while attempting to solve the problems. Analysis of the think-aloud protocols showed that a failure to select the relevant information in the text, together with a failure to retrieve relevant propositional knowledge from memory and a difficulty with logical reasoning combined to produce incorrect responses. Factual knowledge was less likely to be successfully retrieved when it was acquired only recently or when it concerned relationships of a highly abstract nature. Furthermore, the existence of misconceptions appeared to inhibit the use of correct factual knowledge.  2002 Elsevier Science Ltd. All rights reserved. Keywords: Components of statistical knowledge; Statistical problem solving; Propositional knowledge of statistics; Conceptual understanding of statistics

1. Introduction Worldwide, and across academic disciplines as diverse as Biomedics, Business Administration, Psychology and Health Sciences, statistics is deemed to be an indispensable part of the curriculum. Although different disciplines may emphasize different techniques, a quick review of a sample of elementary statistics texts suggests a

* Tel.: +31-43-3882274. E-mail address: [email protected] (N.J. Broers). 0959-4752/02/$ - see front matter  2002 Elsevier Science Ltd. All rights reserved. PII: S 0 9 5 9 - 4 7 5 2 ( 0 1 ) 0 0 0 2 5 - 1

324

N.J. Broers / Learning and Instruction 12 (2002) 323–344

considerable similarity in the content of these courses. Yet, what constitutes statistical knowledge remains surprisingly vague. Very simply, it could be argued that statistical knowledge is represented by the successful acquisition of the facts and procedures that form the content of an elementary statistical textbook. As Hubbard (1997) points out, however, a simple enumeration of facts and a cookbook application of statistical techniques will in general not be considered as a satisfactory demonstration of statistical aptitude. How then, does true understanding of statistical theory differ from simple memorization of facts and procedures? Assessment of statistical knowledge, following a curriculum course, can be made by taking an exam containing either open ended or multiple choice questions. In the latter case, it is customary to assess the internal consistency of the resulting index by calculating Cronbach’s alpha coefficient and studying the inter-item correlations. Usually, the response pattern will not conform to a unidimensional structure, suggesting that statistical knowledge actually represents multiple underlying dimensions. Some insight into the components of statistical knowledge can be gained from research that has been conducted into the nature of errors committed in the process of tackling statistical problems. Allwood and Montgomery carried out several research projects in each of which they had a number of first year university students trying to solve two statistical problems on descriptive statistics while thinking aloud (Montgomery & Allwood 1978, 1990; Allwood & Montgomery 1981, 1982; Allwood, 1984). Their main focus in these studies concerned the nature of errors made by subjects in the process of problem solving. As the problems presented to the subjects required extensive computational effort, it is not surprising that a substantial amount of errors directly or indirectly pertained to faulty calculations (Allwood & Montgomery, 1982). Probably related to this, they noted that good problem solvers could be distinguished from bad ones by the precision with which they tackled the problems. The better problem solvers tended to read the text of the problem more carefully and in addition more often attempted to clarify the concepts with which they were confronted (Montgomery & Allwood, 1978). Allwood (1990) focussed on the relationship between algorithmic (or procedural) and conceptual (or propositional) knowledge. These two types of knowledge, defined by cognitive psychologists as respectively referring to knowing how to do something and knowing about something (see e.g. Anderson, 1983), were shown by Allwood to play a joint role in succesful statistical problem solving. Although it was possible for some subjects to deliver a successful solution to the problems by the use of procedural knowledge alone (in 12% of cases where a correct solution method was chosen, subjects failed to show meaningful comprehension of the methods used or the solutions produced), successful problem solving was clearly enhanced if subjects consciously reflected on the factual justification for the different procedural steps that they executed. Huberty, Dresden, and Bak (1993) conducted a study with the purpose of identifying the multidimensional structure of statistical knowledge. They hypothesized and found empirical support for the existence of three important dimensions, one referring to computational aptitude (procedural knowledge), one covering

N.J. Broers / Learning and Instruction 12 (2002) 323–344

325

propositional knowledge and a third pertaining to conceptual understanding. With propositional knowledge Huberty, Dresden, and Bak meant factual knowledge, or knowledge of concepts, propositions and ideas regarding the subject of statistics. Conceptual understanding was interpreted by them as the existence of links and connections between these various concepts and ideas. It is important to note that in cognitive psychology the term ‘propositional knowledge’ is usually meant to refer both to knowledge of concepts and ideas, as well as to relations between those concepts and ideas. In their use of the term, Huberty, Dresden, and Bak therefore employ a more restricted definition of ‘propositional knowledge’, excluding the relations between concepts. In a related subject like mathematics, a similar distinction between types of knowing is widely used. Procedural knowledge lies at the core of much of mathematics and arithmetic, but the algorithmic manipulation of symbols and formulae can be carried out correctly without any understanding of the procedures. To acquire meaningful knowledge, one must be aware of the underlying concepts and their interrelations. Hiebert and Lefevre (1986) see this process of tying together previously isolated mathematical concepts as a maturation process. As more and more concepts become interrelated, the knowledge of the student becomes elevated to a higher level of abstraction. Lampert (1986) refers to an interconnected body of propositions as principled knowledge, and claims that only this type of knowledge reflects meaningful understanding of mathematics: “the possession of principled understanding (…enables…) the knower to invent procedures that are mathematically appropriate and to recognize that what he or she knows can be applied in a variety of different contexts” (Lampert, 1986, p. 310). In cognitive psychology, a key concept that relates to general and abstract networks of interrelated propositions is that of the ‘schema’. Schemata are formed because students learn to recognize common features in a wide diversity of problem solving situations. Recognizing a particular problem as belonging to a class of structurally similar problems, an advanced problem solver (or ‘expert’) is able to proceed towards a solution by using the appropriate cognitive schemata. Whereas initially simple and concrete propositions are related to form simple schemata, eventually such schemata serve to become cognitive units that are related to other schemata to form new cognitive schemata of ever greater generality and abstraction. A greater level of expertise is gradually attained, not only by linking more propositions to the already existing knowledge structures, but also by deleting or modifying propositions that are shown to be wholly or partially incorrect (Marshall, 1995). In the domain of science, several studies have shown the importance of the distinction between knowledge of facts, terms and procedures on the one hand, and conceptual understanding on the other, in which the individual concepts and ideas have been integrated into a network of interrelations. Bromage and Mayer (1981) showed that comprehensive knowledge of isolated facts concerning the use of cameras did not in itself result in the successful solving of problems related to the use of cameras. Whether or not students possessed knowledge of the principles that tied the individual laws together did prove to be a reliable predictor of succesful problem solving. Likewise, in the field of physics, Chi and Bassok showed that

326

N.J. Broers / Learning and Instruction 12 (2002) 323–344

it is not unusual for a subject to deliver a successful demonstration of propositional knowledge (in the restricted sense as also used by Huberty, Dresden, & Bak, 1993) and at the same time to fail to apply that knowledge in a meaningful way when confronted with a related problem solving task (Chi & Bassok, 1989; Chi, Bassok, Lewis, Reimann, & Glaser, 1989). In their explorative study, Chi and Bassok focussed on differences between good and poor problem solvers. They showed that the successful group differed from the poor performers in that they were better able to monitor the state of their own comprehension. Where they lacked sufficient comprehension, the successful students engaged in self-explanations, in which they tried to map the principles they had previously learned to the problem under consideration. Poor students did not usually detect their own inadequate comprehension, and neither did they engage in self-explanations. The researchers interpret this as due to the fact that the poor students, although they had shown knowledge of the necessary principles and concepts on a preliminary test, had not developed more complex cognitive schemata that would have enabled them to tackle the problems more successfully. Propositional knowledge seems to be a necessary, but not a sufficient condition for obtaining conceptual understanding. As yet little systematic research has investigated the reasons for the failure of subjects to make the transition from propositional knowledge to conceptual understanding. A partial explanation for this failure may be the fact that people are not tabulae rasae when they are being taught new knowledge, but bring along a body of prior ideas and conceptions that are intuitively plausible and have hitherto helped the student to bring order and sense into his or her experiences. The problem is that such prior knowledge is often scientifically incorrect; such pre-existing ideas are therefore known as misconceptions. Frequently occurring misconceptions have been identified in a wide variety of academic fields, including statistics. For example, many novices belief that small, rather than large samples are more representative for a population, that an event is more probable as it becomes more typical (Kahneman, Slovic, & Tversky, 1982), and that a correlation between two variables is indicative of a causal relationship (Estepa & Batanero, 1996; Morris, 1998). The problem with many such misconceptions is that they are not easily substituted by scientifically correct views. For example, Songer and Mintzes (1994), studying the effect of an academic course on cellular respiration on the evolution of knowledge in biology students, found that although the course initially succeeded in replacing naı¨ve conceptions by scientifically correct propositions, in the longer run the students tended to fall back on the earlier misconceptions. They concluded that the older misconceptions form part of a network of interrelated propositions, and unless the scientifically correct propositions are somehow linked to priorly existing knowledge, they will not take root in the overall knowledge structure of the student. Not infrequently, in statistics education new knowledge is presented in a fragmented sort of way, with the result that “…(many students)…may be able to memorize the formulae and the steps to follow in familiar, well defined problems but only seldom appear to get much sense of what the rationale is or how concepts can be applied in new situations” (Garfield & Ahlgren, 1988, p. 46). Since the propositions that are newly taught are not yet linked to each other or to existing

N.J. Broers / Learning and Instruction 12 (2002) 323–344

327

knowledge, students will fail to make proper use of them when faced with complex, less familiar problems. Instead they tend to fall back on the older, naive propositions that are part of the earlier established network of interrelated propositions. For this reason, Cumming and Thomason (1998) stressed the importance of introducing new concepts in statistics from multiple perspectives: “We regard (…) a good conception of understanding as being the ability to express the target idea in a number of different ways, and to show how these relate. If you can describe a concept in words, give a definition, draw a picture, write the formula, explain an application, and see how these different representations link together, then you understand the concept well” (Cumming & Thomason, 1998, p. 948). Apart from being well established in a propositional network, many misconceptions are intuitively plausible and have proved their validity on previous occasions. Often, they constitute overgeneralizations: they become misconceptions because they are used beyond their natural limits of applicability (Smith, di Sessa, & Roschelle, 1993). On the domain of statistics, Mevarech (1983) considered a frequently occurring misconception regarding the computation of weighted averages. He concluded that what he observed was not a “senseless change in mathematical operation, but a semantically meaningful deviation of a correct procedure” (Mevarech, 1983, p. 426). The existence of incorrect prior knowledge may inhibit the integration of a body of new and scientifically correct propositions into a larger network of knowledge but, as Smith, de Sessa, and Roschelle (1993) contend, conversely naive conceptions may sometimes facilitate the development of more mature knowledge: “Interference is a biased assessment of the role of novice conceptions in learning. Though they may be flawed and limited in their applicability, novice conceptions are also refined and reused in expert reasoning. Learning requires the engagement and transformation of productive prior resources, and misconceptions, when taken as mistakes, cannot play that role. Alongside conceptions that appear to interfere with learning are other ideas that can be productively engaged and developed” (Smith, di Sessa, & Roschelle, 1993, p. 153). As Hawkins, Jolliffe, and Glickman (1992) note, in general still “little is known about the statistical reasoning process or about how students assimilate new ideas” (Hawkins, Jolliffe, & Glickman, 1992, p. 99). Whereas prior knowledge may sometimes be an inhibitory factor in the development of new knowledge, other factors may stimulate or further complicate this development. The present study was designed as an attempt to explore reasons for the failure to make the transition from propositional knowledge to conceptual understanding. In contrast to the study of Chi and Bassok (Chi & Bassok, 1989; Chi, Bassok, Lewis, Reimann, & Glaser, 1989), where prime attention was paid to differences in the problem solving behavior of poor and good students, the present study focussed on characteristics of the propositions as well as of the subjects in investigating which factors determine the ease with which individual propositions can be related to each other to form a propositional network for correct conceptual understanding. We created five different multiple choice items on elementary statistics, which all

328

N.J. Broers / Learning and Instruction 12 (2002) 323–344

required conceptual understanding for their correct solution.1 For the purpose of the study, we decided to represent the problem solving process in terms of a few basic operations: (1) selection by the subject of the relevant information units provided in the text of the problem, (2) selection of the necessary factual units from memory, and (3) combination of information units and factual units in a way that permits the logical derivation of the correct solution to the problem. When a subject is confronted with a statistical problem in the form of a multiple choice item, his or her first task is to extract all the relevant information given in the text, in order to represent the problem in such a way that a solution may be found. The individual pieces of relevant information in the text, which may be figures, concepts, statements, etc., we will henceforth refer to as ‘information units’. The findings of Montgomery and Allwood (1978) lead us to suspect that inaccuracy on the part of the subject may result in an incomplete selection of information units and thereby an incorrect problem representation, preventing the finding of the solution to the problem. Such inaccuracy could be interpreted as a deficiency in the problem solving technique, rather than a deficiency in knowledge. But selecting relevant information from the text is unlikely to be solely a function of attentiveness and close reading. Knowing what information in an item is relevant may require propositional background knowledge. For example, an item contains contradictory figures that are presented as the results of a certain statistical technique (e.g. an item states a linear correlation between two variables X and Y of 0.50, and claims that 55% of the variance in Y is explained by X, where one would commonly expect this to be 25%, given the reported correlation). In such a case, the subject is often required to note this contradiction in order to find the correct alternative of the multiple choice item. But this presupposes that the subject possesses the propositional knowledge that allows him to identify the existence of the contradiction. Without this knowledge, the subject will fail to select the relevant information units, without necessarily being inaccurate. So in order to select the relevant information units given in the text of the problem, the subject will frequently need a body of relevant propositional knowledge. Because, as we will show, it is possible to write out this relevant knowledge as a collection of individual statements, we will henceforth refer to this body of relevant knowledge as the collection of relevant ‘factual units’. A factual unit is defined as any elementary piece of knowledge that a subject may retrieve from memory in an attempt to solve the problem with which he or she is confronted. Of course, if a subject lacks the relevant propositional knowledge, he or she cannot answer the item correctly, except by guessing. If the subject does possess the relevant propositional knowledge, he or she may still fail to utilize this knowledge effectively in the search for a 1 The reason why we made use of the multiple choice format is that our intended group of subjects— psychology students of Maastricht University—were accustomed to this type of assessment in the statistics curriculum. By presenting the problems in a familiar format we hoped to facilitate the problem solving activity of our subjects. However, as will become apparent in the Method section, the way we made use of the multiple choice format strongly deviates from its traditional use and actually amounted to an assessment in the form of a loosely structured interview focussing on open ended questions.

N.J. Broers / Learning and Instruction 12 (2002) 323–344

329

solution. This may be due to two deficient operations in the problem solving process: the subject may fail to select the relevant factual units from his or her memory, or he/she may sample the necessary units correctly, but fail to relate the facts to the relevant information units in a way that permits the logical derivation of the solution. Such failure points to a deficiency in the process of reasoning, e.g. the perpetration of logical errors. In this study, we focussed on the following research questions: 1. What is the relative importance of: failure to select relevant information units, failure to select relevant factual units and failure to reason properly in the unsuccessful attempt to solve a statistical problem for which one possesses the required factual knowledge? 2. What factors contribute to a failure to select the relevant factual units? 3. Is the failure to select relevant factual units related to or independent of the failure to reason properly? Following Chi and Bassok, we confronted our subjects with a number of problems, asking them to solve these while thinking aloud (Chi & Bassok, 1989; Chi, Bassok, Lewis, Reimann, & Glaser, 1989). This would permit a detailed analysis of the way subjects make use of propositional knowledge in the course of statistical problem solving.

2. Method 2.1. Subjects Ten subjects (six female, four male) studying psychology volunteered and were paid to participate in this study. Of these, three were freshmen, six were sophomores, and one student was in his third year of study. All subjects were between 20 and 22 years of age and expressed intrinsic motivation to participate in the study. Each participant had a good (though not exceptional) record on the statistics exams covering elementary descriptive and inferential statistics. All subjects had received basic training in descriptive regression analysis. 2.2. Procedure Five multiple choice items on descriptive regression analysis were constructed. Each item contained four alternatives of which only one was correct. The content of the items pertained to course material covered by the elementary statistics course that all students had followed in their first year at university. An example of an item is the following: 앫 In the same study, two variables X and Y correlate rxy=0.50 with each other, whereas the same variable X correlates rxz=0.70 with a third variable Z. Indepen-

330

N.J. Broers / Learning and Instruction 12 (2002) 323–344

dently, two bivariate regression analyses are performed: Y is regressed on X and Z is regressed on X. After analyzing the results it is shown that S 2y⬘/S 2y (the explained variance of Y divided by the total variance of Y) equals 0.55, whereas S 2z⬘/S 2z equals 0.49. Which conclusion may be drawn with certainty? (a) A non-linear model has been used for regression of Y on X (*) (b) A non-linear model has been used for regression of Z on X (c) The prediction of Z based on X is more accurate than the prediction of Y based on X (d) The researcher has made computational errors, for the results are impossible. The star marks the correct alternative.2 A complete list of the items is shown in Appendix A. In advance, a detailed study was made of factual units that were required in order to solve the items. For example, for the item stated above, the necessary factual units are: 앫 r2xy gives the proportion of variance explained in Y based on a linear relationship with X. 앫 S 2y⬘/S 2y gives the proportion of variance explained in Y based on the relationship with X modelled by the regression equation. 앫 The proportion of variance explained in the dependent variable on the basis of a non-linear polynomial relationship with the independent variable X is larger than or equal to the proportion of variance explained in the dependent variable on the basis of a linear relationship with X. 앫 The larger the proportion of variance explained by a regression model, the better the prediction based on that model. Of course, this list of factual units is not exhaustive in the sense that no additional ones could be conceived of. But in the lectures on descriptive regression analysis, factual units such as the four stated above were presented as basic knowledge. More elementary factual units, such as the meaning of a proportion or of variance, were assumed to be already familiar to the students (they were dealt with in more elementary statistics classes preceding the lectures on descriptive regression analysis). So when we present an overview of the factual units pertaining to a given problem, we are actually presenting an overview of those factual units pertaining to the problem that were independently presented and explained during the course on descriptive regression analysis. The total collection of five items required the knowledge of 26 such factual units. In order to assess the subjects’ propositional knowledge of descriptive regression analysis, a questionnaire was developed containing 26 open ended questions per-

2

In this item, ‘non-linear’ refers to a higher degree polynomial, as opposed to a first degree or linear model. Actually, in advanced regression analysis higher degree polynomials are usually referred to as linear models in that they are linear in the parameters.

N.J. Broers / Learning and Instruction 12 (2002) 323–344

331

taining to each of these 26 factual units (see Appendix B for these questions). For example, two such questions read 앫 How does one compute the proportion of variance explained in Y on the basis of the linear correlation with X? 앫 What can one say about the proportion of variance explained in Y on the basis of a non-linear polynomial relationship with X, in comparison to the proportion of variance explained on the basis of a linear relationship with X? Each subject was examined on his or her knowledge of all 26 factual questions. Based on the questions the student could not answer, he or she was given a detailed study assignment in order to acquire the knowledge that was identified as lacking. Approximately two weeks later, the subject was again examined on all 26 questions, to ascertain that he or she had indeed mastered all the facts necessary for finding the solution to the five multiple choice problems. In those few cases where a subject still had difficulty in answering some of the factual questions, the examiner explained the right answer to the subject. Immediately following this re-examination, the subject was asked to complete the first multiple choice problem. The problem was put before him or her on paper and also read out to him or her by the examiner. Subsequently the subject was asked to solve the problem while thinking aloud. After he or she had chosen an alternative as allegedly the right response, he or she was requested to examine each of the remaining alternatives and to explain why these could not be correct. This way, each subject gave an argued response to each of the four alternatives per item. If any one of these four responses was incorrect, the complete discussion of an item was followed by the presentation on paper of all the relevant factual units pertaining to that item. The subject was then requested to use that list of facts for a renewed motivation of the (in)correctness of the four alternatives. After the subject had completely finished his discussion of the item, he or she was presented with the next item, which was dealt with in a similar fashion as the previous one. For all the subjects, the process of problem solving while thinking aloud was later transcribed verbatim. 2.3. Data analysis Apart from the transcripts of the tape recorded think aloud reports, the data consisted of the responses of the subjects to the items. Each item had four different alternatives, and each subject was requested to explain the correctness or incorrectness of each separate alternative. Although the five items together consisted of 20 alternatives, the experimental task yielded 15 independent responses per subject. This was due to the fact that the (in)correctness of some of the alternatives was logically dependent on the (in)correctness of other alternatives in the same item. For instance, the last alternative of item 3 reads ‘The results are impossible’. If true, this logically implies that the other three alternatives cannot possibly be true. Likewise, if either alternatives A, B or C were to be found correct, then it necessarily follows

332

N.J. Broers / Learning and Instruction 12 (2002) 323–344

that alternative D must be incorrect. This implies that the argument of why alternative D is incorrect is the same as the argument of why alternative A is correct. Therefore, this argument constitutes only a single response. Likewise, as can be seen from Appendix A, item 1 yields only one independent response per subject, item 2 and item 4 each four independent responses, and item 5 three (alternative 5D being dependent on the other three alternatives). Altogether therefore, the 10 subjects each gave 15 independent responses, yielding a total of 150 independent responses that could be used for analyses. Each independent response of a subject could be scored as either correct or erroneous. A response was qualified as correct when the subject had properly argued why an alternative was either true or false. A response was marked as erroneous when the subject had failed to provide a correct argument.

3. Results Table 1 gives an overview of the mean, standard deviation and range of the number of correct responses to the 15 independent alternatives. On average, the subjects gave nine correct answers. As the range and standard deviation show, however, there was considerable variation in the ability of the subjects to solve the items. 3.1. Selection of information units and factual units When subjects possess all the relevant propositional or factual knowledge needed to solve a statistical problem, their failure to do so is likely to be connected to a failure to select relevant information units (we will henceforth use the abbreviation Ir) or a failure to select relevant factual units (for which we will henceforth use the abbreviation Fr), or both. In 59 cases subjects gave an erroneous response. To gain insight into the role of failure to select Ir and/or Fr in erroneous problem solving activity, a table was constructed, cross classifying the 59 erroneous responses as due to a failure to select Ir and/or Fr (see Table 2). First, a note on the lower left cell. Of the seven errors contained in this cell, six reflected a failure to registrate all the information provided in the text. This suggests that these errors reflect inaccuracy on the part of the subject. However, four of these six errors pertained to a similar information unit in item 1: S 2y⬘ is smaller than S 2e. Curiously enough, these four subjects had registered the remaining information correctly, retrieved all the relevant facts and reasoned properly, yet they all gave the Table 1 Number of problems solved correctly by the subjects Minimum

Maximum

Mean

Std. dev.

4

13

9.10

3.14

N.J. Broers / Learning and Instruction 12 (2002) 323–344

333

Table 2 Classification of errors as related to (in)adequate selection of Fr and to (in)adequate selection of Ir Ir selected Fr selected

No

Yes

Total

No Yes Total

12 7 19

35 5 40

47 12 59

wrong answer due to their failure to note the information unit mentioned above. It seems unwarranted to classify these four cases as instances of inaccuracy, for at no other stage during their problem solving activity did any of these subjects betray any habitual inattentiveness or inaccurate dealing with the texts of the problems. Clearly, the particular wording of problem 1 provoked inaccurate registration of the text. Further confirmation of this interpretation is given by the fact that this same error frequently occurred in another (as yet unpublished) research in which the same five items were used. Table 2 makes clear that the majority of errors are related to a failure to retrieve the necessary facts. Additional analyses revealed further information on the nature of this failure. In particular, four factors were identified that are conducive to the inability to select Fr: the dominant presence of (one or more) non-existent fact(s), the difficulty of logically combining the facts in order to reach a conclusion, the level of abstraction of the facts, and the recency of acquisition of the facts. 3.2. Use of non-existent facts Confronted with an item, subjects would often start to search for key facts that would guide them to the solution. For instance, confronted with alternative C of item 2, a number of subjects argued that working with a linear model implies that use has been made of the least squares criterion for fitting the regression line. This is an erroneous assumption that guided their search for the solution of the problem. They correctly remembered that if the least squares criterion has been met, the error variation will have taken up its smallest possible value. Combining this valid knowledge with the aforementioned erroneous assumption, they concluded that the reported error variation could indeed not be any smaller than 15, as asserted in alternative C of item 2. Use of the erroneous fact prevented the subjects from searching memory for the appropriate facts that could have led them to the solution. Another example of misguidance by an erroneous fact occurred in the context of item 3. Confronted with this item, a subject started with asserting the non-existing fact that when the correlation coefficient rxy is reported, this automatically implies that use has been made of a linear regression model. On the basis of this supposed fact, she concluded that alternative A had to be wrong. Once again, the dominance of the erroneous fact prevented the subject from searching for the relevant facts.

334

N.J. Broers / Learning and Instruction 12 (2002) 323–344

In total, there were 20 instances in which the use of non-existing facts played a role in preventing the subject to reach a correct conclusion to a problem. Like the examples discussed above, they form misconceptions that are part of the prior existing knowledge of the subject. 3.3. Differences in logical reasoning ability The five items were constructed so as to demand conceptual understanding for their solution. Simple evocation of the relevant I and F was not enough, the relevant facts had to be logically combined with each other and with the relevant information units, in order to solve the problems. When we studied the protocols, only four cases emerged where an erroneous response was directly due to a failure to reason in a logically consistent way. For instance, in the context of problem 4D a subject claimed to know for certain that the correlation between two variables would not change after transformation into standard scores. She felt sure that the proportion of variance explained, which she knew equalled the square of the correlation, would not change after z-transformation. However, then she reflected that the proportion of variance could also be expressed as the ratio between variance in Y explained by X and total variance in Y. She had formerly decided that both of these variances would change after transformation into z-scores, and although she had claimed to be 100% sure that r2 would not change, she now believed that the proportion of variance explained would probably change as the numerator and denominator of the aforementioned ratio changed. She was unable to come to the conclusion that although S 2y⬘ (i.e. variance in Y explained by X) and S 2y will change, their ratio need not (and in fact does not) change. To gain further insight into the extent to which our subjects differed in their ability to logically combine the facts, subjects who had given an erroneous response were subsequently presented with the necessary facts needed to construct the solution. They were then requested to try to reason about the problem, with the necessary facts (Fr) at hand. With the facts at their disposal, failure to come to a correct solution could no longer be due to an inability to select the relevant F, but had to be related to a failure to combine the relevant I and F logically. Nine out of ten subjects were at any one time presented with a collection of necessary facts. Table 3 lists the number of times individual subjects were presented with such a list, and the number of successful outcomes into which this resulted. The results give some indication of differences in logical reasoning ability among subjects. For example, on seven occasions at which subject S5 gave an incorrect solution, he was presented with a list of all the relevant facts, necessary for the construction of the solution. On the basis of this list, he succeeded in correctly solving six out of these seven problems. On the other hand, subject S9 was on 11 occasions at which he had given an incorrect solution provided with a list of necessary factual units. In only four of these instances did he succeed in constructing a correct solution of the problem presented. So, after the relevant F were presented, there were notable differences in the ability of subjects to handle the problems. At

N.J. Broers / Learning and Instruction 12 (2002) 323–344

335

Table 3 Number of problems correctly solved after presentation of Fr Subject

No. of times a subject was presented with the relevant factual units

No. of times a subject Percent of correct correctly solved the solutions problem after presentation of relevant factual units

S1 S2 S3 S4 S5 S6 S7 S8 S9

1 4 5 3 7 1 3 2 11

1 1 0 3 6 1 0 2 4

100% 25% 0% 100% 86% 100% 0% 100% 36%

this stage, these differences appear related to variation in the ability to reason logically. 3.4. Abstract vs verbal factual units Apart from the complexity of the problem, a failure to retrieve necessary F may also be related to the nature of those factual units. In particular, it seems likely that facts that are highly abstract (e.g. a mathematical relationship like r2xy=S 2y⬘/S 2y ) are more difficult to retrieve than facts that come as verbal statements (e.g. ‘A z-transformation is a linear transformation’). Of the 26 factual statements that the subjects knew or were required to study before attempting to solve the multiple choice problems, 19 pertained to propositions that could be classified as primarily verbal statements (although these could involve mathematical symbols, e.g. ‘rxy is a measure that indicates the strength of linear relationship between two variables X and Y’), and only seven propositions were related to equations or computational instructions (e.g. ‘S 2y =S 2y⬘+S 2e ’). Solving the total collection of problems required the subjects to make use of each of the 26 facts. On average, each subject failed to retrieve six different facts. That is to say, when a subject has failed to retrieve the fact ‘S 2y =S 2y⬘+S 2e ’ on three different occasions, and the fact ‘r2xy=S 2y⬘/S 2y ’ on five different occasions, this means that this subject failed to retrieve two different facts. Table 4 relates the abstract vs verbal nature of the facts to whether or not they were retrieved in the course of problem solving. It tells us that of the six different facts that subject 1 failed to retrieve, two (or 33%) belonged to the abstract category. Of the total list of 26 different factual units, only 27% are abstract. As we can see, all subjects had relatively more difficulty in retrieving abstract factual units.

336

N.J. Broers / Learning and Instruction 12 (2002) 323–344

Table 4 Number of different abstract and verbal factual units that were not retrieved, per subjecta Subject

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

Abstract facts

Verbal facts

non-retrieved

non-retrieved

Total no of facts non-retrieved

Freq

Perc

Freq

Perc

Frequency

2 3 3 3 7 3 3 2 6 3

33% 60% 30% 43% 58% 100% 100% 50% 60% 100%

4 2 7 4 5 0 0 2 4 0

67% 40% 70% 57% 42% 0% 0% 50% 40% 0%

6 5 10 7 12 3 3 4 10 3

a The total list of 26 factual units consisted of 7 abstract factual units (27%) and 19 verbal factual units (73%).

3.5. Recency of acquired knowledge A third factor that may influence the probability that a given fact will be retrieved by a subject, concerns the recency of its acquisition. In this study, subjects were first questioned on their knowledge of the 26 facts that they would subsequently need to solve the multiple choice problems. When the subjects revealed a lack of knowledge on certain facts and propositions, they received a specific assignment aimed at closing the gap of knowledge. Before presenting them with the multiple choice problems several weeks later, it was checked whether the subjects had indeed acquired the formerly missing knowledge after completing the study assignment. We may expect that subjects had less difficulty in using those factual units that were shown to be accessible from long term memory during the first session of the study, than in using factual units that were acquired only recently—i.e. those facts that had only been learned sometime between the first and the second session. Table 5 makes it possible to examine whether this was indeed the case. For each subject, two subsets of facts were considered: the subset of facts that he or she demonstrably knew during the first session, and a second subset containing those facts that he or she did not know during the first session (but had acquired during the second). For each subject, we subsequently determined the number of facts belonging to the first subset that he or she did not retrieve during the course of problem solving, relative to the total number of facts belonging to the first subset. Likewise, we determined the number of facts belonging to the second subset that he or she did not retrieve, relative to the total number of facts belonging to the second subset. If recency of acquisition of the facts is indeed a determinant of the likelihood that a fact will be retrieved, with the more recently learned facts less

N.J. Broers / Learning and Instruction 12 (2002) 323–344

337

Table 5 Proportion of previously known facts that were retrieved minus proportion of previously unknown facts that were retrieved (see text for explanation) Subject

Difference of proportion

S1 S2 S3 S4 S5 S6 S7 S8 S9 S10

0.07 0.44 0.15 0.52 0.34 0.16 0.04 0.55 0.52 0.03

likely to be retrieved, than the proportion of retrieved facts belonging to the first subset minus the proportion of retrieved facts belonging to the second subset will yield a positive difference. Although due to the small number of subjects no significance tests could be meaningfully applied, the results show either substantial positive or near zero differences.

4. Discussion Why do people fail to solve a statistical problem for which they possess the relevant factual knowledge? In this study we focussed on three operations in the process of problem solving: the selection of the relevant information units provided in the text of the problem, the selection of necessary factual units from memory, and the logical combination of relevant information units and factual units in order to derive the solution. The results showed that failure to select Ir, failure to select Fr, and failure to reason properly each play an important part in an unsuccessful attempt at problem solving. The 10 subjects together completed 150 problem solving tasks, of which 59 ended unsuccessfully. In 19 out of 59 times a subject failed to select the relevant information units. In 12 of these 19 cases, this failure corresponded with an equal failure to retrieve the relevant facts. These subjects failed to come to a proper representation of the problem. On the other hand, on seven occasions subjects did produce the relevant facts, but still failed to identify all the relevant information units. In four of these cases, this failure could be ascribed to inaccuracy on the part of the subject (that is, he or she had not read the text of the problem precisely enough). However, these four cases were all related to an omission of the same information unit in item 1, and the subjects who made this particular error did not reflect imprecision in their confrontation with any of the other problems. Therefore inaccuracy, in the sense of an imprecise registration of the text of the problem, did not really occur in our study.

338

N.J. Broers / Learning and Instruction 12 (2002) 323–344

This outcome differs from the findings of Montgomery and Allwood (1978) and Allwood and Montgomery (1981). Although in their studies deficiencies in knowledge were shown to play a far bigger role in making errors in the context of statistical problem solving than inaccuracy on the part of the subject, such inaccuracy was found to play a bigger role than we observed in this study. This may be connected to the different type of problems used by Allwood and Montgomery, who presented open ended questions requiring a substantial amount of computation. Such a type of problem will allow inaccuracy and imprecision to surface more easily and prominently than would the multiple choice items we used. In 47 out of these 59 cases, subjects failed to retrieve some of the facts that were necessary to find the solution. Four factors that are conducive to such failure are the interference of non-existing facts, the level of abstraction of the facts (in our study we simply distinguished between abstract equations and verbal statements), the recency of acquisition of the facts, and the ability to reason logically. We found that in 20 of the 59 erroneous responses, the error was due to the interference of non-existing facts. Faced with a complex problem that demanded the combined consideration of various propositions, many subjects tended instead to fall back on misconceptions regarding the domain of the problem. We interpret this finding as indicative for the lack of interrelationships between the various propositions in the cognitive organization of the subjects. They demonstrated knowledge of the individual propositions, but obviously this knowledge had not evolved into complex interrelationships. No cognitive schemata of greater generality and abstraction, based on these propositions, had as yet been formed. As a result, the problems we presented to our subjects could not be solved by them by evoking complex schemata in which the various propositions are brought into relationship with each other. Instead, the subjects had to relate the propositions themselves by a complex process of selecting the relevant Ir and Fr, and logically combining all these cognitive units in a manner appropriate for solving the problem. This makes it easy for misconceptions to step into the reasoning process. The misconceptions bring immediate structure into the problem and this obviates the need for the complex reasoning process described above. This finding is reminiscent of that of Songer and Mintzes (1994), who found that newly acquired ideas that were not yet well organized in a cognitive framework, tended after a while to be replaced by the older misconceptions that were still firmly embedded in the knowledge base of the students. Based on our personal experience, we believe that most psychology students do not tend to think mathematically but in terms of concrete verbal theories of reality. This seems a likely explanation for the fact that abstract facts are more difficult to remember than verbal propositions. Whereas science and maths students habitually work on the development of mental schemata consisting of mathematical relationships, most psychology and social science students generally theorize in terms of verbal concepts and relationships, making it harder for them to retain relationships that are purely abstract in nature. Such abstract relationships do not easily fit into their existing cognitive schemata, and hence tend remain isolated rather than become integrated. A similar interpretation explains why recency of acquisition determines the likeli-

N.J. Broers / Learning and Instruction 12 (2002) 323–344

339

hood that factual units will be retrieved. Subjects were less likely to retrieve those factual units that they had learned only in the past few weeks. Unlike the facts that were already accessible in long term memory when the study began, the recently acquired knowledge is not yet fully integrated into the previously existing knowledge structure. The newly acquired facts exist as more or less isolated propositions, and only after repeated exercise and confrontation with these new factual units will the subject be able to see them as part of a larger network of interrelated facts and concepts. Only when that later stage of development is reached, will those factual units be readily accessible in memory. The fourth factor contributing to faulty selection of Fr concerns the students’ failures in logical reasoning. Interestingly, this weakness of some students did not become manifest in their initial problem solving attempts. Studying the protocols, only four cases emerged where an erroneous conclusion was directly related to the failure to reason in a logically consistent way. However, when confronted with the relevant factual units pertaining to a particular problem, differences in the ability to logically combine these facts in order to produce the solution to the problem were revealed. We surmise that students who have difficulties with logical reasoning will have a hard task in selecting the relevant facts pertaining to a complex problem. The complexity of the problem shows up in the fact that multiple information units have to be related to multiple factual units. Logically combining some information units with some tentative factual units results in the selection of yet more factual units, eventually permitting the logical derivation of the conclusion. Students who are weak at logical reasoning will not easily succeed at fully selecting all the relevant information units and factual units. Further research will have to reveal the extent to which a failure to reason logically is responsible for a failure to select Ir and Fr. Comparing the role of failure to select all the relevant I with the importance of failure to select all the relevant F in the process of unsuccessful problem solving, the results of our study indicate that most of the unsuccessful attempts at problem solving derive directly or indirectly from the inability to consider the relevant facts. Once the relevant I and the relevant F have been successfully selected by the subject, a successful outcome of the problem solving process was almost guaranteed. Only few instances remained, where a subject failed to come up with the right solution after successful selection. As we noted above, deficiencies in the process of logical reasoning did contribute to an unsuccessful outcome, but mainly indirectly by preventing the subject from properly selecting the relevant I and F. For the practice of statistics education, the results of our study suggest three main recommendations. First, it is obvious that knowledge of a complex field like statistics needs time to mature. Novel ideas and concepts will not easily integrate into complex schemata, but need a lot of effort and repetitive exercise to do so. For any particular domain of statistical knowledge, such as that of descriptive regression analysis in our study, it is important to identify popular misconceptions that are likely to interfere in this maturation process. Since a third of all the errors made by our subjects involved the use of invalid concepts and ideas, it is obvious that priorly existing naı¨ve knowledge can be a hindering factor in the development of more complex knowledge.

340

N.J. Broers / Learning and Instruction 12 (2002) 323–344

Insofar as the erroneous facts used by subjects are strictly idiosyncratic in nature, there is not much that can be done about it in the form of a general approach. But as shown by the examples that we cited in Section 3, many misconceptions are widely held and can therefore be specifically addressed. It seems to us important to make an effort of charting these common misconceptions, pertaining to a specific domain of statistical theory. The way they should subsequently be dealt with—aiming at their replacement by more sound ideas via the repeated exposition of their inappropriateness or aiming at their refinement into more productive forms of knowledge (following Smith, di Sessa, & Roschelle, 1993), is dependent on the specific nature of the misconceptions. But only by knowing which common misconceptions tend to intervene in the development of interrelationships between the propositions we wish to convey, and by knowing how they are likely to intervene in this developmental process, will we be able to help students realize their learning objectives. Second, although statistics courses aimed at psychology and social science students primarily focus on the conceptual ideas and on the valuable use that can be made of statistics, there are a number of formulae and mathematical relationships that are important for a proper understanding of the theory. We found that in particular these highly abstract propositions are difficult to embed in the overall knowledge structure. Following Cumming and Thomason (1998), we would therefore recommend that such abstract propositions are not presented in isolation, as facts onto themselves, but always in relationship to more concrete knowledge items such as a graphical display of the abstract idea, a conceptual explanation of the abstract proposition (most mathematical formulae in elementary statistics can be fully explained in words), and a concrete application of it. By presenting the most abstract ideas already embedded in a network of more accessible ideas, they will be more readily understood and retrieved. Third and finally, we believe that a proper training of conceptual understanding of statistics amounts to stimulating the students to develop multiple interrelationships between the various propositions that we teach or, in other words, to helping them building gradually more complex cognitive schemata. A technique that may help students to integrate their knowledge, and with which we have had some successful experience, is by providing the students with an exhaustive list of important mathematical (as well verbal) relationships that pertain to the statistical subject they need to master. Subsequently, problems can be presented with the request to the students to specify which of the concepts, propositions and mathematical relationships pertain to the particular problem at hand. Using the relevant concepts and propositions, they are than asked to construct a logical chain of argument, showing what the correct answer to the problem should be. By repeatedly confronting subjects with such an assignment, the important abstract relationships gradually become familiar and recognizable as belonging to a larger network of concepts and propositions. Because such a didactic approach is deliberately designed to stimulate conceptual understanding on the part of the subjects, the problems suggested above should be specifically designed to evoke a mental search for relationships between statistical concepts and ideas. This means that multiple choice items of the type we have used in this study are likely to be more suitable for this purpose than open ended questions

N.J. Broers / Learning and Instruction 12 (2002) 323–344

341

requiring the subject to concentrate on computation. In a forthcoming study, we aim to put the effectiveness of this method of learning and instruction to an experimental test.

Appendix A. The five multiple choice problems used 1. We have collected data on two variables X and Y. We decide to regress Y linearly on X. The results show that the error variance Se2 equals 1 and is smaller than Sy⬘2, the amount of variance explained by X, and Sy2, the total variance in Y. Are these results possible, or has there been some sort of an error in the data analysis? (a) No, these results cannot possibly be right (b) Yes, these results are possible when variable X explains less than 50% of the variance in Y (c) Yes, these results are possible when variable X explains more than 50% of the variance in Y (*) (d) Yes, these results are possible irrespective of the percentage of variance explained in Y by X 2. In a study, 30 subjects obtained a score on two variables X and Y. Regression of Y on X yields the following linear model: Y⬘=0.90X+1.5. In addition, the analysis shows the mean on Y to equal 6.3 and the variance to equal 3. The correlation between X and Y, expressed in Pearson’s rxy, equals 0.91. The variance of the predicted Y-values, Sy⬘2, equals 2.5 and the sum of squares ⌺(y⫺y⬘)2 equals 15. Which of the following statements is incorrect? (a) In conjunction with the other data reviewed above, the value of Sy⬘2 suggests that the linear model describes the relationship between X and Y fairly well (b) The value of the correlation coefficient rxy suggests that the chosen regression model allows for an accurate prediction of Y, based on X (c) The data suggest that no other regression model will yield a value of ⌺(y⫺y⬘)2 smaller than 15 (d) The value of the regression coefficient shows that Y can be accurately predicted from X (*) 3. In the same study, two variables X and Y correlate 0.50 with each other, whereas the same variable X correlates 0.70 with a third variable Z. Independently, two bivariate regression analyses are performed: Y is regressed on X and Z is regressed on X. After analyzing the results it is shown that Sy⬘2/Sy2 (the explained variance of Y divided by the total variance of Y) equals 0.55, whereas S 2z⬘/S 2z equals 0.49. Which conclusion may be drawn with certainty? (a) A non-linear model has been used for regression of Y on X (*) (b) A non-linear model has been used for regression of Z on X (c) The prediction of Z based on X is more accurate than the prediction of Y based on X (d) The researcher has made computational errors, for the results are impossible 4. We regress Y on X using a linear equation Y⬘=bX+a. Suppose that we convert X and Y into standard scores Zx and Zy, and then regress Zy linearly on Zx. Which

342

N.J. Broers / Learning and Instruction 12 (2002) 323–344

of the following statistical quantities will, in comparison with the original regression of Y on X, have remained unchanged? (a) The value of the error variance (b) The intercept of the regression line (c) The slope of the regression line (d) The proportion of variance in Y explained by X (*) 5. Two variables X and Y have a correlation of rxy=0.80. Amongst other results, regression of Y on X yields the following: S 2e /S 2y =0.50. Which of the following conclusions can now be drawn? (a) This results indicates that use was made of a non-linear regression model (b) The regression equation that was used does not satisfy the least squares criterion (*) (c) It is possible to come to a better prediction of Y based on X, but only if a non-linear regression model is used (d) Each of the alternatives listed above is correct

Appendix B. The 26 open ended questions for the assessment of factual knowledge 1. Sy2 is a measure that pertains to the spread of Y-scores about the mean of Y. To what sort of spread does Sy⬘2 pertain to? Clarify your answer with help of a scattergram. 2. Sy2 is a measure that pertains to the spread of Y-scores about the mean of Y. To what sort of spread does Se2 pertain to? Clarify your answer with help of a scattergram. 3. What do we mean by ‘regression of Y on X’? Which is the dependent variable? 4. Suppose we compute the linear correlation between two variables X and Y. Will the value of this correlation change if we were to compute it on the basis of standardized scores Zx and Zy? 5. How do we compute the proportion of variance explained in Y by X, using the linear correlation coefficient rxy? 6. How are Sy2, Sy⬘2 and Se2 related to each other? How can we compute one of these quantities on the basis of the other two? 7. What do we mean by a higher-order polynomial regression model? 8. What is the meaning of the intercept a in the regression equation Y⬘=bX+a? 9. How can we compute the proportion of variance explained in Y by X, on the basis of Sy2 and Sy⬘2? 10. What is rxy? What sort of association does this measure quantify? 11. What is the maximum value of Se2? 12. Can Se2 take up a maximum value greater than 1? Why? 13. What is the meaning of the regression coefficient b in the regression equation Y⬘=bX+a? 14. What can we say about the proportion of variance explained in Y by X on the

N.J. Broers / Learning and Instruction 12 (2002) 323–344

15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26.

343

basis of a higher-order polynomial regression model, in comparison with the proportion of variance explained on the basis of a linear model? What quantity in the regression equation expresses the slope of the regression line? What criterion is commonly used in regression analysis to determine the best fitting regression line in a scatterplot? How can we compute the proportion of variance in Y, left unexplained by X, on the basis of Sy2 and Se2? How can we tell from a scatterplot whether a linear regression model will allow us to make an accurate prediction of Y on the basis of X? In the context of regression analysis, how does rxy reflect the spread of points about the regression line? What does the proportion of variance explained tell us about the accuracy of our predictions of Y on the basis of X? What happens to the mean and the standard deviation after transformation of Yscores into standard scores Zy? What sort of a transformation (linear, quadratic, logarithmic, etc.) is a Z-transformation? Can Sy2 take up a maximum value greater than 1? Why? What conclusion may be drawn when it appears (after doing regression analysis) that r2xy⬎S 2y⬘/S 2y ? What does the least squares criterion signify, how is it used? What does the regression coefficient b tell us on the spread of points about the regression line?

References Allwood, C. M. (1984). Error detection in statistical problem solving. Cognitive Science, 8, 413–437. Allwood, C. M. (1990). On the relation between justification of solution method and correctness of solution in statistical problem solving. Scandinavian Journal of Psychology, 31, 181–190. Allwood, C. M., & Montgomery, H. (1981). Knowledge and technique in statistical problem solving. European Journal of Science Education, 3 (4), 431–450. Allwood, C. M., & Montgomery, H. (1982). Detection of errors in statistical problem solving. Scandinavian Journal of Psychology, 23, 131–139. Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: MIT Press. Bromage, B. K., & Mayer, R. E. (1981). Relationship between what is remembered and creative problemsolving performance in science learning. Journal of Educational Psychology, 73 (4), 451–461. Chi, M. T. H., & Bassok, M. (1989). Learning from examples via self explanations. In L. B. Resnick, Knowing, learning and instruction: essays in honor of Robert Glaser (pp. 251–282). Hillsdale, NJ: Erlbaum. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: how students study and use examples in learning to solve problems. Cognitive Science, 18, 145–182. Cumming, G., & Thomason, N. (1998). StatPlay: Multimedia for statistical understanding. In L. PereiraMendoza, L. Seu Kea, T. Wee Kee, & W. Wong, Statistical education—expanding the network. Proceedings of the 5th International Conference on Teaching of Statistics, 21–26 June 1998, Singapore (Vol. 2) (pp. 947–952). International Statistical Institute.

344

N.J. Broers / Learning and Instruction 12 (2002) 323–344

Estepa, A., & Batanero, C. (1996). Judgments of correlation in scatterplots: Students’ intuitive strategies and preconceptions. Hiroshima Journal of Mathematics Education, 4, 25–41. Garfield, J., & Ahlgren, A. (1988). Difficulties in learning basic concepts in probability and statistics: implications for research. Journal for Research in Mathematics Education, 19 (1), 44–63. Hawkins, A., Jolliffe, F., & Glickman, L. (1992). Teaching statistical concepts. London: Longman. Hiebert, J., & Lefevre, P. (1986). Conceptual and procedural knowledge in mathematics: an introductory analysis. In J. Hiebert, Conceptual and procedural knowledge: the case of mathematics (pp. 1–27). Hillsdale, NJ: Erlbaum. Hubbard, R. (1997). Assessment and the process of learning statistics. Journal of Statistics Education, 5 (1) online. Huberty, C. J., Dresden, J., & Bak, B. (1993). Relations among dimensions of statistical knowledge. Educational and Psychological Measurement, 53, 523–532. Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press. Lampert, M. (1986). Knowing, doing, and teaching multiplication. Cognition and Instruction, 3, 305–342. Marshall, S. (1995). Schemas in problem solving. Cambridge: Cambridge University Press. Mevarech, Z. R. (1983). A deep structure model of students’ statistical misconceptions. Educational Studies in Mathematics, 14, 415–429. Montgomery, H., & Allwood, C. M. (1978). On the subjective representation of statistical problems. Scandinavian Journal of Educational Research, 22, 107–127. Morris, E. J. (1998). Link: The principled design of a computer-assisted learning program for correlation. In L. Pereira-Mendoza, L. Seu Kea, T. Wee Kee, & W. Wong, Statistical education—expanding the network. Proceedings of the 5th International Conference on Teaching of Statistics, 21–26 June 1998, Singapore (Vol. 2) (pp. 947–952). International Statistical Institute. Smith, J. P., Di Sessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: a constructivist analysis of knowledge in transition. The Journal of the Learning Sciences, 3 (2), 115–163. Songer, C. J., & Mintzes, J. J. (1994). Understanding cellular respiration: an analysis of conceptual change in college biology. Journal of Research in Science Teaching, 31 (6), 621–637.