Intelligence, dual coding theory, and the brain

Intelligence 47 (2014) 141–158

Contents lists available at ScienceDirect

Intelligence

Intelligence, dual coding theory, and the brain Allan Paivio ⁎ Western University, Department of Psychology, Social Science Centre, London, Ontario N6A 5C2, Canada

a r t i c l e

i n f o

Article history: Received 25 November 2013 Received in revised form 2 September 2014 Accepted 3 September 2014 Available online xxxx Keywords: IQ theories IQ tests Conceptual/empirical flaw DCT a unified theory IQ neuroscience

a b s t r a c t The distinction between verbal and nonverbal cognitive abilities has been a defining feature of psychometric theories of intelligence for the past century. Despite their popularity, however, these theories have not included functional connections between verbal and nonverbal systems that are necessary if they are to explain performance in intellectual tasks involving interactions between language and nonverbal knowledge. This functional gap limits the capacity of psychometric theories to explain and predict fundamental aspects of individual differences in cognitive abilities that have long been studied experimentally. This article summarizes the history, nature, and possible causes of the problem, and then concludes with a neuroscientificallyenhanced, multimodal dual coding approach to intelligence that focuses on the synergistic interactivity of verbal and nonverbal systems. © 2014 Elsevier Inc. All rights reserved.

Contents 1. 2.

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . Historical views on intelligence . . . . . . . . . . . . . . . . . . 2.1. Nonverbal imagery epoch . . . . . . . . . . . . . . . . . 2.2. Verbal dominance epoch . . . . . . . . . . . . . . . . . 2.3. Verbal and nonverbal solitudes . . . . . . . . . . . . . . . 3. Dual coding theory and intelligence . . . . . . . . . . . . . . . . 3.1. DCT representational structures and processes . . . . . . . . 3.1.1. Logogens and imagens . . . . . . . . . . . . . . 3.1.2. Representational connections and activation processes 3.2. DCT behavioral analysis of intelligence theories and tests . . . 3.3. Real life referential processing . . . . . . . . . . . . . . . 3.4. Psychometric tests of referential processing . . . . . . . . . 3.5. Individual differences in referential ability . . . . . . . . . . 3.6. Why the neglect of referential abilities? . . . . . . . . . . . 3.7. Computational modeling and DCT . . . . . . . . . . . . . 4. Brain correlates of intelligence and DCT . . . . . . . . . . . . . . 4.1. General intelligence and the brain . . . . . . . . . . . . . 4.2. Neural correlates of DCT . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

⁎ Tel.: +1 519 679 8499 (home). E-mail address: [email protected].

http://dx.doi.org/10.1016/j.intell.2014.09.002 0160-2896/© 2014 Elsevier Inc. All rights reserved.

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

141 142 142 143 144 145 146 146 146 147 148 149 149 150 151 152 152 153 156

142

A. Paivio / Intelligence 47 (2014) 141–158

1. Introduction This article provides arguments and evidence that functional interconnections between nonverbal and verbal cognitive systems are essential to human intelligence and that such interconnections have been neglected historically and in current theories and tests of intelligence. Such interconnections are, however, well-supported features of dual coding theory (DCT). The contrast is qualified by the traditional reliance of intelligence theories on correlational analyses of individual differences in cognitive abilities whereas DCT has emphasized experimental evidence. This methodological difference was the focus of Chronbach's (1957) classic article on the two disciplines of scientific psychology. He found a rapprochement in interactions of individual difference and situational variables that were beginning to be studied. Such interactions are emphasized in DCT research. Also relevant are contrasts between structural and functional definitions of intelligence. Cognitive abilities expert John B. Carroll was referring to factorial structure of abilities when he wrote that the concept of intelligence is “an inexact, unanalyzed popular concept that has no scientific status unless it is restated to refer to the abilities that compose it” (Carroll, 1993, p. 627). Intelligence theorist Robert J. Sternberg focused additionally on adaptive functions, stating that “Intelligence comprises the mental abilities necessary for adaptation to, as well as shaping and selection of, any environmental context” (Sternberg, 1997, p. 1030). Dual coding theory capitalized similarly on the distinction between structural availability and effective use of multimodal mental representations in various tasks (Paivio, 1971, Chap. 11, 1986, Chap. 9; Sadoski & Paivio, 2013). The available representational structures are specialized for dealing with verbal and nonverbal stimuli and responses. Their adaptive functions include memory, evaluation, anticipation, motivation (and emotion), problem solving, and communication (Paivio, 2007, Chap. 4). Memory in general dominates all other functions because they depend on it — we must remember the past in order to evaluate the survival value of objects and events in the present, anticipate future events, satisfy our needs, solve problems that arise in connection with all of these, and empower language. All types of memory abilities are prominent in both DCT and standard intelligence theories, with procedural and working memory being especially notable because survival ultimately depends on adaptive behavior resulting from them. This article repeatedly emphasizes the neglect of referential processing in historical and modern approaches to intelligence. The neglect is puzzling because referential activities abound in everyday life. Naming objects or imaging to their names begin early during language acquisition and are subsequently elaborated in more complex referential relations between language and perceptual scenes. Visual referents dominate in such examples but referential processing occurs in all modalities— auditory, olfactory, haptic, and motor. The details are reviewed later in the context of the relevant structural and processing assumptions of DCT, which began and evolved to deal with issues related to memory and intelligence that originated in antiquity. The following sections accordingly summarize the major historical influences that eventually came together as the synergistically interactive DCT that is applied in later sections to the analysis of intelligence with a specific focus on

the nonverbal–verbal referential link that has eluded explanations in factor-analytic theories. The review also touches on relevant ancient issues that could be clarified by DCT, but a full retrospective re-analysis of that kind is beyond the scope of this article. 2. Historical views on intelligence Three overlapping epochs emphasized nonverbal imagery, verbal processes, and their joint operation as separate systems in intellectual tasks. Historian Frances Yates (1966) focused on the nonverbal and verbal epochs in her classic analysis of the impact of the art of memory on all areas of intellectual activity over centuries (summarized from the DCT perspective in Paivio, 1971, Chap. 6; Sadoski & Paivio, 2013, Chap. 2). Remarkably, the historians and expositors of the art did not mention the possible interactive role of nonverbal imagery and the verbal systems, or notice that it was absent in the historical approaches to memory and intelligence. 2.1. Nonverbal imagery epoch Intellectual abilities were interpreted in the western world in terms of memory-related nonverbal imagery for 2000 years. These interpretations originated from Greek anthropomorphic religion that included a memory goddess and muses that presided over arts and sciences. The mythical imagery connection to memory became reality when the Greek lyrical poet, Simonides (c. 500 BCE), invented a memory technique in which ideas and things were remembered by visualizing them in ordered places (loci) such as the rooms of a house. The technique first became systematically associated with rhetoric and religious practices and concomitantly began a profound influence on the arts, education, and science that surged later during the Middle Ages in Europe despite being “a clear case of a marginal subject, not recognized as belonging to any of the normal disciplines, having been omitted because it was no one's business. And yet it has turned out to be everyone's business” (Yates, 1966, p. 389). The first detailed descriptions of the method of loci appeared in three Latin works on rhetoric around the beginning of the Christian era by the anonymous author of Rhetorica Ad Herennium (c.86–82 BCE), Cicero (106–43 BCE), and Quintilian (30–96 AD). The method was said to involve “artificial memory” in contrast to spontaneous “natural memory” that occurred simultaneously with thought. These sources included instructions on how to select natural places or construct imagined ones along with effective images of things, thus resembling modern meta-memory procedures that are designed to increase knowledge about memory (e.g., Lima-Silva, Ordonez, dos Santos, et al., 2010). An influential imaginary source of memory loci was introduced by Metrodorus of Scepsis (c. 145–70 BCE), a Greek teacher of rhetoric who was known for the excellence of his memory and was credited with perfecting the method of loci invented by Simonides. His innovation involved the use of the signs of the zodiac as memory places. Astrologers had divided each of the 12 signs of the zodiac into 30 “decans” associated with decan figures. These provided 360 numbered places to which Metrodorus could mentally attach things to be remembered. Here the “places” have become numbered zodiacal names (Ares, Pisces, etc.) and their picture images. Such memory

A. Paivio / Intelligence 47 (2014) 141–158

places served as retrieval cues that subsequently re-emerged as various kinds of memory pegs or hooks in mnemonists' tool kits up to the present. Examples are given in a later section. General memory treatises proliferated in manuscript and printed form throughout the Middle Ages. The Phoenix by Peter of Ravenna, first published at Venice in 1491 and followed by many editions in many countries, became the best known of all the memory textbooks and remained influential for over two centuries. Peter himself demonstrated prodigious memory power using variants of the method of loci. He recommended forming memory places during one's travels as he did, so that he had more than one hundred thousand loci that he used to surpass anyone in knowledge of scriptures and the law. For example, he could repeat from memory all of Italian canon law (he was a jurist trained in Padua). The validity of the reputed feats was attested by independent witnesses just as has been done in the case of the memory feats of modern mnemonists (e.g., see Paivio, 2007, p. 338 ff), which have also been validated by direct experimental evidence that was not available to the ancients. The method of loci implicated both verbal and nonverbal codes in memorizing speeches (an important application of rhetoric) but its practitioners did not allude to possible joint effects of both codes on performance. The connection between memory imagery and intelligence was first discussed systematically by Aristotle who knew about the method of loci and incorporated its imagery based on his theory of knowledge and reasoning. As expounded in De anima (see Yates, 1966, pp. 31–35), knowledge derives from sense impressions that are treated by the imaginative faculty to become images that are essential to thought. Thus, the “thinking faculty thinks of its forms in mental pictures” and “the soul never thinks without a mental picture.” Memory is an organized collection of such mental pictures as used, for example, in mnemonic systems which show that “it is possible to put things before our eyes as those do who invent mnemonics and construct images.” Aristotle's views on imagery and thought influenced subsequent writings including Rudolf Arnheim's (1969) modern classic, Visual Thinking. Adaptive functions of intelligence were prominent in ancient analyses of religious life, especially in connection with ethical conduct. The ethical virtue of Prudence – knowledge of what is good or bad (Yates, 1966, p. 20) – was viewed by Cicero as being composed of memory, intelligence, and foresight. A thousand years later, theological philosophers Albertus Magnus (1193–1206) and Thomas Aquinas (1225– 1274) similarly recommended artificial memory as part of the virtue of Prudence. The general adaptive functions of memory and intelligence in this case applied to religious ethical conduct that would lead to salvation in heaven and avoidance of hell (e.g., see Yates, 1966, p. 93 ff). The emphasis on general intellectual functions of imagery and memory peaked during the 14th–16th century Renaissance, especially through Giordano Bruno. He was an Italian Dominican priest and philosopher who adopted Metrodorus-type magical astrological images as memory places along with architectural loci to develop encyclopedic memory systems intended to unify earthly knowledge and mystical religious knowledge. Bruno's memory system was generally like modern semantic memory theories in that it focused on long-term verbal and nonverbal knowledge. It also resembles the Cattell–Horn concept of crystallized intelligence with elements related to

143

fluid intelligence captured by his adaptation of a dynamic Lullian combinatorial system described in the next section. In its encyclopedic scope Bruno's system anticipated philosophical Pansophism (universal encyclopedic knowledge). Bruno never explained how such knowledge becomes “unitized” in his system, a puzzle we address later in terms of the content and organization of mental representations. Educational implications of the method of loci influenced Johan Amos Comenius, the father of modern educational science. The influence showed up most concretely as a picture book, his “Orbis Sensualim Pictus” (The World In Sensible Pictures) published in 1658 and followed by many further editions including a modern computerized version. The Orbis displays categorically-organized pictured objects along with their verbal descriptions (Sadoski & Paivio, 2013, pp. 18–19). It expresses Comenius' general insistence on the importance of concrete experience with things in children's education, so that both things and words should be presented to the intellect at the same time. This necessarily implicates referential processing but Comenius did not discuss this or why concretization helps learning. Answers have come only recently from psychological studies. The method of loci and its connection to universal knowledge directly affected the development of science, in different ways. For example, Francis Bacon promoted empiricism in his 1620 Novum Organum (new method in science) in which he advocated observation of concrete facts followed by non-mathematical induction of general principles from them. His constant refrain was things before words (Rossi, 2000, pp. 145–150). In contrast, Gottfried Leibniz saw numbers as the basic characters to which images are transformed and organized into encyclopedic knowledge. His invention of calculus as a method of reasoning with numbers may have been motivated by his search for “real” characters that would advance all branches of knowledge (Yates, 1966, pp. 382–383). Numbers, however, are very abstract and their relations to imagery entails the same general problem that rhetoricians Quintilian and St. Augustine had encountered centuries earlier when they tried to understand how images could be used to remember abstract ideas in natural language rhetoric (Yates, 1966, pp. 21–26, 46– 49). Apparently they had not considered systematically how verbal language itself, especially abstract language, could serve as an abstract code. Comparing memory for concrete and abstract words using any task, as done in modern DCT research, gets at the abstraction problem directly. Nevertheless, in the ancient context, the abstraction problem was sufficiently understood to provide partial justification for the shift to verbal explanations of memory and its intellectual functions. The other major reason for the shift was inner iconoclasm that paralleled the outer iconoclasm of the Protestant Reformation, so that prohibition and destruction of the idols of the Catholic Church were extended to prohibition of mental images. In particular the “lively images” recommended by mnemonists were ostracized because they aroused depraved “carnal passions” (Yates, 1966, p. 277) and emotionally-neutral verbal memory techniques were to be used instead. 2.2. Verbal dominance epoch Ramon Lull and Peter Ramus particularly influenced the emerging verbal emphasis that conflicted with nonverbal

144

A. Paivio / Intelligence 47 (2014) 141–158

imagery. Lull used the idea of revolving concentric wheels that functioned like slide rulers as a combinatorial approach to memory and reasoning. It involved symbolic use of letters and words that presumably derived from more ancient systems of letter and word magic. Lull's version combined religious and philosophical concepts beginning with the attributes or Dignities of God – his goodness, greatness, eternity, wisdom, etc. – that permeated everything in the Divine physical universe. These attributes were symbolized as letters to be memorized on the concentric wheels so that each letter corresponded to an attribute of God. The attributes were organized logically on other wheels so that, for example, the Bonitas (goodness) of God was repeated as the goodness of an angel, man, imagination, animal creation, vegetable creation, the four ‘elements’ (e.g., fire), and in the virtues and arts and sciences (Yates, 1966, pp. 179–183). Lull interpreted his art as “artificial memory” comparable to places in the classical method of loci. However, images were totally absent in his memory system and everything was reduced instead to verbal abstractions, letters and names, which he applied to memory, intellect, and will, the three powers of the soul according to Christian doctrine. Lull viewed his system as an Art of Intellect. Leibniz was directly and profoundly influenced by Lull (e.g., Leibniz's dissertation was about Lullism) and by that route Lull reached forward to the present in that some computer scientists have adopted Lull as a kind of founding father whose system of logic was the beginning of information science (Bonner, 2007, p. 290). As mentioned above, Bruno also incorporated a form of Lullism in his system but Bruno's adaptation departed from the original in that he translated the letter symbols into images of things, which served as the basic intellectual code. Peter Ramus was a French Protestant who proposed a logical verbal memory system for organizing knowledge. It was taught in prominent educational settings in Europe before being brought to America by the Pilgrims and used by preachers and teachers, especially in the New England states. In the Ramist epitomy, a topic would be memorized as a verbal logical hierarchy with the most general level at the top, which then branches dichotomously into more specific levels. Ramism influenced American fundamentalist Protestantism and by that route may have reinforced the opposition to imagery in behaviorist psychology (Buckley, 1989; Thomas, 1989) so that J. B. Watson, the father of behaviorism, rejected all mentalistic concepts, especially imagery. He was thus an inner iconoclast who, entirely without evidence, engendered “iconophobia” in generations of psychologists. The scientific “psychotherapy” for this condition was initiated only as recently as the 1960s. Watson also reinforced verbal processing views by insisting that thinking is nothing but talking to ourselves (Watson, 1930, p. 238). Verbal processes also dominated the first systematic scientific studies of memory by Ebbinghaus (1964), who pioneered as well in intelligence testing in that he invented the sentence completion test to measure combinative mental ability. His verbal memory focus guided research and theory for 80 years. The research included an experiment by Bower, Clark, Lesgold, and Winzenz (1969) that confirmed the episodic memory benefits of presenting verbal information in a Ramus-type hierarchy. Formal hierarchical models of longterm semantic memory knowledge similar to the Ramist model

also evolved from the verbal memory tradition (e.g., Collins & Loftus, 1975). The verbal emphasis became generally associated with intelligence in the 20th century with the emergence of IQ tests. Historian Murdoch (2007, p, 2), for example, concluded that “for a hundred years, IQ tests have largely been based on verbal ability.” This generalization has been qualified by the emergence of tests of nonverbal (performance) abilities that supplemented but did not interact with verbal abilities. 2.3. Verbal and nonverbal solitudes It is widely known that Binet and Simon (1905a,b) created an intelligence test to identify children likely to have learning difficulties in Parisian schools. The test included nonverbal abilities (e.g., picture tests of vocabulary knowledge) that were not separately analyzed because only the overall scores served as predictors. This was also true of the 1916 English language revision, the Stanford–Binet Intelligence Scales. In the United States, tests of nonverbal intelligence were developed (e.g., by Healy & Fernald, 1911; Pintner & Paterson, 1917) for use with language-impaired individuals and foreign language immigrants. The latter were of special interest to H. A. Knox (see Richardson, 2011), one of the physicians employed at the Ellis Island immigration station early in the 20th C, who developed a wide range of verbal and nonverbal tests to assess the intelligence of potential immigrants. The US military developed both verbal and nonverbal measures, the Army Alpha and Beta Examinations (Yerkes, 1921), to help place recruits in positions that suited their skills. This led to psychometric theories and test batteries that featured factorially-independent verbal and nonverbal scales (see Boake, 2002). David Wechsler constructed the most popular of such tests. He first encountered intelligence tests while assigned to score the Army Alpha examination in 1917, leading eventually to his development of the Wechsler–Bellevue Intelligence Scale (Wechsler, 1939) and its subsequent revisions. He selected largely from the Army Alpha and Beta Examinations, which were themselves based on earlier sources. Verbal and nonverbal abilities also emerged as separate factors in later psychometric models (e. g., Horn & Cattell, 1966; Kaufman & Kaufman, 1997; Vernon, 1961) as well as nonmetric multidimensional approaches, such as Guttman's (1965) “radex” theory of intelligence, which was used to re-analyze Thurstone's (1938) seven-factor data set to yield a clustering of verbal, figural, and numerical test points (Snow, Kyllonen, & Marshalek,1984; see Carroll, 1993, p. 640). Carroll's (1993) Three-Stratum Theory of Cognitive Abilities is the most comprehensive theory of the factor structure of intelligence that distinguishes between verbal and nonverbal abilities. It is based on a re-analysis of more than 460 factor analytic studies of cognitive abilities organized into a common structure. Thus a detailed evaluation of Carroll's theory and its empirical test base is justified here because it captures features shared by most factor analytic studies of intelligence. The specific question is whether referential processing might somehow emerge from Carroll's analysis despite the fact that he, too, failed to recognize such processes explicitly. Carroll was influenced by the theory of fluid and crystallized intelligence (Cattell, 1963; Horn & Cattell, 1966) to the extent that a later modification was named the Cattell–Horn–Carroll

A. Paivio / Intelligence 47 (2014) 141–158

(CHC) theory of cognitive abilities (see McGrew, 2009). The present focus is on the original Carroll (1993) model. It has a hierarchical structure consisting of three levels with 65 ability factors at the first level, eight at the second level, and a single General Intelligence (G) factor at the apex (analogous to Spearman's g). The factor tests reveal how the different levels incorporate verbal and nonverbal abilities. Most informative are the second level factors ranked in the order of their contribution to G. The first two second-level factors correspond to Horn and Cattell's (1966) Fluid and Crystallized Intelligence. Six others were identified as General Memory and Learning, Broad Visual Perception, Broad Auditory Perception, Broad Retrieval Ability, Broad Cognitive Speediness, and Processing Speed. The verbal abilities are defined by tests that involve verbal stimulus material and responses. These tests clustered under Crystallized Intelligence and were also scattered under narrower second-level verbal factors, Verbal Memory Span, Free Recall, Speech Sound Discrimination, and Verbal Fluency. Conversely, nonverbal abilities are defined by their nonverbal stimulus and response content. Especially exemplary are the second-level factors Broad Visual Perception and Broad Auditory Perception, the former “involved in any task that requires the perception of visual form [and] is involved only minimally, if at all, in the perception of printed language forms” (Carroll, 1993, p. 625). This broad nonverbal factor dominates a first-level Visualization factor defined as “the ability to manipulate or transform the image of spatial patterns into other visual arrangements” (Carroll, 1993, p. 316) as measured by such tests as Form Board, Paper Folding, and Surface Development. Visualization is one of the factors most frequently found in factorial investigations, so that “It would be possible to construct a list of some 800 variables that have salient loadings on the factor in [his] data sets” (p. 321). Carroll summarized a more manageable set of examples consisting of 147 “token” Visualization tests from 144 data sets (Carroll, 1993, Table 8.3; pp. 317–320). Broad visual perception includes four other major firstorder nonverbal factors: Spatial Relations (defined by speed of mentally manipulating visual patterns); two Perceptual Closure factors (Speed and Flexibility) involved in identification of fragmented nonverbal or verbal visual patterns; and a more general Perceptual Speed factor (Carroll, 1993, pp. 362– 363). More tentative visual perceptual factors include Serial Perceptual Integration, Spatial Scanning, Imagery Ability, and Length Estimation. The nonverbal “purity” of the defining tests would be a relevant selection criterion that was not consistently satisfied. For example, Closure Speed tests include fragmented patterns of printed letters or words as well as pictured objects. Although the nonverbal–verbal distinction did not emerge as independent factors from Carroll's analysis, it has shown up clearly in experimental studies (e.g., Weldon & Roediger, 1987). Another example of mixed purity is Serial Perceptual Integration as defined by tasks involving sequential presentation of picture elements as well as letters in words. This contrasts with DCT-motivated experiments in which sequential processing was expected and found to be better with printed words than pictured objects (see later). Similarly, Perceptual Speed tests included a mix of printed verbal material and pictured objects as stimuli. Carroll's second-level factor Broad Auditory Perception also mixes nonverbal and verbal first-level factors defined by “any

145

task or performance that requires the perception or discrimination of auditory patterns of sound or speech” (Carroll, 1993, p. 625). The patterns consist of specific tonal qualities and complex musical structures. From the DCT perspective, however, in addition to the mixing of nonverbal and verbal auditory abilities, there is a complete absence of ability tests concerning processing of natural environmental sounds associated with animals or inanimate objects. This large domain is ecologically important to humans and other hearing animals. Normative lists of environmental sounds have been used in perceptual and memory experiments, including ones specifically inspired by DCT (e.g., Crutcher & Beer, 2011; Thompson & Paivio, 1994). Individual differences have also been studied using special populations, such as hearing-impaired participants with cochlear implants as compared to individuals with normal hearing (see Marschark, 2007), but general cognitive abilities involved in dealing with environmental sounds are yet to be systematically studied. It is also noteworthy that, although relevant referential processing abilities, such as picture naming, are scattered among Carroll's re-analyzed test batteries, their adaptive functions never emerged clearly from those analyses and Carroll did not draw attention to their importance to intellectual functioning. His analyses implicated availability of referential processing abilities but not their functional effectiveness. This issue is discussed in detail later in the context of DCT. 3. Dual coding theory and intelligence Modern imagery and dual coding theories began in the 1960s as reactions to issues that arose during the historical nonverbal and verbal epochs. Inner iconoclasm was countered by research that confirmed the mnemonic potency of imagery memory techniques analogous to the method of loci. Dual coding research additionally revealed benefits of verbal memory processes. An inaugural dual coding theory was described in a research review article (Paivio, 1969) and elaborated in a subsequent book (Paivio, 1971). These publications summarized research results reported in numerous articles, including memory effects and effective variables associated with imagery instructions, language concreteness–abstractness (empirically attributable to superior imagery-evoking value of concrete words), and picture superiority over words in free recall, an effect that was reversed in sequential recall of listed items. The reversal was expected from a DCT hypothesis concerning differences in the way nonverbal and verbal information are organized and processed mentally. Later research increased the range of tasks to which an improved DCT could be applied, so that a summary chapter (Paivio, 1983) described 60 findings that were consistent with DCT but not single code theories that emphasized verbal processes or abstract mental codes. This conclusion was confirmed and extended to additional phenomena (Paivio, 1991) including most recently a DCT interpretation of mind evolution (Paivio, 2007). Thus the following statement remains currently appropriate. … a relatively small set of theoretical assumptions suffice to account for a wide range of independent phenomena … [including] specific changes and even reversals of effects as a function of task conditions. [Moreover,] different classes of

146

A. Paivio / Intelligence 47 (2014) 141–158

variables often produce parallel effects, presumably because the variables converge on the same underlying processes. [Paivio, 1983, pp. 3i3–3i4] The theory was empirically constructed and tested from the outset using operational procedures to access and use nonverbal and verbal mental codes. The ultimate classes of procedural defining variables included stimulus attributes, experimental manipulations (e.g., task instructions), individual difference tests, neural correlates, and subjective reports. The procedures converged on the DCT model presented in Fig. 1, which shows the internal structural entities and interconnections that underlie the DCT systems. The signature DCT features are the referential interconnections that enable “Nonverbal mind and verbal mind [to be] interlocked in a synergistic relation that evolved into the nuclear power source of our intellect” (Paivio, 2007, pp. 3–4). 3.1. DCT representational structures and processes The DCT structures can be characterized as semantic memory representations, defined generally as the sum of all knowledge (Thompson & Madigan, 2007). They also parallel crystallized intelligence viewed as long-term memory information derived from cultural experience. These views differ from DCT in that they typically focus on verbal knowledge. Tulving defined semantic memory as knowledge of the world but acknowledged that much of the related research “has to do with people's knowledge of language” (Tulving, 1983, p. 55). Neuroscientists Binder, Desai, Graves, and Conant (2009) also used only the word stimuli in their search for the brain locations of semantic systems while recognizing that nonverbal objects have rich semantic properties. However, DCT emphasizes both nonverbal and verbal knowledge even when language is the object of study (e.g., Paivio & Begg, 1981, p. 272). Such knowledge is appropriately termed apperceptive mass, used in psychology and education to refer to knowledge structures that continually change and expand as new experiences are assimilated into

Fig. 1. DCT model of multimodal logogen and imagen units, connections, and implied processes. Adapted from Paivio (1986), Figs. 4-1, p. 67; adapted originally from Paivio (1978b, Fig. 1, p. 381).

them. Its DCT relevance stems from evidence that domains of personal interests or values (e.g., religious, theoretical, economic) are represented by expandable verbal and nonverbal associative structures (Paivio, 2007, pp. 31–32; Paivio & Steeves, 1967). The terms logogen and imagen refer respectively to verbal and nonverbal representational units that generate consciously experienced mental words and images and can function unconsciously to mediate cognitive performance. The lines in Fig. 1 represent pathways that connect units within and across systems, with directional arrows symbolizing mutual bi-directional processing. Also shown are the connections of representational units to stimulus and response systems. The DCT systems were originally defined by behavioral data but it was always assumed that they have neural correlates that have been increasingly confirmed by functional brain evidence (Paivio, 2007, 2010). Detailed explanations follow. 3.1.1. Logogens and imagens Morton's (1979) word-recognition model uses modalityspecific visual and auditory input logogens and output logogens to account for recognition performance. In DCT, the variety of modality-specific representations expanded to include auditory, visual, haptic, and motor logogens, as well as separate logogen systems for the different languages of multilingual individuals. Moreover, DCT treats logogens as hierarchical sequential structures of increasing length, from phonemes (or letters) to syllables, conventional words, fixed phrases, idioms, sentences, and longer discourse units – anything learned and remembered as an integrated language sequence. Imagens also are multimodal (visual, auditory, haptic, motor) representations organized hierarchically into spatial nested sets that are most apparent in visual objects – for example, we see pupils within eyes within faces within rooms within houses within larger scenes, and so on. Importantly, the modality–specificity of logogens and imagens excludes abstract mental representations such as propositions. Thus the functional domains associated with stimulus meaning and cognitive abilities are conceptualized entirely in terms of modality specific logogens and imagens. 3.1.2. Representational connections and activation processes Processing begins with direct activation of imagens and logogens that correspond to stimulus patterns, a kind of template matching process involved in stimulus recognition. Higher level processing involves indirect activation of representations by already-activated logogens or imagens. Such processing occurs within systems when logogens activate other logogens and imagens activate other imagens. Referential processing is the special kind of associative mutualism that involves cross-system activation of logogens by activated imagens or vice versa. The underlying structures all result from associative experiences that promote growth of anatomical and biochemical “connectors” between neurons. Their activation is probabilistically determined by individual differences in associative experiences and priming by contextual information in natural or experimental situations. Direct activation of mental representations is necessary but not sufficient to account for the functional meaningfulness of the activating stimuli. Modality-specific logogens have no meaning in the semantic sense. They are directly “meaningful” only in that they can be activated by stimuli similar to those involved in the original formation of the corresponding

A. Paivio / Intelligence 47 (2014) 141–158

logogens. This direct activation is reflected in the familiarity and recognizability of the verbal stimuli. Imagens, however, are intrinsically meaningful in that, when directly activated by perceptual objects, the imaginal memory traces resemble the perceived objects and scenes they represent. Further meaning for both logogens and imagens arises from activation of their associative connections to other representations of either class. Associative connections between different logogens and between different imagens allow for within-system associative processing that defines associative meaning as measured, for example, by word association tests on the verbal side and analogous procedures on the nonverbal side. Referential associations between logogens and imagens permit objects to be named and names to activate images that represent world knowledge. The referential responses define referential meaning that especially differentiates concrete and abstract language. Concrete words are referentially meaningful by virtue of their connections to nonverbal imagens, whereas abstract words lack such referential connections and derive their higherorder meaning primarily from associative connections to other logogens. However, abstract words also can be indirectly concretized. For example, the activated logogen for the word “religion” can activate the associated logogen for “church,” which then activates an imagen that might be consciously experienced as the mental image of a church. Analogous concepts are found in other psychological, linguistic, and intelligence theories. The variable size of logogen and imagen hierarchical units is consistent with the concepts of chunk (Miller, 1956), integral stimulus (Garner, 1974), “blob” (Lockhead, 1972), and lexical cluster (Divjak & Gries, 2008). Some linguists accept fixed expressions of any size – morphemes, stems, words, compounds, phrases, idioms, and even longer expressions – as lexical units (e.g., Langacker, 1991). Other theorists assume that lexical words are meaningless units that function as pointers, addresses, or clues to meaning rather than being semantically meaningful in themselves (e.g., Arnheim, 1969; Elman, 2004; Milner, 1999, p. 82; Rumelhart, 1979; Strømnes, 2006). The symbolic content category in Guilford's Structure-of Intellect model similarly includes logogen-like “denotative signs that have no significance in and of themselves, such as letters, numbers, musical

147

notations, codes, and words (as ordered letter combinations)” (Guilford & Hoepfner, 1971, p. 20). The role of interword associations on the verbal side of DCT has parallels in corpus linguistic and psycholinguistic studies of word collocations in text and speech (e.g., Aitchison, 2003; Kiss, 1975; Landauer & Dumais, 1997) as well as neural associative networks (Crutch & Warrington, 2005). The DCT multimodal representational systems have a partial parallel in Morton's (1979) logogen theory in that it includes imagen-like “pictogens” to account for picture recognition. Likewise, Coltheart (2004) found neuropsychological evidence for visual and auditory lexicons along with a nonverbal “lexicon” involved in object recognition. The terms perceptual simulator (Barsalou, 1999) and geon (Biederman, 1987) also are near-synonyms for imagen. 3.2. DCT behavioral analysis of intelligence theories and tests Memory abilities are prominent in all psychometric theories and tests of intelligence, to the extent that working memory is sometimes used as a proxy measure of IQ. Even so, standard theories and measures tend to underestimate the scope and power of memory in intellectual abilities. Moreover, although such theories include both verbal and nonverbal abilities, the following analysis shows that only DCT explicitly emphasizes the interconnectedness and interplay of the two systems. The critical difference is illustrated in Fig. 2 by the structure of the Wechsler Adult Intelligence Scale (WAIS III) shown alongside the DCT model. The comparison is appropriate here because the Wechsler tests are the most widely used and durable IQ tests, and because WAIS III served as the principle measure of intelligence in neuroscientific studies reviewed below. The models differ in that WAIS measures individual differences in intellectual abilities whereas DCT depicts the general structures and processes that underlie all cognition. The models become directly comparable, however, when DCT is linked to individual difference variables (see later). Most notable in Fig. 2 are the bi-directional referential pathways that connect verbal and nonverbal systems in DCT but are absent in the WAIS, at both the general systems and individual test levels. For example, even name–picture and picture–name comparisons, the simplest tasks requiring referential processing, are missing in WAIS. Such tests were included

Fig. 2. Comparison of WAIS III and DCT structures to show the presence of referential connections between DCT verbal and nonverbal representational systems that are absent between WAIS verbal and (nonverbal) performance tests.

148

A. Paivio / Intelligence 47 (2014) 141–158

in the original Binet and Simon IQ test but were not separately analyzed. Subsequent factor-analyzed test batteries included picture–name relations as measures of vocabulary, but their potential contribution to verbal–nonverbal interactive processing was not examined. Possible explanations are considered following discussion of referential processing in real-life and experimental tasks. 3.3. Real life referential processing Referential processing occurs in a wide range of domains including language acquisition, education, technology, and creative thinking. It occurs early in language acquisition when children begin to learn names of objects, and soon extends to learning grammar, which initially depends on experiencing correlated relations between language and object structures (Moeser & Bregman, 1973; Stromnes & Iivonen, 1985). Further progress continues later on the basis of experience with grammatical language alone. These observations are consistent with a DCT hypothesis concerning imagery and verbal experience in learning syntactic structures (Paivio, 1971, pp. 437–438). Educational achievement has been the gold standard in intelligence test validation ever since Binet and Simon's pioneering work. Lindamood–Bell Learning Processes developed remedial programs for children with reading problems. Comprehension is taught by guided visualizing and verbalizing (V/V) based on DCT principles (Bell, 1991a,b; Paivio, 2007, pp. 446–448; Sadoski & Paivio, 2013, pp.127–129). Students learn to construct nonverbal images that correspond to progressively longer language segments – words, phrases, sentences, texts – and are encouraged to describe their images

in detail. Sadoski and Willson (2006) evaluated the results of a multi-school augmentative intervention program that used V/V to improve the reading performance of students in grades 3, 4, and 5 with low initial reading achievement. The important results were that students in these schools eventually outperformed other comparable schools in tests of reading. Fig. 3 shows the yearly improvement for students who started in grade 3. Similar large-scale educational benefits on reading were obtained in Australia using nonverbal and verbal integration instructions patterned after DCT and Bell's Visualizing and Verbalizing technique (Woolley, 2011). Krueger (1976); summarized in Paivio (2007, pp, 388–391) investigated the role of imagery and verbal processes in creative invention. He used fifteen applied scientists with strong records of creative work who verbally reconstructed personal inventions (e.g., magnetic tape, radar pulse tracking, induction plasma torch, solid state TV tube, and HP-35 pocket calculator circuitry). The reconstructions suggested that visual imagery was most often used as the mode by which technical fragments were brought together as the cognitive basis of the invention. The interviews indicated as well that synergistic dual coding was at work in every case in the form of self-directed questions and comments that kept the inventor's mind “on the right track.” In a second study, Krueger gave each of eight scientists four invention problems (e.g., an artificial iris, a fan without moving parts), to which they suggested solutions using a method of isomorphic reporting in which they tape recorded their verbal thinking and drew their images, thus externalizing referential processing in both directions. This aspect of Krueger's research captured the processes involved in expert performance under

Fig. 3. Reading performance test scores over eight years for students in Pueblo School District 60 in the State of Colorado who, in grade 3, started a multischool augmentative reading intervention program using the Lindamood–Bell Learning Processes programs. Their growth curve significantly exceeds that of students in other comparable Colorado schools. Updated from Sadoski and Willson (2006).

A. Paivio / Intelligence 47 (2014) 141–158

standardized conditions that allowed them to be recorded and measured, a recommended procedure in the scientific study of expertise (Ericsson, 2003, p. 50 ff). All of the scientists “used visualizing heavily” (Krueger, 1976, pp. 30–31) in ways that contributed to solutions of the invention problems. Dual coding was especially evident in the protocols as thinking operations that entailed referential processing in both directions: (a) verbal–imaginal (“words elicit images which represent technical building blocks and effects”), (b) imagery–verbal (“judgments about images stated in words”), and (c) an interplay of the two (“back and forth shifts between images and words”). Krueger's studies justify similar DCT interpretations of creative processes and products associated with major contributors to the basic sciences and arts. Examples analyzed in Paivio (2007, Chap. 18) include Darwin, immunology; the DNA molecule; Einstein; mathematics; psychologists Freud, Skinner, and Hebb; and artists Shakespeare, Beethoven, and Picasso. Metatheoretical DCT analyses showed additionally how the theories themselves entail combinations of nonverbal and verbal components and processes, as in the nonverbal image of the double helix structure of the DNA molecule accompanied by its chemical–verbal description. Such metatheoretical contributions are reflected in how scientists use graphs and tables to illustrate meaning in their theories, a topic that Gross and Harmon (2014) recently analyzed systematically in terms of DCT, augmented by organizational principles of Gestalt psychology. 3.4. Psychometric tests of referential processing Numerous tasks have been used to measure referential processing ability in factor analytic studies. The most objective method involves referentially-related words and objects (or pictures) as in receptive and expressive vocabulary tests. Johnson, Paivio, and Clark (1996) reviewed picture naming research from the DCT perspective (selectively updated in Paivio, 2010, pp. 219–220). Paivio, Clark, Digdon, and Bons (1989) investigated processing in both directions by comparing reaction times for naming pictures and imaging to their names. The results were that naming and imaging RTs correlated .58 over items and .71 over participants (the latter reported earlier, Paivio, 1986, p. 106). Thus naming pictures and imaging to their names are moderately correlated, presumably because referential experiences during learning word meanings are often but not always bi-directional. The study of referential relations has also been extended to attributes of objects and their names. Thus, there is a growing literature on rapid automatized naming (RAN) of object pictures, colors, letters, and digits, which focuses on the relation between naming speed and general language abilities (Norton & Wolf, 2012). The studies have used either summed scores over the four types of stimuli (e.g., Arnell, Joanisse, Klein, Busseri, & Tannock, 2009) or separate scores for the individual stimulus types. The latter have revealed the nonverbal–verbal difference between pictures and colors on the one hand and letters and digits on the other. For example, van den Bos, Zijlstra, and Spelberg (2002) found that naming speed for the four types of stimuli combined into separate alphanumeric and color/picture factors for participants from age 12 years to 46 years. The DCT interpretation is that picture and color

149

naming require referential processing whereas letter and digit naming primarily implicate verbal associative processing. Memory effects of referential processing have been studied by presenting items as pictures and words, or as repeated words with instructions to silently pronounce the word on one presentation and image to it on the other. Both procedures produced additive effects in free recall, with the memory contribution of the nonverbal code (picture or generated image) being twice that of the verbal code (e.g., Paivio & Lambert, 1981). This differential contribution is even greater in cued recall. For example, Begg (1972) found that free recall of concrete phrases (e.g., white horse) or their constituent words was about twice that of abstract phrases (e.g., basic truth). This difference increased six-fold for concrete phrases when recall was cued with one of the phrase words, but did not increase at all for cued abstract phrases. Begg interpreted these effects in terms of the integrated organization of memory images, which are redintegrated (re-activated) by a high imagery retrieval cue. This effect, termed the conceptual peg effect in DCT, parallels the role of imaged loci in the ancient memory technique. Other referential processing tasks require participants to compare words on long-term memory attributes of their referents when asked, for example, to decide which is larger, a cat or a toaster, or similar questions concerning their shape, color, weight, etc. Comparison times generally increased as differences in remembered attributes decreased, a “symbolic distance” effect much like that obtained in corresponding perceptual comparison tasks (reviewed e.g., in, Paivio, 1986, Chap. 9). Even more complex referential processing occurs entirely internally in Block Visualization (Guilford and Hoepfner, 1971, p. 378), a problem-solving task that entails a continual dual coding interplay between covert descriptions and mental images of cutting colored blocks into smaller cubes and counting the number of smaller blocks with one, two, or three colored faces. Such interactive use of verbal and nonverbal representational systems corresponds to long-term working memory activity as described by Ericsson and Kintsch (1995). 3.5. Individual differences in referential ability Richardson (1978) provided an inferential bridge between experimental and individual difference approaches to referential processing. Using post-learning questionnaires, Richardson discovered that the proportion of word pairs for which subjects reported using imagery in paired-associate learning was an excellent predictor of their performance with concrete pairs but not with abstract pairs. This is consistent with prior experimental studies that have shown that imagery is more helpful as a memory aid with concrete than abstract language (e.g., Paivio & Foth, 1970). Successful learners relied especially on transforming concrete word pairs into nonverbal images. Individual differences in word-picture tests of referential processing ability have been used as intelligence tests. For example, the Peabody Picture Vocabulary Test (PPVT) includes a large sample of pictures and names of common objects. Scores are based on selection of correct pictures for names. Carvajal, Nowak, Fraas, and McConnell (2000) obtained correlations of .74 and .67 between PPVT scores and WAIS Verbal and Full Scale IQ scores for university students. A comparable relation would not be expected if vocabulary is measured entirely verbally, as

150

A. Paivio / Intelligence 47 (2014) 141–158

in word fluency tests that requires writing as many words as possible that contain or begin with a specific letter. Yet another example of cognitive implications of individual differences in referential naming ability is a hypothesis from a bilingual version of DCT, namely that bilingual individuals have shared and separate imagen systems linked to logogens of first (L1) and second (L2) languages (Paivio, 2014; Paivio & Desrochers, 1980). A prediction from the hypothesis was tested by Jared, Poh, and Paivio (2013) using picture naming speed of Mandarin–English bilinguals who had learned Mandarin in China and English in Canada. The participants named aloud in both Mandarin (L1) and English (L2) culturally biased and unbiased pictorial object images. Culturally biased images were named significantly faster in the culturally congruent language than in the incongruent language. These results indicate that some image representations are more strongly connected referentially to one language than the other, consistent with bilingual DCT and with implications for studies of individual differences in referential ability. Bucci and her colleagues (Bucci, 1984; Bucci & Freedman, 1978; Bucci & Kabasakalian-McKcbeay, 1992) were the first to systematically study individual differences in referential processing ability, which they called Referential Activity (RA). It is defined as connecting non-verbal experience with language, and was initially measured by color naming speed with reading time for printed color names subtracted out. Bucci et al. found that RA correlated with a broad range of verbal behaviors, characterized generally by relatively frequent use of concrete metaphors and descriptive terms suggestive of imagery in language processing by high RA scorers. Low RA scorers tended to be more abstract and vague, as though they relied relatively more on word-to-word associative links. It is specifically relevant that RA scores correlated significantly (r = .42) with comprehension scores from the WAIS, which was expected because the WAIS includes items that refer to concrete situations and actions, so that word-activated referential imagery could be used to answer the questions. Such results suggest that individuals who are high in RA are habitually better than those low in RA at moving back and forth between referentially related verbal and nonverbal representations in various domains. Finally, individual differences in referential processing involved in symbolic comparisons correlate significantly with relevant cognitive abilities. For example, in a clock-time comparison task (Paivio, 1978a), participants saw pairs of digital times such as 3:20 and 7:50 and chose the time in which the hour and minute hands of a comparable analog clock would have a larger or smaller angle. The task yielded the usual symbolic distance RT effect in that comparative judgments for the digital times became slower as the angular size difference on an (imaged) analog clock got smaller (e.g., judgments took longer for 3:20 than 7:50 in the example given). Moreover, the decision times correlated modestly but significantly (.32 and .29 respectively) with a spatial relations imagery test and a verbal reasoning test (Paivio, 1980, 150–151), thereby linking the mental comparison task better than chance to both nonverbal and verbal tasks that are typically included in intelligence test batteries. Schultheiss and Strasser (2012) found that individual differences in the Paivio (1978a) clock comparison task even predicted participants' mood judgment speed, consistent with a DCT analysis of the interplay of nonverbal and verbal systems in affective judgments (Paivio, 1978b).

3.6. Why the neglect of referential abilities? Given the pervasive role of referential processing in real-life and laboratory intellectual tasks, it is perplexing that interactive dual coding abilities are not similarly manifested in major psychometric intelligence theories, including Carroll's (1993) re-analysis of hundreds of data sets. How can this be? One possible methodological reason is that intelligence researchers have not included enough test candidates for a referential processing factor to emerge. The general effect of having too few tests for potentially-relevant cognitive abilities was repeatedly noted by Carroll and summarized as follows: “Studies have often failed to include sufficient numbers of univocal measures of given factors. Ideally, at least three or four measures of a given factor should be included (Carroll, 1993, p. 691).” In the case of referential processing, this test paucity coincided with historical factors that led test developers to focus on a dichotomous verbal–nonverbal view of intelligence, and perhaps for that reason, purge test batteries of factorially unclear cases. A second possibility is that referential processing demands are hidden within multidimensional factors. As Carroll noted, many tests are inevitably multidimensional, a problem that perhaps could be resolved by hierarchical factor analytic research “that could determine how best to produce ‘factorially pure’ (unidimensional) scores from measures that are found fundamentally multidimensional” (Carroll, 1993, p. 691.) This goal is reachable in principle through research guided by his three-stratum theory of cognitive abilities. The DCT view is that such research must take systematic account of the fundamental role of verbal–nonverbal interactive abilities, which were not explicitly included in any of the factor analytic research reviewed by Carroll. Nonetheless, such abilities are implicitly involved in many psychometric tests. Inspection of factor analytic results reveals numerous individual tests and factors that implicate the verbal–nonverbal referential interplay that fails to appear at more general levels of the hierarchical models. A hint of such a factor in Carroll's model is Naming facility, which appears under the second level factor, Broad retrieval ability, but this is only one of 65 Stratum I factors listed in his model. His Stratum II Fluid Intelligence factor is saturated with tests of reasoning skills, which are traditionally considered to be “at or near the core of what is ordinarily meant by intelligence” (Carroll, 1993, p. 196). These tests generally do not require verbal responses, but many of the tasks nonetheless involve combinations of language, number, and spatial skills (Carroll, 1993, p. 199) that may include implicit referential processing. Carroll (1993, pp. 201–210) listed more than 150 test variables from the reasoning domain. The test descriptions suggest that about 25% of the tasks involve dual coding referential processes. These include General Verbal Reasoning tasks defined as tasks that involve problems stated verbally, but some of which are accompanied by diagrams or require nonverbal mental images. For example, an Unlikely Things test requires respondents to select the two more unlikely or incongruous of four verbally stated features for a shown sketch of common objects. Moreover, imagery and dual coding are implicated in syllogistic reasoning (reviewed in Paivio, 1986, pp. 205–206) and other verbal deductive reasoning tasks (e.g., Goel & Dolan, 2001; Knauff, Fangmeier, Ruff, & JohnsonLaird, 2003; Knauff, Mulack, Kassubek, Salih, & Greenlee, 2002).

A. Paivio / Intelligence 47 (2014) 141–158

However, the 25% of implicit referential processing tasks came from a large number of factor analytic studies many of which included no more than one test item that implicates referential processing. They were not all included in any single factor analytic study. As already mentioned, Carroll points out more generally that many of the original studies he re-analyzed had relatively small numbers of variables related to individual factors. Matrix reasoning tasks that the use only nonverbal figural patterns seem to be especially relevant from the DCT perspective. For example, Raven's Progressive Matrices is generally regarded as the best single test of general intelligence. It is nonverbal except for the need to understand task instructions, but Vernon (1983) found that performance on the Raven's correlated about as highly with the WAIS verbal subtests Vocabulary (.44) and Arithmetic (.52) as with the nonverbal subtests Block Design (r = .62) and Object Assembly (.52), thus suggesting significant dual coding interplay in this “nonverbal” task. Inasmuch as the nonverbal test material is always present, however, systematic referential processing seems unnecessary and the verbal involvement presumably is idiosyncratic and variable, serving perhaps to guide different pattern comparison strategies. This analysis applies as well to other studies that have revealed multiple factorial correlates of Raven's tests. For example, Lynn, Allik, and Irwing (2004) identified three components they called verbal-analytic reasoning, visuospatial ability, and gestalt continuation (also presumably nonverbal). Kunda, McGreggor, and Goel (2012) summarized a number of studies that supported the existence of visual and verbal Raven's Matrix strategies and concluded that individuals use various types of mental representations in the task. More generally, higher-order levels in hierarchical models are permeated with reasoning tasks. We saw, for example, that the Cattell–Horn concept of fluid intelligence is largely defined by first-level reasoning tasks, some of which depend on referential processing skills. Nonetheless, such skills are generally not explicitly incorporated into intelligence test batteries and we are thus left with the conclusion that the test designers were simply unaware of the importance of a referential processing interface. This appears to be true as well at the level of Spearman's general intelligence, g, which also emerged empirically from Carroll's hierarchical factor analysis as the common third-stratum factor he labeled G. His listing of the average G loadings of first-level factors {Carroll, 1993, Table 155, p. 597) shows the separate contributions of nonverbal and verbal factors to G. Thus imagery-loaded nonverbal Visualization and verbally dominant Induction have the highest G loadings (57, median loading), but Carroll's Table 15-5 does not include a referential processing factor that provides the bridging interface that would allow the separate systems to interact. 3.7. Computational modeling and DCT A reviewer of the initial submitted version of this article suggested that a computational model of DCT would clarify the theory and its implications. A number of attempts were made to develop computational simulations of imagery processes around the time of Pylyshyn's (1973) formal critique of imagery. None of the models succeeded in modeling imagery. Recently, Paivio and Sadoski (2011) commented on Elman's (2004)

151

proposal to develop an improved computational model of the role of world knowledge in comprehension of event descriptions, which would include representations of referent objects. The proposed representations turned out to be abstract schema rather than image-like modality-specific representations of objects, events, and scenes. We pointed out that the “scenes” are first modeled as natural-language descriptions that are then transformed into abstract formal descriptions (e.g., propositions, structural descriptions) that are necessary for computer programming. This was the case with early and recent attempts to simulate imagery. The nonverbal nature of images disappears in such approaches. Kintsch (2004), a computational psychologist who has relied on abstract propositions to represent nonverbal situational information, remarked that “Situational models may be imagery based, in which case the propositional formalism used by most models fails us.” Given just that problem, it is hard to appreciate how a computational implementation of DCT would enhance its explanatory power. Kosslyn, a major imagery theorist and researcher, abandoned his early commitment to computational modeling of imagery because he became convinced that the computer metaphor is inappropriate and he switched to a brain-based approach (Kosslyn, Van Kleeck, & Kirby, 1990). Computational modeling of DCT dual coding processes would be even more problematic. Computational models nonetheless continue to be proposed for dealing with issues relevant to intelligence and DCT. For example, Kunda et al. (2012) developed computational models of reasoning on a Raven's Progressive Matrices (RPM) test based entirely on nonverbal images (“iconic visual representations’) rather than on propositional transformations of visual inputs as used in most previous computational accounts. Kunda et al. based their models directly on comparisons of patterns of visual features of RPM problems (arrays of geometric forms) with the patterns of test alternatives. Visual feature patterns were operationalized as computer pixel patterns. Their most successful model yielded correct choices of RPM solutions highly comparable to the performance of human subjects. This computational rendering of RPM problem solving is compatible with DCT processing based on the nonverbal system alone, but does not bear on the interactive effects of verbal and nonverbal processes. More general computational models of imagery also continue to be proposed. For example, Wintermute and Laird (2009) suggested how simulated imagery could compensate for an imperfect abstract (propositional) problem representation in computational (AI) cognitive architecture. Their proposal is relevant here for two reasons. First, it clearly acknowledges that abstract representations do not capture all the details necessary for correct internal reasoning, which agrees completely with what has been the DCT position for more than forty years. Second, their compensating imagery simulation agent uses a simple fixed, programmer-supplied language to describe imaging (e.g., imagine cube A centered on top of cube B), the implications of which admittedly are yet to be fully characterized (Wintermute & Laird, 2009, p, 6). The direct DCT implication is that imaging directly to natural language descriptions (i. e., real-life referential processing) is even further beyond the capacity of current computational AI. Another recent computational approach to intelligence is based metaphorically on biological mutualism (van der Maas

152

A. Paivio / Intelligence 47 (2014) 141–158

et al., 2006), which is defined as a symbiotic relationship between individuals of different species in which both individuals benefit from the association. The van der Maas et al. psychological “species” are basic cognitive processes such as perception, memory, decision, and reasoning that have different developmental growth parameters and interact in ways that explain the positive correlations between intelligence tests that Spearman had sought to explain in terms of a latent common g variable. Such a factor plays no explanatory role in the van der Maas et al. computer simulations. The van der Maas et al. (2006) theory is formulated in general terms that do not depend on any particular cognitive architecture or brain model. In DCT, however, mutualism is based entirely on specific cognitive/neural architecture involving individual effects and interactions between modalityspecific nonverbal and verbal representations. All have dynamic processing capacities that serve specific adaptive functions as summarized throughout this article, and which correspond to the uninstantiated van der Maas et al. general-level processing “species.” Thus, in principle, the van der Maas model could be instantiated in DCT terms. 4. Brain correlates of intelligence and DCT This final section reviews neural correlates of psychometric and DCT approaches to intelligence as further evidence on key issues. Of special interest is whether referential processing shows up in general neuropsychological studies of intelligence as it is expected to do in DCT brain-related research. A historical neuropsychological sketch will pave the way to the answer. In the 19th C, verbal processing regions of the brain were inferred from deficits in both expressive and receptive language functions that resulted from damage to the left hemisphere. Comparable injuries to the right hemisphere did not produce similar deficits, prompting the conclusion that the left hemisphere is dominant for language. However, “the scientific world generalized [the left dominance view] to conclude that the left hemisphere was dominant not only for language but also for all psychological processes. The right hemisphere was seen as a mere relay station … an unthinking automaton. From pre-19th century whole-brained creatures, we had become half-brained” (Levy, 1985, p. 38). Levy's analysis thus parallels psychological and philosophical verbal-dominance interpretations of intelligence reviewed above. Neuropsychological interpretations of nonverbal imagery began to emerge in the 1960s when it was discovered that lesions to the right hemisphere selectively impaired performance on various nonverbal tasks (e.g., Milner, 1961). This led to the idea that nonverbal processes are controlled primarily by the right hemisphere just as verbal processes seemed to be controlled by the left. Qualifications arose from evidence that correlates of both verbal and nonverbal intelligence are found in each hemisphere…

involved normal individuals of various ages and participants with brain lesions due to injuries. The intelligence measures included “intelligence in general” as defined by full-scale scores on WAIS as well as Spearman-type g loadings of WAIS subscales or individual tests such as the Raven's Advanced Progressive Matrices and tests of working memory. The brain correlates included location-sensitive structural measures of gray matter volume and white matter integrity along with such functional measures as fMRI. The results showed that a network of regions of the frontal, temporal, and parietal cortex as well as white matter pathways (notably the arcuate fasciculus that connects Broca's and Wernicke's language areas) correlated with individual differences in intelligence and reasoning. The regional brain correlates of higher intelligence and reasoning scores are summarized in their Fig. 1 (reproduced here as Fig. 4) and accompanying descriptions in terms of Brodmann areas. Suggestive relations to separate verbal and nonverbal brain correlates are discussed but none implicate the verbal–nonverbal referential connections emphasized in DCT. This is an inevitable consequence of their reliance on the WAIS as the behavioral definition of intelligence with its lack of functional connections between verbal and nonverbal (performance) abilities. The connections identified in P-FIT mainly involve the arcuate fasciculus, which is classically interpreted as connecting auditory and visual language brain regions. Jung and Haier (2007, p. 141) mentioned that a measure of white matter integrity of the arcuate fasciculus correlated more highly with verbal ability (r = .57 than with nonverbal ability (r = .33), with no reference to brain connections between the two abilities. This gap means that the Jung and Haier interpretation of the brain results does not include the crucial (empiricallysupported) aspect of DCT. A number of peer commentaries in the concluding section of Jung and Haier (2007) also criticized the P-FIT causal emphasis on the brain. Demetriou and Mouyi argued that PFIT has little to say about cognitive development of intelligence. Norgate and Richardson asserted that “bigger” brain areas are not a cause of higher IQ but both are instead the result of social

4.1. General intelligence and the brain The possibility that the two systems interact arises from recent neuroscientific studies of general intelligence inspired by Jung and Haier's (2007) Parieto-Frontal Integration Theory of intelligence (P-FIT). The theory was based on a review of 37 published studies of neural correlates of intelligence that

Fig. 4. Brodmann-area numbered brain regions associated with better performance measures of intelligence and reasoning that define the P-FIT model. Dark circles = predominant left hemisphere associations; white circles = predominant bilateral associations; white arrow = arcuate fasciculus. From Jung and Haier (2007, Fig. 5).

A. Paivio / Intelligence 47 (2014) 141–158

experience. Roring, Nandakopal, and Ericsson noted that intense training can even change structural aspects of the brain. Sternberg emphasized that intelligence does not reside in the brain but rather in the interaction of brain and environment. Such views are consistent with the DCT emphasis on experiential determinants of intelligence but they, too, miss the specific importance of interactive verbal–nonverbal experience for the development of complete intellectual systems. In related research, Gläscher et al. (2010) studied 241 patients with single lesions in various brain locations. General intelligence (g) scores were computed from WAIS scores individually for all patients and compared with g for patients without lesions in those regions. Significant associations were found between g and damage to a circumscribed set of regions located in left frontal and right parietal cortex, as measured by voxel-based lesion-symptom matching. Glascher et al. concluded that the results are consistent with P-FIT. However, their factor analysis “extracted g and three first-order factors (verbal abilities, visuospatial abilities, and working memory)” (Gläscher et al. (2010, p. 682) with no evidence that verbal and visuospatial regions interact. The study did reveal more extensive neural white matter connections than were revealed by Jung and Haier (2007), but their possible relevance to DCT remains indeterminate. Colom et al. (2009) tested P-FIT using 120 healthy university undergraduates rather than brain-lesioned patients. The students were given a battery of nine tests that measured fluid, crystallized, and spatial cognitive abilities, together with structural MRI scans of areas of gray matter volume. It turned out that “pure” estimates of crystallized and spatial intelligence were related only to their specific defining tests and not to general (fluid) intelligence. “Therefore, the findings for [these] estimates speak about verbal … and spatial … intelligence controlling for the pervasive influence of g.” (Colom et al., 2009, p. 130). However, verbal–nonverbal connections that allow for synergistic interactions as in DCT also were not revealed by their results. A further study bears directly on the conceptual status of g. Hampshire, Highfield, Parkin, and Owen (2012) tested a “multiple demand” neural model (e.g., Duncan, 2010) based on activity networks of frontal and parietal regions, of which. P-FIT is cited as an example. A factor analysis was conducted on a battery of 12 intellectual ability tests administered by internet to a large group of participants. The behavioral patterns were compared with factor analytic models of brain organization based on fMRI scans of other participants doing the same tests. The tests drew on “well-established paradigms from the neuropsychology literature [that measure] a range of the types of planning, reasoning, attentional, and working memory skills that are considered akin to general intelligence tests” (Hampshire et al., 2012, p. 1226). The analysis of behavioral results yielded three factors, short-term memory, reasoning, and verbal. Analyses of the fMRI patterns yielded two factors in which reasoning and short-term working memory tasks activated separate sub-regions within the frontal–parietal multiple demand network. Separate analyses also revealed verbal-task brain correlates located outside the multipledemand network. Hampshire et al. concluded that g is “an emergent property of anatomically distinct cognitive systems” rather than a single neural system, which is generally compatible with DCT but

153

excludes the specific DCT emphasis on interconnected verbal and nonverbal systems. However, some parallels to DCT can be found in Hampshire et al. First, their analyses reveal distinct verbal and nonverbal abilities. Second, the verbal side appears as a separate factor that also includes a high loading on the STM digit span task. These are independent of nonverbal STM. Finally, two of the three tasks that load most highly on the Verbal factor clearly implicate referential processing: First, Verbal Reasoning requires participants to indicate whether printed sentences (e.g., “The square is encapsulated by the circle.”) correctly describe a pair of pictured objects. Second, Color Word Mapping is a complex variant of the Stroop test in which participants must indicate which of two colored words describes the print color of a color name that also appears on the page. The mappings of colors to color words may be congruent, incongruent, or doubly incongruent, so that, in DCT terms, the task requires participants to produce the correct referential color name while suppressing the referentially incorrect response. What is unclear in Hampshire et al. (2012) is how the modality specific systems interface with the general MD systems. As noted by one of MD's chief proponents: “MD systems must interface with the many separate brain systems executing specific cognitive operations, for example identification of a specific visual object, or retrieval of specific facts from memory” (Duncan, 2010, p. 78). Duncan concludes that much work is needed to define the separate functions of MD components. 4.2. Neural correlates of DCT Brain research explicitly linking dual coding variables to intelligence began with an experiment on neural correlates of perceptual recognition of nonverbal and verbal stimuli (Paivio & Ernest, 1971). The experiment was motivated by prior demonstrations of left hemisphere superiority in picture recognition, which was attributed to mediation by verbal labeling. Ernest and I used verbal and nonverbal stimuli (alphabetic letters and pictures of objects or geometric forms), which were presented to university students who differed in imagery ability according to a battery of tests that included Space Relations, the Minnesota Paper Form Board, and a questionnaire (published later by Paivio & Harshman, 1983) designed to measure participants' habitual use of verbal and imaginal strategies in thinking. Standardized individual test scores were summed for each subject so that a positive total score defined high imagery ability and a negative score defined low imagery ability. The stimuli were flashed to the left or right visual field, thus initially activating the opposite hemispheres, a procedure commonly used to study hemispheric functional specialization before brain-scanning techniques were perfected. The participants indicated recognition by naming the stimuli. The mean numbers of correct responses for each condition are shown in Fig. 5. Letters yielded only a significant right field (thus left hemisphere) advantage, as typically found for verbal material. Low imagers notably performed very poorly when the forms were presented to the left field, presumably because they depended on name availability, which was attenuated in the right hemisphere. Most importantly for present purposes, pictures were recognized better in both visual fields by participants with high imagery ability than by those with low imagery ability. This result may be the earliest evidence that

154

A. Paivio / Intelligence 47 (2014) 141–158

Fig. 5. Recognition accuracy scores as a function of imagery ability, stimulus attribute, and visual field/hemisphere. Adapted from Paivio and Ernest (1971).

the neural representations (DCT imagens) that generate visual images of familiar objects are available in both hemispheres. The results also support DCT relations to intelligence. Binet and Simon included picture-naming tasks in their IQ test battery and these are in some modern tests (e.g., the Vocabulary subtest of the fifth edition of the Stanford–Binet Intelligence Scale). Moreover, numerous studies have shown that spatial tests of imagery ability load substantially on more general intelligence factors (e.g., see Carroll, 1993, p. 597). Thus the Paivio and Ernest (1971) study anticipated numerous implications of DCT for intellectual performance, especially the referential processing that is necessarily activated in picture naming. Studies comparing processing of concrete (high imageability) and abstract (low imageability) verbal material also are relevant because they differentially implicate referential processing. Kounios and Holcomb (1994) were the first to test and confirm a prediction from DCT that concrete and abstract words would activate spatially separate neural regions. The activation sequence was subsequently revealed in detail by Dhond, Witzel, Dale, and Halgren (2007) using magnetoencephalography (MEG, similar to EEG) to record brain responses while participants made concrete-abstract decisions. Both word types first activated a common back-tofront sequence involving occipital, temporal, parietal, and frontal areas. Initially bilateral, the occipital activity shifted after 135 ms to left occipital areas typically involved in visual word identification, followed within 200 ms by activation of classical language areas, including temperoparietal regions implicated in phonological processing. Later responses predominated in anterior temporal, left prefrontal, and right parietal regions that showed differential responding to abstract and concrete words beginning as early as 350 ms. Left frontotemporal differences suggested a stronger verbal representation for abstract words in that area. Still later, peaking at 550 ms, the brain response was more significant to concrete words in the right medial occipitotemporal and lateral parietal areas, often associated with imagery processing.

Dhond et al. concluded that both concrete and abstract words “may be initially understood using a left lateralized fronto-temporal verbal-linguistic system that for concrete words is followed after a short delay by a right parietal and medial occipital imagistic network” (2007, p. 355). Thus, the results support a DCT interpretation of verbal–nonverbal referential processing that begins with activation of visual logogens for both abstract and concrete words in posterior regions, which then moves to frontal areas, and finally regresses to activation of imagens in posterior regions for concrete words. Concrete-abstract and imageability effects consistent with DCT also emerged from numerous other studies involving words, sentences, or longer passages (e.g., Binder, Westbury, Mckiernan, Possing, & Medler, 2005; Holcomb, Kounios, Anderson, & West, 1999; Just, Newman, Keller, McKelney, & Carpenter, 2004; Mellet et al., 2002; Wallentin, Østergaard, Lund, Østergaard, & Roepstorff, 2005; Welcome, Paivio, McRae, & Joanisse, 2011; West, O'Rourke, & Holcomb, 1998; Xu, Kemeny, Park, Frattali, & Braun, 2005). The notable findings include (a) activation of different brain locations for concrete and abstract material of any length, (b) more frontal verbal areas activated during processing of abstract sentences as compared to more posterior areas for concrete sentences; (c) concreteness and word imageability effects distinguishable in posterior areas 550–800 ms after stimulus presentation, presumably reflecting the time necessary for imagery arousal even to abstract material, and (d) more activation of both hemispheres by concrete than abstract words and sentences. A study by Reichle, Carpenter, and Just (2000) is especially relevant here because it connected DCT variables to cognitive abilities. The researchers used fMRI to investigate cortical activity in participants who differed on tests of verbal and visual–spatial abilities and were given instructions to use either a verbal or a visual strategy to verify whether sentences are true or false relative to pictured scenes. Thus referential processing was directly implicated in the decisions. The verbal strategy produced more activation in Broca's and other

A. Paivio / Intelligence 47 (2014) 141–158

language-related regions, whereas the visual–spatial strategy produced more activation in the parietal cortex and other regions that are typically involved in visual–spatial reasoning. These relations were modified by individual differences so that subjects with better verbal skills had less activation in Broca's area when they used a verbal strategy than when they used a visual–spatial strategy. Conversely, individuals with better visual–spatial skills had less activation in the left parietal cortex when they used the visual–spatial strategy. The pattern of reduced activation was interpreted to mean that skill-compatible strategies helped to minimize cognitive overload. Simply put, the task was easy for brain regions with efficient verbal or nonverbal processing systems when primed by appropriate cues, especially in the context of the referential processing demands of the sentence–picture verification task. Remarkable brain evidence for referential imagery effects was found by Owen et al. (2006) using fMRI scans of a brain damaged patient in a vegetative state who showed no evidence of conscious awareness according to behavioral criteria. The patient was instructed alternately to produce either spatial imagery of moving around a house or motor imagery of tennis being played. The results in Fig. 6 show that the spatial imagery instructions activated areas in the parahippocampal gyrus, posterior parietal cortex, and premotor cortex whereas the tennis instructions activated the supplementary motor cortex. The activation patterns were indistinguishable statistically from those obtained from normal control participants given the same instructions. Monti, Coleman, and Owen (2009) concluded that “It is impossible to explain these results without accepting that this patient retained the ability to comprehend verbal instructions, to remember them from the time they were given them before the scan began to the appropriate time during the scan itself, and to act on those instructions, thereby

155

willfully producing mental/neural states.” (Monti et al., 2009, p. 86). By definition, these brain activity patterns reflect complex referential processing. We conclude with a neural version of our thematic issue, namely the location and nature of the brain's referential processing systems. The question evolved from the concept of neural association systems in general with special emphasis on white matter pathways that connect different gray matter regions. Associative brain activity could also involve other mechanisms such as similarity matching between perceptual and memory trace patterns, and biochemically-based cell signaling between gray matter areas. Here we consider only white matter pathways. Historically, the arcuate fasciculus was first viewed as the major neural connector for language-related brain areas, especially as a one-way path from Wernicke's receptive auditory “word-form area” to Broca's expressive “motor word form area.” This interpretation evolved over the years to include bi-directional connections involving the arcuate fasciculus and other cortical and subcortical pathways (e.g., Matsumoto et al., 2004). The traditional emphasis, however, focused on the role of these connections in language phenomena until the very recent attention to DCT-defined verbal–nonverbal connections involved in, for example, object (or picture)-name relations. A plausible brain correlate of referential processing ability and intelligence was revealed by Label and Beaulieu (2009) using diffusion tensor imaging and tractography. The research focused on lateralization of the arcuate fasciculus in 68 healthy children aged 5–13 years, who also underwent cognitive assessments that included the Peabody Picture Vocabulary Test (PPVT). Some children had trackable arcuate fasciculus located only in the left hemisphere, and others were relatively more left-lateralized or right-lateralized. The pertinent result

Fig. 6. Neural regions activated by instructions to imagine playing tennis and moving around a house are statistically indistinguishable for a vegetative state patient and healthy controls. From Owen et al. (2006).

156

A. Paivio / Intelligence 47 (2014) 141–158

here was that the group whose arcuate fasciculus was trackable only in the left hemisphere out-performed the right-hemisphere lateralized group. In DCT terms, this demonstrates a specific white-matter correlate of a referential processing ability (picture–name matching) appropriately localized in the left hemisphere. Referential processing, however, also occurs across hemispheres, as inferred for example from Norman Geschwind's (1965) classic summary of neuropsychological studies of disconnection syndromes. He described visual and other modalities of agnosias as “modality-specific naming defects resulting from the isolation of the primary sensory cortex from the speech area {Geschwind, 1965, p. 641). He discussed visual imagery deficits similarly, concluding that “Loss of the ability to describe an absent scene might reasonably result from a disconnexion of the visual regions from the speech area” (p. 603). In some patients, such agnosias could be attributed to lesions of the corpus callosum that connects the two hemispheres, thus providing evidence that intact DCT referential connections had existed in that anatomical structure. We are left with the hope that the availability of the new and efficient measures of white matter tracts will eventually result in a flood of experiments that will reveal the various modalities and types of pathways that bridge the functional gap that has plagued the verbal–nonverbal solitudes of traditional theories of intelligence. References Aitchison, J. (2003). Words in mind: An introduction to the mental lexicon (3rd ed.). Kindon: Blackwell. Arnell, K. M., Joanisse, M. F., Klein, R. M., Busseri, M. A., & Tannock, R. (2009). Decomposing the relation between rapid automatized naming (RAN) and reading ability. Canadian Journal of Experimental Psychology, 63, 173–184. Arnheim, R. (1969). Visual thinking. Berkeley & Los Angeles: University of California Press. Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660. Begg, I. (1972). Recall of meaningful phrases. Journal of Verbal Learning and Verbal Behavior, 11, 431–439. Bell, N. (1991a). Gestalt imagery: A critical factor in language comprehension. Annals of Dyslexia, 41, 246–260. Bell, N. (1991b). Visualizing and verbalizing for language comprehension and thinking. Paso Robles, CA: NBI Publications. Biederman, I. (1987). Recognition-by-components: A theory of human image understanding. Psychological Review, 94, 115–147. Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009). Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex, 19, 2767–2796. Binder, J. R., Westbury, C. F., Mckiernan, K. A., Possing, E. T., & Medler, D. A. (2005). Distinct brain systems for processing concrete and abstract language. Journal of Cognitive Neuroscience, 17, 905–917. Binet, A., & Simon, T. (1905a). Méthodes nouvelles pour le diagnostic du niveau intellectual des anormaux. L'Année Psychologique, 11, 191–336. E. Kite (1916) translation. Binet, A., & Simon, T. (1905b). The development of intelligence in children. Baltimore, MD: Williams & Wilkins (Retrieved 12/4/ 2010 from http://psychclassics.yorku.ca,/Binet/binet.htm). Boake, C. (2002). From the Binet–Simon to the Wechsler–Bellevue: Tracing the history of intelligence testing. Journal of Clinical and Experimental Neuropsychology, 24, 383–405. Bonner, A. (2007). The art and logic of Ramon Llull. A user's guide. Leiden: Brill. Bower, G. H., Clark, M. C., Lesgold, A. M., & Winzenz, D. (1969). Hierarchical retrieval schemes in recall of categorized word lists. Journal of Verbal Learning and Verbal Behavior, 8, 323–343. Bucci, W. (1984). Linking words and things: Basic processes and individual variations. Cognition, 17, 137–153. Bucci, W., & Freedman, N. (1978). Language and hand: The dimension of referential competence. Journal of Personality, 46, 594–622. Bucci, W., & Kabasakalian-McKcbeay (1992). Scoring referential activity: Instructions for use with transcripts of spoken narrative texts. Ulm, Germany: Ulmer textbank.

Buckley, K. W. (1989). Mechanical man: John Broadus Watson and the beginnings of behaviorism. New York: Guilford. Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press. Carvajal, H., Nowak, S. J., Fraas, A., & McConnell, M. A. (2000, June). Saving time: Using the Peabody Picture Vocabulary Test-III as a screening test of intelligence. College Student Journal, 34(2). Cattell, R. B. (1963). Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology, 54, 1–22. Chronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671–684. Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82, 407–428. Colom, R., Haier, R. J., Head, K., Alvarez-Linera, J., Quiroga, M. A., Shi, P. C., et al. (2009). Gray matter correlates of fluid, crystallized, and spatial intelligence: Testing the P-FIT model. Intelligence, 37, 124–135. Coltheart, M. (2004). Are there lexicons? The Quarterly Journal of Experimental Psychology, 57, 1153–1171. Crutch, S. J., & Warrington, E. K. (2005). Abstract and concrete concepts have structurally different representational frameworks. Brain, 128, 615–627. Crutcher, R. J., & Beer, J. M. (2011). An auditory analog of the picture superiority effect. Memory & Cognition, 39, 63–74. Dhond, R. P., Witzel, T., Dale, A. M., & Halgren, E. (2007). Spatiotemporal cortical dynamics underlying abstract and concrete word reading. Human Brain Mapping, 28, 355–362. Divjak, D., & Gries, S. T. (2008). Clusters in the mind? Converging evidence from near synonymy in Russian. The Mental Lexicon, 3, 188–213. Duncan, J. (2010). The multiple-demand (MD) system of the primate brain: Mental programs for intelligent behaviour. Trends in Cognitive Sciences, 14, 172–179. Ebbinghaus, H. (1964). Memory: A contribution to experimental psychology. New York: Dover (Original work published 1885). Elman, J. L. (2004). An alternative view of the mental lexicon. Trends in Cognitive Sciences, 8, 301–305. Ericsson, K. A. (2003). The acquisition of expert performance in problem solving: Construction and modification of mediating mechanisms through deliberate practice. In J. E. Davidson, & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 31–83). Cambridge, UK: Cambridge University Press. Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 107, 211–245. Garner, W. R. (1974). The processing of information and structure. Potomac, MD: Lawrence Erlbaum Associates. Geschwind, N. (1965). Disconnection syndromes in animals and man. Brain, 88(Part III), 585–644. Gläscher, J., Rudrauf, D., Colom, R., Paul, I. K., Tranel, D., Damasin, H., et al. (2010). Distributed neural system for general intelligence revealed by lesion mapping. Proceedings of the National Academy of Sciences of the United States of America, 107, 4705–4709. Goel, V., & Dolan, R. J. (2001). Functional anatomy of three-term relational reasoning. Neuropsychologia, 39, 901–909. Gross, A. G., & Harmon, J. E. (2014). Science from sight to insight: How scientists illustrate meaning. Chicago: The University of Chicago Press. Guilford, J. P., & Hoepfner, R. (1971). The analysis of intelligence. New York: McGraw-Hill. Guttman, L. (1965). A faceted definition of intelligence. Studies in Psychology: Scripta Hierosolymitana, 14. (pp. 166–1811). Jerusalem: Hebrew University, 166–1811. Hampshire, A., Highfield, R. R., Parkin, B. L., & Owen, A. M. (2012). Fractionating human intelligence. Neuron, 76, 1225–1237. Healy, W., & Fernald, G. M. (1911). Tests for practical mental classification. Psychological Review Monograph, 13 (Whole no. 54). Holcomb, P. J., Kounios, J., Anderson, J. E., & West, W. C. (1999). Dual-coding, context-availability, and concreteness effects in sentence comprehension: An electrophysiological investigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 721–742. Horn, J. L., & Cattell, R. B. (1966). Refinement and test of the theory of fluid and crystallized intelligence. Journal of Educational Psychology, 57, 253–270. Jared, D., Poh, R. P. Y., & Paivio, A. (2013). L1 and L2 picture naming in Mandarin–English bilinguals: A test of Bilingual Dual Coding Theory. Bilingualism: Language and Cognition, 16(Special Issue 02), 383–396. Johnson, C. J., Paivio, A., & Clark, J. M. (1996). Cognitive components of picture naming. Psychological Bulletin, 120, 113–139. Jung, R. E., & Haier, R. J. (2007). The parieto–frontal integration theory (P-FIT) of intelligence: Converging neuroimaging evidence. Behavioral and Brain Sciences, 30, 135–187. Just, M. A., Newman, S. D., Keller, T. A., McKelney, A., & Carpenter, P. A. (2004). Imagery in sentence comprehension: An fMRI study. NeuroImage, 21, 112–124.

A. Paivio / Intelligence 47 (2014) 141–158 Kaufman, A. S., & Kaufman, N. L. (1997). The Kaufman Adolescent and Adult Intelligence Test. In D. P. Flanagan, J. L. Genshaft, & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 209–229). New York: Guilford. Kiss, G. R. (1975). An associative thesaurus of English: Structural analysis of a large relevance network. In A. Kennedy, & A. Wilkes (Eds.), Long term memory (pp. 103–121). New York: Academic Press. Knauff, M., Fangmeier, T., Ruff, C. C., & Johnson-Laird, P. N. (2003). Reasoning, models, and images: Behavioral measures and cortical activity. Journal of Cognitive Neuroscience, 15, 559–573. Knauff, M., Mulack, T., Kassubek, J., Salih, H. R., & Greenlee, M. W. (2002). Spatial imagery in deductive reasoning: A functional MRI study. Brain Research. Cognitive Brain Research, 2002(2), 203–212. Kosslyn, S. M., Van Kleeck, M. H., & Kirby, K. M. (1990). A neurologically plausible model of individual differences in visual mental imagery. In P. J. Hampson, D. F. Marks, & J. T. F. Richardson (Eds.), Imagery: Current: Developments (pp. 39–77). London: Routledge. Kounios, J., & Holcomb, P. J. (1994). Concreteness effects in semantic processing: ERP evidence supporting dual-coding theory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 804–823. Krueger, T. H. (1976). Visual imagery in problem-solving and scientific creativity. Derby, CT: Seal Press. Kunda, M., McGreggor, K., & Goel, A. (2012). Reasoning on the Raven's Advanced, Progressive Matrices Test with Iconic Visual representations. Proceedings of the Cognitive Science Society, Sapporo, Japan (pp. 1828–1833). Label, C., & Beaulieu, C. (2009). Lateralization of the arcuate fasciculus from childhood to adulthood and its relation to cognitive abilities in children. Human Brain Mapping, 30, 3563–3573. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240. Langacker, R. W. (1991). Foundation of cognitive grammar 2. Stanford, CA: Stanford University Press. Levy, J. (1985, May). Right brain, left brain: Fact and fiction. Psychology Today, 38–44. Lima-Silva, T. B., Ordonez, T. N., dos Santos, Dias, et al. (2010). Effects of cognitive training on metamemory and mental images. Dementia Neuropsychology, 2, 114–119. Lockhead, G. R. (1972). Processing dimensional stimuli. Psychological Review, 79, 410–419. Lynn, R., Allik, J., & Irwing, P. (2004). Sex differences on three factors identified in Raven's Standard Progressive Matrices. Intelligence, 32, 411–424. Marschark, M. (2007). Raising and educating a deaf child (2nd ed.). New York: Oxford. McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1–10. Mellet, E., Bricogne, S., Crivello, F., Mazoyer, B., Denis, M., & Tzourio-Mazoyer, N. (2002). Neural basis of mental scanning of a topographic representation built from a text. Cerebral Cortex, 12, 1322–1330. Miller, G. A. (1956). The magic number seven, plus or minus two: Some limits of our capacity for processing information. Psychological Review, 63, 81–97. Milner, B. (1961). Interhemispheric differences in the localization of psychological processes in man. British Medical Bulletin, 27, 272–277. Milner, P. M. (1999). The autonomous brain: A Neural Theory of Attention and Learning. Mahatma, NJ: Erlbaum. Monti, M. M., Coleman, M. R., & Owen, A. M. (2009). Neuroimaging and the vegetative state: Resolving the behavioral assessment dilemma. Disorders of Consciousness: Annals of the New York Academy of Science, 1157, 81–89. Morton, J. (1979). Facilitation in word recognition: Experiments causing change in the logogen model. In P. A. Kolers, M. Wrolstead, & H. Bouma (Eds.), Processing of Visible Language, 1, New York: Plenum Press. Murdoch, S. (2007). IQ: A smart history of a failed idea. Wiley. Owen, A. M., Coleman, M. R., Boly, M., Davis, M. S., Laureys, S., & Pickard, J. D. (2006). Detecting awareness in the vegetative state. Science, 313, 1402. Paivio, A. (1969). Mental imagery in associative learning and memory. Psychological Review, 76, 241–263. Paivio, A. (1971). Imagery and verbal processes. New York: Holt, Rinehart, & Winston. Paivio, A. (1978a). Comparison of mental clocks. Journal of Experimental Psychology: Human Perception and Performance, 4, 61–71. Paivio, A. (1978b). The relationship between verbal and perceptual codes. In E.C. Carterette, & M. P. Friedman (Eds.), Handbook of perception. Perceptual processing, Vol. IX. (pp. 113–131). New York: Academic Press. Paivio, A. (1980). On weighing things in your mind. In P. W. Jusezyk, & T. W. Klein (Eds.), The nature of thought: Essays in honor of D. O. Hebb (pp. 133–154). Hillsdale, N.J.: Lawrence Erlbaum Associates. Paivio, A. (1983). The empirical case for dual coding. In J. C. Yuille (Ed.), Imagery, memory and cognition: Essays in honor of Allan Paivio (pp. 307–332). Hillsdale, N. J.: Lawrence Erlbaum Associates.

157

Paivio, A. (1986). Mental representations: A dual coding approach. New York: Oxford University Press. Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45, 255–287. Paivio, A. (2007). Mind and its evolution: A dual coding theoretical approach. New York: Psychology Press (Previously published by Lawrence Erlbaum Associates). Paivio, A. (2010). Dual coding theory and the mental lexicon. The Mental Lexicon, 5, 205–230. Paivio, A. (2014). Bilingual dual coding theory. In R. R. Heredia, & J. Altarriba (Eds.), Foundations of bilingual memory (pp. 41–62). New York: Springer. Paivio, A., & Begg, I. (1981). Psychology of language. Englewood Cliffs, NJ: Prentice Hall. Paivio, A., Clark, J. M., Digdon, N., & Bons, T. (1989). Referential processing: Correlates of naming pictures and imaging to words. Memory & Cognition, 17, 163–174. Paivio, A., & Desrochers, A. (1980). A dual-coding approach to bilingual memory. Canadian Journal of Psychology, 34, 390–401. Paivio, A., & Ernest, C. (1971). Imagery ability and visual perception of verbal and nonverbal stimuli. Perception & Psychophysics, 10, 429–432. Paivio, A., & Foth, D. (1970). Imaginal and verbal mediators and noun concreteness in paired-associate learning: The elusive interaction. Journal of Verbal Learning and Verbal Behavior, 9, 384-290. Paivio, A., & Harshman, R. A. (1983). Factor analysis of a questionnaire on imagery and verbal habits and skills. Canadian Journal of Psychology, 37, 461–483. Paivio, A., & Lambert, W. (1981). Dual coding and bilingual memory. Journal of Verbal Learning and Verbal Behavior, 20, 532–539. Paivio, A., & Sadoski, M. (2011). Lexicons, contexts, events, and images: Commentary on Elman (2009) from the perspective of dual coding theory. Cognitive Science, 35, 198–209. Paivio, A., & Steeves, R. (1967). Relations between personal values and the imagery and meaningfulness of value words. Perceptual and Motor Skills, 24, 357–358. Pintner, R., & Paterson, D. G. (1917). A scale of performance tests. New York: Appleton. Pylyshyn, Z. (1973). What the mind's eye tells the mind's brain: A critique of mental imagery. Psychological Bulletin, 80, 1–24. Reichle, E. D., Carpenter, P. A., & Just, M. A. (2000). The neural basis of strategy and skill in sentence–picture verification. Cognitive Psychology, 40, 261–295. Richardson, J. T. E. (1978). Reported mediators and individual differences in mental imagery. Memory & Cognition, 6, 376–378. Richardson, J. T. E. (2011). Howard Andrew Knox: Pioneer of intelligence testing at Ellis Island. New York: Columbia University Press. (S. Clucas trans.). Rossi, P. (2000). Logic and the art of memory: The quest for a universal language. Chicago: University of Chicago Press. Rumelhart, D. E. (1979). Some problems with the notion that words have literal meanings. In A. Ortony (Ed.), Metaphor and thought (pp. 71–82). Cambridge: Cambridge University Press. Sadoski, M., & Paivio, A. (2013). Imagery and text: A dual coding theory of reading and writing (2nd ed.). New York: Routledge. Sadoski, M., & Willson, V. L. (2006). Effects of a theoretically based large-scale reading intervention in a multicultural urban school district. American Educational Research Journal, 43(137), 154. Schultheiss, O. C., & Strasser, A. (2012). Referential processing and competence as determinants of congruence between implicit and explicit motives. In S. Vazire, & T. D. Wilson (Eds.), Handbook of self-knowledge (pp. 39–62). New York: Guilford. Snow, R. E., Kyllonen, P. C., & Marshalek, B. (1984). The topography of ability and learning correlations. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence, Vol. 2. (pp. 47–103). Hillsdale, NJ: Erlbaum. Sternberg, R. J. (1997). The concept of intelligence and its role in lifelong learning and success. American Psychologist, 52, 1030–1037. Strømnes, F. J. (2006). The fall of the word and the rise of the mental model: A reinterpretation of the recent research on the use of language and spatial cognition. Frankfurt-am Main: Peter Lang. Thomas, N. J. T. (1989). Experience and theory as determinants of attitudes toward mental representation: The case of Knight Dunlap and the vanishing images of J. B. Watson. American Journal of Psychology, 102, 395–412. Thompson, R. F., & Madigan, S. A. (2007). Memory: The key to consciousness. NJ: Princeton University Press. Thompson, V., & Paivio, A. (1994). Memory for pictures and sounds: Independence of auditory and visual codes. Canadian Journal of Experimental Psychology, 48, 380–398. Thurstone, L. L. (1938). Primary mental abilities. Chicago: Education Industry Service. Tulving, E. (1983). Elements of episodic memory. New York: Oxford University Press.

158

A. Paivio / Intelligence 47 (2014) 141–158

van den Bos, K. P., Zijlstra, B. J. H., & Spelberg, H. C. (2002). Life-span data on continuous-naming speeds of numbers, letters, colors, and pictured objects, and word-reading speed. Scientific Studies of Reading, 6, 25–49. van der Maas, H. L. J., Dolan, C. V., Grasman, R. P. P. P., Wicherts, J. M., Huizenga, H.M., & Raijmakers, M. E. J. (2006). A dynamical model of general intelligence: The positive manifold of intelligence by mutualism. Psychological Review, 113, 842–861. Vernon, P. E. (1961). The structure of human abilities (2nd ed.). London: Methuen. Vernon, P. A. (1983). Speed of information processing and general intelligence. Intelligence, 7, 53–70. Wallentin, M., Østergaard, S., Lund, T. E., Østergaard, L., & Roepstorff, A. (2005). Concrete spatial language: See what I mean? Brain & Language, 92, 221–233. Watson, J. B. (1930). Behaviorism. Chicago: University of Chicago Press. Wechsler, D. (1939). Measurement of adult intelligence. Baltimore: MD Williams & Wilkins. Welcome, S., Paivio, A., McRae, K., & Joanisse, M. (2011). A study of concreteness effects in imagery and associate generation: Evidence for the dual coding theory. Journal of Experimental Brain Research, 212, 347–358.

Weldon, M. S., & Roediger, H. L. (1987). Altering retrieval demands reverses the picture superiority effect. Memory and Cognition, 15, 269–280. West, W. C., O'Rourke, T. B., & Holcomb, P. J. (1998). Event related brain potentials and language comprehension: A cognitive neuroscience approach to the study of intellectual functioning. In S. Soraci, & W. J. McIlvane (Eds.), Perspectives on fundamental processes in intellectual functioning (pp. 133–168). Stamford, CT: Ablex. Wintermute, S., & Laird, J. E. (2009). Imagery as compensation for an imperfect abstract problem representation. Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1–6). Woolley, G. (2011). Reading comprehension: Assisting children with learning difficulties. New York: Springer. Yates (1966). The art of memory. London: Routledge & Kegan Paul. Yerkes, R. M. (Ed.). (1921). Psychological examining in the United States Army. Memoirs of the National Academy of Sciences, No. 15.