- Email: [email protected]
From Brain–Environment Connections to Temporal Dynamics and Social Interaction: Principles of Human Brain Function
Neuron
Perspective From Brain–Environment Connections to Temporal Dynamics and Social Interaction: Principles of Human Brain Function Riitta Hari1,* 1Department of Art, Aalto University, PO Box 31100, 00076 AALTO, Helsinki, Finland *Correspondence: [email protected] http://dx.doi.org/10.1016/j.neuron.2017.04.007
Experimental data about brain function accumulate faster than does our understanding of how the brain works. To tackle some general principles at the grain level of behavior, I start from the omnipresent brainenvironment connection that forces regularities of the physical world to shape the brain. Based on topdown processing, added by sparse sensory information, people are able to form individual ‘‘caricature worlds,’’ which are similar enough to be shared among other people and which allow quick and purposeful reactions to abrupt changes. Temporal dynamics and social interaction in natural environments serve as further essential organizing principles of human brain function. Brain in the World In a recent small workshop (Dynamical Systems Interdisciplinary Network workshop, 2016), all speakers were challenged to give their answers to the question ‘‘how does the brain work.’’ The current paper is based on the ideas presented in my introductory talk of that workshop, aiming to look at some overarching principles of brain function at the grain level of behavior and to emphasize the importance of both temporal processing and social interaction as topics for further research. Although many of the discussed concepts are not new, they are not shared by all scientists and, in some aspects, the separation of views is deep. Thus, it seems worthwhile to open the can of worms, even at the risk of repetition and too strong simplifications. If we consider the heart as a pump and the liver as a filter, then the brain can be said to be a device that predicts the future on the basis of the past. To go further, it may be useful, or even necessary, to think along the lines of Theodosius Dobzhansky (1900– 1975), who noted that ‘‘nothing in biology makes sense except in the light of evolution.’’ During evolution, the environment has imprinted itself very strongly to the structure and function of the brain. The most evident examples are the omnipresent circadian rhythms driven by and adjusted to the day/night cycle on the Earth. More generally, many functioning principles of the brain can be understood on the basis of the regularities of the world, although such an approach is seldom taken. Thus, let’s, as an example, consider the macroscopic physical law that describes the movements of objects in the world and the corresponding organization principles in the brain. Newton’s second law, schematically presented in Figure 1A, tells that force is equal to mass times acceleration (F = ma). In other words, the same force will give higher acceleration to light objects compared with heavy objects. For example, the wind moves the (spatially) smaller and thinner branches of a tree faster (i.e., with higher temporal frequency or shorter temporal scales) than the (spatially) larger branches that move relatively more slowly. Accordingly, one would expect that small spatial windows (receptive fields) in the (visual) brain are associated with
more narrow temporal windows. In other words, the high spatial and temporal frequencies should coincide, and in fact, the spatial receptive fields are the smallest at the early projection cortices, increasing further away along the processing stream, as is indicated schematically in Figure 1B. Moreover, brain imaging experiments, using both functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG), have indicated that the time windows of processing get longer (corresponding to lower temporal frequencies) as the spatial receptive windows increase, as is also indicated in Figure 1B; similar organization has been observed both in visual and auditory processing streams (Hari and Parkkonen, 2015; Hari et al., 2010; Hasson et al., 2008; Lerner et al., 2011). Thus, because of the tight connections of temporal and spatial scales in the physical world, the temporal and spatial scales should be—and are—linked in the brain as well. Many other organizational principles of the brain can be understood from the connection of the brain to the world. For example, the frequencies of brain rhythms are surprisingly similar in different mammals independently of the brain size varying by a factor of 17,000 (Buzsa´ki et al., 2013). This intriguing similarity of brain-rhythm frequencies in different species can be tracked back to the organism-world interface: the rhythmicity was suggested to be related, at least in part, to muscle-contraction mechanisms, especially to the time constants of myosin and actin filaments that have remained similar across the mammals (Buzsa´ki et al., 2013). In general, the motor system is central to brain function from simple movements to cognition, even so that—to state it boldly—during evolution, minds have emerged only to self-moving organisms. It has been essential to act in the world, to change it, and to actively search for perceptual information. Accordingly, the individual’s active involvement in the actions is often decisive for shaping the brain circuitries. For example, a rake in the hand of a monkey changes the brain’s visuo-tactile receptive fields only if the monkey actively uses the tool (Iriki et al., 1996). Neuron 94, June 7, 2017 ª 2017 Elsevier Inc. 1033
Neuron
Perspective A
B
Figure 1. How Movement of Objects in the World May Have Affected Brain Organization during Evolution (A) Schematic description of the Newton’s second law (force = mass 3 acceleration; F = ma) applied to the effects of wind on a tree and the resulting relationship between the temporal and spatial frequencies of various points of the tree, here indicated by the numbers 1–3 from thin branches to the trunk. (B) Schematic presentation of the increase of visual (spatial) receptive fields when progressing from the primary projection area to higher levels of the processing hierarchy; at the same time, the temporal processing windows get broader (temporal frequency is decreasing). The numbers 1–3 correspond to those in (A).
Action–perception (A–P) loops bridge sensory inputs with the sensory predictions associated with the actions (Hari and Kujala, 2009). Mirroring—activation of motor circuitries also during observation of other persons’ actions (for a review, see Rizzolatti and Craighero, 2004)—requires similarities between low-level brain activity across individuals. It is possible that the sensorimotor transformations required by both own and observed actions, as well as the control of sequential actions, are reflected in the relatively larger size of the cerebellum with respect to the neocortex in great apes and humans compared with non-apes (Barton and Venditti, 2014). Repeated activation of the A–P loop transforms to habits— automatized behavior—and many actions, originally directed to the outside world, can be internalized to thoughts (Hesslow, 2002). An illustrative example is the use of the abacus (called soroban in Japanese). When the young soroban wizards participate in competitions of mental calculation, where they are no more allowed to use the soroban, their fingers still move across the thin air with movements that they used to do while moving the beads of the soroban. Within the context of the A–P loop—which includes the brain, body, and the environment—it is easy to understand the important role of the physical environment, other people, and culture on all human behavior, including thinking. Both the physical environment and other people can act as memory surfaces to support cognition and memory, which becomes especially important during old age memory decline. Humans search for continuous stimulation and novelty, almost like they crave for nutrition, and during sensory deprivation—when the person is kept within a sound-proof, dimly lit room, even floating in water—thoughts start to stagger and deteriorate. The omnipresent brain–environment connection can explain why psychotic delusions have contents closely associated to the contemporary society and technology, thereby varying across centuries, and why artistic expressions follow more the cultural traditions than the regularities of brain function or environment-independent imagery. Another example is depression, which is treated best when antidepressant drugs are combined with environmental changes. Moreover, a ‘‘smart’’ environment helps and supports behavior, and one may feel quite stupid when the mobile phone with all its address information is lost or when one cannot turn the water tap on for warm water in the hotel bathroom. 1034 Neuron 94, June 7, 2017
Thus, it seems evident that if we study only isolated brains, the results may not be relevant for understanding the brain basis of natural behavior. Some delighting and instructive examples are presented in papers asking ‘‘Can a biologist fix a radio?’’ (Lazebnik, 2002) or ‘‘Can a neuroscientist understand a microprocessor?’’ (Jonas and Kording, 2017). Both of these exercises gave rather disappointing results, probably because the environmental connections and developmental and evolutionary histories of the systems under study were not included in the analyses. Importantly, the brains (and their owners) are interactive, not only reactive, although often studied in unidirectional stimulus– response setups. We have argued earlier that social interaction, as such, is central for the whole human brain function (Hari et al., 2015; Hari and Kujala, 2009). Below, the pre-requisites for quick reactions and social interaction are discussed, noting that the brain architecture is a compromise between rapid temporal signaling (providing time markers) and slower, but denser, signaling (needed for detailed feature analysis). People are able to react fast and purposefully because they already have abundant information (‘‘priors’’) about the world and because they are able to obtain a gist of the surroundings with a single glance. Timing of Behavior and the Corresponding Brain Function The behaviorally relevant timescales range from under 1 ms (for directional hearing) to hundreds of milliseconds (for reaction times, attentional modulations, short-term memory, etc.); for reviews of timing in the brain, see Hari et al. (2010, 2015). For survival, some reactions need to be as quick as the sensory and motor pathways and the brain can allow. However, in general, the brain is very sluggish: compared with copper wires, the conduction speeds of mammal nerve fibers are slower by a factor of 107. Time needed for processing of sensory input would shorten if the communication between involved brain areas would be faster, as would happen if the connecting fibers would be thicker. Such a design would, however, result in serious packaging problems within the brain and skull (Ringo et al., 1994). Instead, the nature has a clever solution for this problem so that only a small proportion of all fibers is scaled according to the brain size. For example, the fibers in corpus callosum are about the same diameters (and thus have about the same conduction velocities) in
Neuron
Perspective mammals of different sizes, resulting in humans to inter-hemispheric delays of 5–300 ms (Aboitiz et al., 1992). However, in a small proportion of the thickest fibers, the diameters scale with brain size. In other words, the larger the brain, the faster (and thicker) axons are in a small proportion of all fibers, so that the inter-hemispheric conduction delays can stay within 5 ms in mammal brains of all sizes (Wang et al., 2008). This scaling of only a part of the fibers allows the fast, thick myelinated fibers (that take much space) to connect far-away areas with good temporal accuracy and to provide time markers for processing. The shorter-distance connections occur via dense, thin (and therefore slow) myelinated and unmyelinated fibers. What Can We See with a Single Glance? To react purposefully in urgent situations, we need to quickly evaluate the scene and the items to which to react. If a person notices a snake on a path that she is walking barefooted, she will react automatically, readjusting the trajectory of her foot— possibly prolonging the step—within a fraction of a second, even before the snake as an object has been carefully analyzed and consciously perceived. The visceral feelings of fright will come later. But how can such a purposeful reaction happen so fast although an average reaction time for a simple light stimulus is of the order 200 ms and prolongs considerably when choices have to be made between multiple options? Recognition of a moving elongate ‘‘something’’ as a snake likely requires much more analysis than does a two-choice reaction-time task, even though humans seem to have evolved to be oversensitive to any moving objects: it is better to overestimate the possible dangers and escape—even when it was unnecessary—than vice versa. Such quick purposeful reactions to complex stimuli challenge the traditional view about the exploration of the visual world, which assumes humans to be ‘‘foveal animals’’ who sample the world in a discontinuous snapshot manner by bringing objects and other targets into foveal vision and thereby into the center of attention. Vision is lost between the fixations because of saccadic suppression. Thus, one is assumed to be viewing and seeing the world through a hole the size of the fovea, cumulating the percept of the whole scene from sequential pinhole images, each collected during a fixation lasting a few hundred milliseconds. Thus, the experienced richness of the visual scene is surprising, Similarly, the immediate feeling of the ambience of a new place, for example, a cathedral, seems difficult to explain even with a multisensory ‘‘foveal-animal’’ approach, and consequently, the whole experience of the rich ambience has been claimed to be an illusion (Dennett, 1991). However, both behavioral and computational evidence indicate that people can recognize, with a single glance, the ‘‘gist’’—the essence—of the global spatial layout of the surroundings, even when the stimulus is presented only for 30 ms (Oliva and Torralba, 2006). It is unlikely that foveal vision could explain such quick gists that would need several successive fixations, as discussed above. Gathering visual input via fovea only would further implicate that the retina outside the fovea, which is outside the central 2 arc degrees, would be saved mainly for night vision, which sounds wasteful and therefore an improbable solution to serve human behavior. Thus, it is assuring that
increasing evidence indicates the importance of peripheral vision and that people seem to rely on it much more than is commonly acknowledged (Rosenholtz, 2016). Although visual acuity is lower in peripheral than foveal vision, the peripheral visual vision can—by integrating perceptual information over sizeable areas of the visual field—provide powerful summary statistics about various objects and orientations, luminance, and other physical features (Rosenholtz, 2016). For example, summary statistics of shape information from the peripheral retina would explain how a person who fixates the eyes to the center of a screen at the time of a scene-image appearance, tends to direct the first saccade to human faces and other social stimuli located clearly outside the foveal vision (Henriksson et al., 2016). Top-down effects—e.g., from the frontal lobe (Bar et al., 2006)—are expected to help the detection and recognition of relevant objects. How Complex Is Our Environment? Both the brain and the environment are extremely complex, with several hierarchical temporospatial levels of organization. Therefore, the neuroscientists’ attitude in trying to understand how the brain works has been to first explore the simple and small-scale (microscopic or neuronal level) principles and then (one day) to derive from them the large-scale behavioral principles. One example is the description of natural images as Gabor pyramids that provide unambiguous representations of natural images by specifying all possible spatial frequencies and orientations at each pixel of the scene image. Such computational representations of natural images have been successfully used in the decoding of vision-related brain states (Kay et al., 2008), with implicit assumptions that the percepts might rely on a corresponding summed activity of feature-specific neurons. The neurophysiological generalizations from the Gabor pyramid approaches would thus assume that we can—and should—build complex functions from simple elements. This is certainly a feasible approach, but trying to understand behavior as the summed activity of single neurons and action potentials seems equally difficult as it is to explain the formation of sand dunes on the basis of the properties of the single sand grains or to explain the behavior of gas on the basis of molecular dynamics only, without introducing the (emergent) property of pressure. Importantly, at each new emergence level—be it in the brain or in the environment—the description will get simpler (and not more complex) than at the lower level (compare computations with gas pressure versus the associated molecular dynamics). Indeed, the dimensionality of a naturalistic scene is lower than that of the corresponding pixel-by-pixel Gabor pyramid representation because of the clear structure (e.g., continuities, labels, objects, buildings, trees, etc.) of the natural environment. In other words, the natural environment is not random and the neighboring points of the scene are not independent (Chandler and Field, 2007) in contrast to what is assumed in the Gabor pyramid descriptions of scenes. Moreover, the objects of the real world move according to the laws of physics that people are accustomed to (such as gravitation, object movement, and behavior of light). That the abundant pre-information of the environment affects the perception becomes evident from many Neuron 94, June 7, 2017 1035
Neuron
Perspective A
B
C
Figure 2. The Idea of the Caricature Worlds Three caricature face images are presented as if they would be examples of caricature worlds of three different viewers. The images (A)–(C) are distorted in different manner but may still allow the viewer to identify the person who has inspired them.
illusions where, for example, the shading is interpreted to indicate concave or convex shapes, in accordance with light coming from above (from the sun). Thus, we can conclude that although the world is very complex, it is well structured and thus not so difficult to understand as it would be if comprised of totally independent elements. Dimension reduction—description in terms of a few summarizing features—can strikingly decrease the complexity of both the physical world and brain functions (Pang et al., 2016). Since ecologically valid stimuli often elicit stronger brain responses than do artificial and static stimuli, the difficulties of using naturalistic stimuli in laboratories of human brain imaging may have been overemphasized. Moreover, much of the data obtained in strict laboratory setting are not ecologically valid and thus may not fulfill the expectations in elucidating brain function. Living in Caricature Worlds One mechanism of the rapid gist of the environment could be that each individual constructs his/her own, individually distorted ‘‘caricature world,’’ where the ecologically most important features are accentuated. Considerable individual variability of the constructed caricatures of the reality is to be expected because the percepts and experiences depend on the previous experience and on many other individual features, although, at the receptor level, identical physical stimuli elicit quite similar responses across people. The caricature worlds could arise whenever sparse sensory information is added to predictive top-down effects so that features that most differ from the average (expected) scene are emphasized. We already know that people can make advanced inferences about the world and other people on the basis of very sparse information. For example, a set of about 15 moving pointlight dots recorded with a motion-capture camera can unravel whether the person is female or male, whether she/he is happy or depressed, etc. (for a review, see Blake and Shiffrar, 2007). Thus, people may, for action, need only a small subset of all the rich environmental information. Emphasis of such sparse features of course does not downplay the importance of the 1036 Neuron 94, June 7, 2017
rich information that reaches the brain and can be decoded even from single-trial electrophysiological activity (e.g., Ramkumar et al., 2013). Figure 2 illustrates the idea of the caricature world by means of three face caricatures that, despite their differences, can still be recognized as pictures of the same individual. The individually distorted caricatures here, and in the caricature world in general, do not contradict or match the reality but are similar enough to be shared and communicated with others. In other words, the categorization of stimuli matches across subjects well enough for people to share the common world by discussing and comparing percepts and experiences and by behaving coherently and interactingly in the world. Although the faces are here used only as a simplification of the whole caricature world, it is interesting to note that some face cells in the monkey middle temporal lobe react best to extreme deviations of facial features from the mean (Freiwald et al., 2009). In general, people react strongly to changes and to discrepancies between the expectations and the sensory input, and such deviations from the expected world would be salient in the individual caricature realities. A good analogy for this kind of behavior is video compression: because of redundancy (repeated scenes of almost identical content), it is most efficient to code only the changes between successive frames, which leads to considerable data compression. Discrepancies between predictions and the realized outcomes of actions or the sensory feedback are, in the Bayesian framework of brain processing (Friston et al., 2014), also assumed to be the main reasons for the organism to (re)act and update the internal models. When people learn to see and perceive their environment, discriminate objects, make distinctions between figure and ground, and stabilize ambiguous images, the elements of their worlds become labeled (i.e., can be automatically named) the more completely the more familiar the environment is. Consequently, the environment (caricature world) will be reacted upon in a more automatic and efficient manner than previously.
Neuron
Perspective Thus, it is beneficial during ontogeny to simplify the very rich sensory input and to learn to react only to the behaviorally relevant and most salient features. One well-studied example of such perceptual narrowing, including categorization of input and adaptation to the local sensory environment, is tuning to phonemes of own language (Kuhl, 1994). However, the backside is that the learned habits (skills) and the automatized behavior cannot be easily suppressed, and people thus ‘‘fly on autopilot.’’ For example, reading skill, once learned, cannot be switched off, and any seen text will be automatically read (think, for example, light advertisements in the evening city). Other examples of similarly rutted behavior are many adults’ drawing skills that are stuck to the level of 7 to 10 year olds. One possible explanation is that—without special motivation and proper teaching—the adults cannot any more perceive the building blocks of the environment to which they have learned to react automatically. Artwork can sometimes break this automaticity of perception, as many artists have explicitly commented. For example, Monet has been cited (Sacks, 1995; p. 197) to say that ‘‘whenever you go to paint, try to forget what objects you have in front of you.’’ and he continues by instructing the painter to instead focus on colors and shape. From a neuroscientist’s viewpoint, these are clear advises to avoid the automatized routines, or habits, of viewing and perceiving the world. Along similar lines, art students are advised not to focus on the object when drawing it but rather on the surrounding ‘‘empty spaces’’ that cannot be labeled or named. Moreover, copying a drawing upside down will keep the proportions better, likely because one then cannot label the parts of the scene as hands, chairs, etc. (Edwards, 1979). People thus have to make strong efforts to fight against the learned holistic views and reaction patterns to perceive the world in a richer and more detailed manner. Centrality of Social Interaction When considering the most important functional principles of the human brain, we need to include social interaction—after all, other people form our most important environment and are essential for the development of our minds. We are thinking about others most of the time, even when they are not present: what are the others doing, what have they done, and what are they intending to do? What are they thinking (especially of me)? Are they friendly? Can I trust them? Should I ally with them? Do they perceive the world as I do, and do they appreciate similar things? In primates, social interaction in the form of grooming takes much more time than would be needed just for skin care. In humans, social interaction includes, besides carefully regulated physical contact (Suvilehto et al., 2015), much verbal interaction: teaching, chatting, storytelling, and gossiping. Interestingly, the invention of fire and the concomitant prolongation of the day by about 4 hr may have significantly increased sociality, allowing plenty of time for social ‘‘fire talk,’’ which is very different from the verbal communication during daytime (Dunbar, 2014; Wiessner, 2014). Social interaction starts in early infancy, or even in the womb. It includes turn-takings that, in their most primitive form, can be seen during breast feeding. Proto-conversations between the
caregiver and the infant are already well-structured gestural and vocational interchanges that evolve into more advanced and synchronized infant-parent interactions that please both the baby and the caregiver, thus resulting in better care. Temporal synchrony and interaction are thus needed from early on, as is also evident from still-face experiments, where the infant becomes anxious very soon after the mother freezes her face in front of the infant. In later life, accurate inter-subject synchrony and successful social interaction are closely related, and being in sync can create affiliation (Lakin et al., 2003). Preference for time-accurate, often synchronous (in-phase or outphase) movements with others continues throughout the adult life, and people align at many levels, even in their speaking rhythms without any instructions (Himberg et al., 2015). People very easily sense the engagement of another person by means of timing of the interaction; delayed responses or highly varying response times are felt annoying. One may also think that the omnipresent joy of being moved— both physically and mentally—by music is related to the tendency to be in sync with others. Music-induced synchrony and turn-taking may also have therapeutic effects, improving communication skills in autistic children (Spiro and Himberg, 2016). We can read others’ mental states, including intentions, only on the basis of their motor outputs, including speech and other utterances. During face-to-face interaction, the actionperception loops between two persons fuse into a figure-ofeight loop, which incorporates mutually regulated dynamic coupling and co-adaptation and is affected by emotions (Hari et al., 2015). Temporally accurate alignment occurs at many levels simultaneously and goes beyond mirroring (Hasson and Frith, 2016). Many complex joint tasks—such as carrying a heavy object together, playing in a musical ensemble, or having a conversation with accurate turn-takings—succeed surprisingly smoothly, with mutual alignment, adaptation, adjustments, repairs, and repetition, despite the fragmentary and improvisational nature of the interaction. Social interaction is a characteristic of a dyadic system, depending on both partners involved in the dynamical interaction, without being the property of either of them (De Jaegher et al., 2016). In other words, social interaction has an autonomous and emergent character. We can thus conclude that social interaction is not just a context for social cognition and that it does not need explicit leaders and followers. Most importantly, social interaction is not like a table tennis game with sending (and coding) and receiving (and decoding) messages but rather like a threelegged race where the participants are partly bound together because they have a common goal that they can reach only together (Cummins, 2011; Hari et al., 2016). It is to be noted that even during successful interaction, the moments of togetherness, ‘‘being in the zone,’’ last only for a fraction (say, 15%) of the total time (Noy et al., 2015), and the ability to smoothly tune in and exit the inter-individual synchrony characterize successful interaction. Social interaction has been suggested to be the default that enables social cognition and understanding (Di Paolo and De Jaegher, 2012). Such primacy of the social interaction is hinted Neuron 94, June 7, 2017 1037
Neuron
Perspective for by many pieces of evidence, including the developmental arguments presented above, but more experiments are needed with setups that can capture the rapid-paced social interaction. Importantly, the systemic properties of the social interaction are absent in the typical ‘‘spectator-science’’ setups, where various social stimuli are presented to a single person who is assumed to be just a passive stimulus spectator and response producer. However, during real social interaction, people are engaged interactors that could be studied in setups of two-person neuroscience, where brain activity is simultaneously recorded from both partners (Hari et al., 2015). In such settings, both the partners’ behavior and the co-regulation of their interaction should be taken into account. How to Proceed? The experimental data of brain function increase with exponential speed but are in part contradictory and too often poorly replicable. Moreover, the (fragmentary) data are frequently not connected to any theoretical framework. Thus, the understanding of brain function accumulates much more slowly than could be expected (and hoped for) on the basis of the amount of data and the efforts. Theoretical frameworks are therefore needed, and the current paper presented some general principles that may be worth paying attention to in further research. Specifically, more emphasis should be put to brain dynamics, both with respect to the external world and in relation to one’s own and other people’s behavior. Future studies should explore social interaction in natural settings—that are not so complex as commonly thought—and to include characterization of behavior with reasonable granularity. There is need to further explore the centrality, or even primacy, of social interaction in human brain function by considering the interaction an autonomous property of the dyadic system. Such studies would implicate new experimental setups, as has been recently argued (Hari et al., 2015). One novel approach worth exploring is to implement social interaction as a latent variable (Bollen, 2002) into the modeling and analysis of interactional setups. Consequently, more clever research setups could be designed with falsifiable hypotheses, and one may then not only ask ‘‘how does the human brain work,’’ but also ‘‘how do the brains of a dyad work.’’ To improve our understanding of the behaviorally relevant principles of human brain function, it will be necessary to carefully examine the brain–environment relationship, as well as the smooth and temporally accurate behavior based on the combination of top-down processing and the coarse sensory information that single glances of the surroundings, including other people, can provide. ACKNOWLEDGMENTS This work has been financially supported by the Finnish Cultural Foundation (Eminentia grant from Ester and Uuno Kokki Fund) and by an award from the Louis Jeantet Foundation (Switzerland).
Bar, M., Kassam, K.S., Ghuman, A.S., Boshyan, J., Schmid, A.M., Dale, A.M., €ma € la €inen, M.S., Marinkovic, K., Schacter, D.L., Rosen, B.R., and Halgren, E. Ha (2006). Top-down facilitation of visual recognition. Proc. Natl. Acad. Sci. USA 103, 449–454. Barton, R.A., and Venditti, C. (2014). Rapid evolution of the cerebellum in humans and other great apes. Curr. Biol. 24, 2440–2444. Blake, R., and Shiffrar, M. (2007). Perception of human motion. Annu. Rev. Psychol. 58, 47–73. Bollen, K.A. (2002). Latent variables in psychology and the social sciences. Annu. Rev. Psychol. 53, 605–634. Buzsa´ki, G., Logothetis, N., and Singer, W. (2013). Scaling brain size, keeping timing: evolutionary preservation of brain rhythms. Neuron 80, 751–764. Chandler, D.M., and Field, D.J. (2007). Estimates of the information content and dimensionality of natural scenes from proximity distributions. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 24, 922–941. Cummins, F. (2011). Periodic and aperiodic synchronization in skilled action. Front. Hum. Neurosci. 5, 170. €kyla €, A., and Stevanovic, M. (2016). The co-creation of De Jaegher, H., Pera meaningful action: bridging enaction and interactional sociology. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150378. Dennett, D. (1991). Consciousness Explained (Penguin Books). Di Paolo, E., and De Jaegher, H. (2012). The interactive brain hypothesis. Front. Hum. Neurosci. 6, 163. Dunbar, R.I. (2014). How conversations around campfires came to be. Proc. Natl. Acad. Sci. USA 111, 14013–14014. Edwards, B. (1979). Drawing on the Right Side of the Brain (J.P. Tarcher). Freiwald, W.A., Tsao, D.Y., and Livingstone, M.S. (2009). A face feature space in the macaque temporal lobe. Nat. Neurosci. 12, 1187–1196. Friston, K.J., Stephan, K.E., Montague, R., and Dolan, R.J. (2014). Computational psychiatry: the brain as a phantastic organ. Lancet Psychiatry 1, 148–158. Hari, R., and Kujala, M.V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Physiol. Rev. 89, 453–479. Hari, R., and Parkkonen, L. (2015). The brain timewise: how timing shapes and supports brain function. Philos. Trans. R. Soc. Lond. B Biol. Sci. 370, 20140170. Hari, R., Parkkonen, L., and Nangini, C. (2010). The brain in time: insights from neuromagnetic recordings. Ann. N Y Acad. Sci. 1191, 89–109. Hari, R., Henriksson, L., Malinen, S., and Parkkonen, L. (2015). Centrality of social interaction in human brain function. Neuron 88, 181–193. Hari, R., Sams, M., and Nummenmaa, L. (2016). Attending to and neglecting people: bridging neuroscience, psychology and sociology. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 1–9. Hasson, U., and Frith, C.D. (2016). Mirroring and beyond: coupled dynamics as a generalized framework for modelling social interactions. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150366. Hasson, U., Yang, E., Vallines, I., Heeger, D.J., and Rubin, N. (2008). A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550. Henriksson, L., O¨lander, K., and Hari, R. (2016). Cortical dynamics of saccadetarget selection during free-viewing of natural scenes. bioRxiv. http://dx.doi. org/10.1101/075929.
REFERENCES
Hesslow, G. (2002). Conscious thought as simulation of behaviour and perception. Trends Cogn. Sci. 6, 242–247.
Aboitiz, F., Scheibel, A.B., Fisher, R.S., and Zaidel, E. (1992). Fiber composition of the human corpus callosum. Brain Res. 598, 143–153.
Himberg, T., Hirvenkari, L., Mandel, A., and Hari, R. (2015). Word-by-word entrainment of speech rhythm during joint story building. Front. Psychol. 6, 797.
1038 Neuron 94, June 7, 2017
Neuron
Perspective Iriki, A., Tanaka, M., and Iwamura, Y. (1996). Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7, 2325–2330. Jonas, E., and Kording, K.P. (2017). Could a neuroscientist understand a microprocessor? PLOS Comput. Biol. 13, e1005268. Kay, K.N., Naselaris, T., Prenger, R.J., and Gallant, J.L. (2008). Identifying natural images from human brain activity. Nature 452, 352–355.
Ramkumar, P., Jas, M., Pannasch, S., Hari, R., and Parkkonen, L. (2013). Feature-specific information processing precedes concerted activation in human visual cortex. J. Neurosci. 33, 7691–7699. Ringo, J.L., Doty, R.W., Demeter, S., and Simard, P.Y. (1994). Time is of the essence: a conjecture that hemispheric specialization arises from interhemispheric conduction delay. Cereb. Cortex 4, 331–343.
Kuhl, P.K. (1994). Learning and representation in speech and language. Curr. Opin. Neurobiol. 4, 812–822.
Rizzolatti, G., and Craighero, L. (2004). The mirror-neuron system. Annu. Rev. Neurosci. 27, 169–192.
Lakin, J.L., Jefferis, V.E., Cheng, C.M., and Chartrand, T.L. (2003). The chameleon effect as social glue: evidence for the evolutionary significance of nonconscious mimicry. J. Nonverbal Behav. 27, 145–162.
Rosenholtz, R. (2016). Capabilities and limitations of peripheral vision. Annu. Rev. Vis. Sci. 2, 437–457. Sacks, O. (1995). An Anthropologist On Mars (Picador).
Lazebnik, Y. (2002). Can a biologist fix a radio?–Or, what I learned while studying apoptosis. Cancer Cell 2, 179–182. Lerner, Y., Honey, C.J., Silbert, L.J., and Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915. Noy, L., Levit-Binun, N., and Golland, Y. (2015). Being in the zone: physiological markers of togetherness in joint improvisation. Front. Hum. Neurosci. 9, 187.
Spiro, N., and Himberg, T. (2016). Analysing change in music therapy interactions of children with communication difficulties. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150374. Suvilehto, J.T., Glerean, E., Dunbar, R.I., Hari, R., and Nummenmaa, L. (2015). Topography of social touching depends on emotional bonds between humans. Proc. Natl. Acad. Sci. USA 112, 13811–13816.
Oliva, A., and Torralba, A. (2006). Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36.
Wang, S.S., Shultz, J.R., Burish, M.J., Harrison, K.H., Hof, P.R., Towns, L.C., Wagers, M.W., and Wyatt, K.D. (2008). Functional trade-offs in white matter axonal scaling. J. Neurosci. 28, 4047–4056.
Pang, R., Lansdell, B.J., and Fairhall, A.L. (2016). Dimensionality reduction in neuroscience. Curr. Biol. 26, R656–R660.
Wiessner, P.W. (2014). Embers of society: Firelight talk among the Ju/’hoansi Bushmen. Proc. Natl. Acad. Sci. USA 111, 14027–14035.
Neuron 94, June 7, 2017 1039
Perspective From Brain–Environment Connections to Temporal Dynamics and Social Interaction: Principles of Human Brain Function Riitta Hari1,* 1Department of Art, Aalto University, PO Box 31100, 00076 AALTO, Helsinki, Finland *Correspondence: [email protected] http://dx.doi.org/10.1016/j.neuron.2017.04.007
Experimental data about brain function accumulate faster than does our understanding of how the brain works. To tackle some general principles at the grain level of behavior, I start from the omnipresent brainenvironment connection that forces regularities of the physical world to shape the brain. Based on topdown processing, added by sparse sensory information, people are able to form individual ‘‘caricature worlds,’’ which are similar enough to be shared among other people and which allow quick and purposeful reactions to abrupt changes. Temporal dynamics and social interaction in natural environments serve as further essential organizing principles of human brain function. Brain in the World In a recent small workshop (Dynamical Systems Interdisciplinary Network workshop, 2016), all speakers were challenged to give their answers to the question ‘‘how does the brain work.’’ The current paper is based on the ideas presented in my introductory talk of that workshop, aiming to look at some overarching principles of brain function at the grain level of behavior and to emphasize the importance of both temporal processing and social interaction as topics for further research. Although many of the discussed concepts are not new, they are not shared by all scientists and, in some aspects, the separation of views is deep. Thus, it seems worthwhile to open the can of worms, even at the risk of repetition and too strong simplifications. If we consider the heart as a pump and the liver as a filter, then the brain can be said to be a device that predicts the future on the basis of the past. To go further, it may be useful, or even necessary, to think along the lines of Theodosius Dobzhansky (1900– 1975), who noted that ‘‘nothing in biology makes sense except in the light of evolution.’’ During evolution, the environment has imprinted itself very strongly to the structure and function of the brain. The most evident examples are the omnipresent circadian rhythms driven by and adjusted to the day/night cycle on the Earth. More generally, many functioning principles of the brain can be understood on the basis of the regularities of the world, although such an approach is seldom taken. Thus, let’s, as an example, consider the macroscopic physical law that describes the movements of objects in the world and the corresponding organization principles in the brain. Newton’s second law, schematically presented in Figure 1A, tells that force is equal to mass times acceleration (F = ma). In other words, the same force will give higher acceleration to light objects compared with heavy objects. For example, the wind moves the (spatially) smaller and thinner branches of a tree faster (i.e., with higher temporal frequency or shorter temporal scales) than the (spatially) larger branches that move relatively more slowly. Accordingly, one would expect that small spatial windows (receptive fields) in the (visual) brain are associated with
more narrow temporal windows. In other words, the high spatial and temporal frequencies should coincide, and in fact, the spatial receptive fields are the smallest at the early projection cortices, increasing further away along the processing stream, as is indicated schematically in Figure 1B. Moreover, brain imaging experiments, using both functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG), have indicated that the time windows of processing get longer (corresponding to lower temporal frequencies) as the spatial receptive windows increase, as is also indicated in Figure 1B; similar organization has been observed both in visual and auditory processing streams (Hari and Parkkonen, 2015; Hari et al., 2010; Hasson et al., 2008; Lerner et al., 2011). Thus, because of the tight connections of temporal and spatial scales in the physical world, the temporal and spatial scales should be—and are—linked in the brain as well. Many other organizational principles of the brain can be understood from the connection of the brain to the world. For example, the frequencies of brain rhythms are surprisingly similar in different mammals independently of the brain size varying by a factor of 17,000 (Buzsa´ki et al., 2013). This intriguing similarity of brain-rhythm frequencies in different species can be tracked back to the organism-world interface: the rhythmicity was suggested to be related, at least in part, to muscle-contraction mechanisms, especially to the time constants of myosin and actin filaments that have remained similar across the mammals (Buzsa´ki et al., 2013). In general, the motor system is central to brain function from simple movements to cognition, even so that—to state it boldly—during evolution, minds have emerged only to self-moving organisms. It has been essential to act in the world, to change it, and to actively search for perceptual information. Accordingly, the individual’s active involvement in the actions is often decisive for shaping the brain circuitries. For example, a rake in the hand of a monkey changes the brain’s visuo-tactile receptive fields only if the monkey actively uses the tool (Iriki et al., 1996). Neuron 94, June 7, 2017 ª 2017 Elsevier Inc. 1033
Neuron
Perspective A
B
Figure 1. How Movement of Objects in the World May Have Affected Brain Organization during Evolution (A) Schematic description of the Newton’s second law (force = mass 3 acceleration; F = ma) applied to the effects of wind on a tree and the resulting relationship between the temporal and spatial frequencies of various points of the tree, here indicated by the numbers 1–3 from thin branches to the trunk. (B) Schematic presentation of the increase of visual (spatial) receptive fields when progressing from the primary projection area to higher levels of the processing hierarchy; at the same time, the temporal processing windows get broader (temporal frequency is decreasing). The numbers 1–3 correspond to those in (A).
Action–perception (A–P) loops bridge sensory inputs with the sensory predictions associated with the actions (Hari and Kujala, 2009). Mirroring—activation of motor circuitries also during observation of other persons’ actions (for a review, see Rizzolatti and Craighero, 2004)—requires similarities between low-level brain activity across individuals. It is possible that the sensorimotor transformations required by both own and observed actions, as well as the control of sequential actions, are reflected in the relatively larger size of the cerebellum with respect to the neocortex in great apes and humans compared with non-apes (Barton and Venditti, 2014). Repeated activation of the A–P loop transforms to habits— automatized behavior—and many actions, originally directed to the outside world, can be internalized to thoughts (Hesslow, 2002). An illustrative example is the use of the abacus (called soroban in Japanese). When the young soroban wizards participate in competitions of mental calculation, where they are no more allowed to use the soroban, their fingers still move across the thin air with movements that they used to do while moving the beads of the soroban. Within the context of the A–P loop—which includes the brain, body, and the environment—it is easy to understand the important role of the physical environment, other people, and culture on all human behavior, including thinking. Both the physical environment and other people can act as memory surfaces to support cognition and memory, which becomes especially important during old age memory decline. Humans search for continuous stimulation and novelty, almost like they crave for nutrition, and during sensory deprivation—when the person is kept within a sound-proof, dimly lit room, even floating in water—thoughts start to stagger and deteriorate. The omnipresent brain–environment connection can explain why psychotic delusions have contents closely associated to the contemporary society and technology, thereby varying across centuries, and why artistic expressions follow more the cultural traditions than the regularities of brain function or environment-independent imagery. Another example is depression, which is treated best when antidepressant drugs are combined with environmental changes. Moreover, a ‘‘smart’’ environment helps and supports behavior, and one may feel quite stupid when the mobile phone with all its address information is lost or when one cannot turn the water tap on for warm water in the hotel bathroom. 1034 Neuron 94, June 7, 2017
Thus, it seems evident that if we study only isolated brains, the results may not be relevant for understanding the brain basis of natural behavior. Some delighting and instructive examples are presented in papers asking ‘‘Can a biologist fix a radio?’’ (Lazebnik, 2002) or ‘‘Can a neuroscientist understand a microprocessor?’’ (Jonas and Kording, 2017). Both of these exercises gave rather disappointing results, probably because the environmental connections and developmental and evolutionary histories of the systems under study were not included in the analyses. Importantly, the brains (and their owners) are interactive, not only reactive, although often studied in unidirectional stimulus– response setups. We have argued earlier that social interaction, as such, is central for the whole human brain function (Hari et al., 2015; Hari and Kujala, 2009). Below, the pre-requisites for quick reactions and social interaction are discussed, noting that the brain architecture is a compromise between rapid temporal signaling (providing time markers) and slower, but denser, signaling (needed for detailed feature analysis). People are able to react fast and purposefully because they already have abundant information (‘‘priors’’) about the world and because they are able to obtain a gist of the surroundings with a single glance. Timing of Behavior and the Corresponding Brain Function The behaviorally relevant timescales range from under 1 ms (for directional hearing) to hundreds of milliseconds (for reaction times, attentional modulations, short-term memory, etc.); for reviews of timing in the brain, see Hari et al. (2010, 2015). For survival, some reactions need to be as quick as the sensory and motor pathways and the brain can allow. However, in general, the brain is very sluggish: compared with copper wires, the conduction speeds of mammal nerve fibers are slower by a factor of 107. Time needed for processing of sensory input would shorten if the communication between involved brain areas would be faster, as would happen if the connecting fibers would be thicker. Such a design would, however, result in serious packaging problems within the brain and skull (Ringo et al., 1994). Instead, the nature has a clever solution for this problem so that only a small proportion of all fibers is scaled according to the brain size. For example, the fibers in corpus callosum are about the same diameters (and thus have about the same conduction velocities) in
Neuron
Perspective mammals of different sizes, resulting in humans to inter-hemispheric delays of 5–300 ms (Aboitiz et al., 1992). However, in a small proportion of the thickest fibers, the diameters scale with brain size. In other words, the larger the brain, the faster (and thicker) axons are in a small proportion of all fibers, so that the inter-hemispheric conduction delays can stay within 5 ms in mammal brains of all sizes (Wang et al., 2008). This scaling of only a part of the fibers allows the fast, thick myelinated fibers (that take much space) to connect far-away areas with good temporal accuracy and to provide time markers for processing. The shorter-distance connections occur via dense, thin (and therefore slow) myelinated and unmyelinated fibers. What Can We See with a Single Glance? To react purposefully in urgent situations, we need to quickly evaluate the scene and the items to which to react. If a person notices a snake on a path that she is walking barefooted, she will react automatically, readjusting the trajectory of her foot— possibly prolonging the step—within a fraction of a second, even before the snake as an object has been carefully analyzed and consciously perceived. The visceral feelings of fright will come later. But how can such a purposeful reaction happen so fast although an average reaction time for a simple light stimulus is of the order 200 ms and prolongs considerably when choices have to be made between multiple options? Recognition of a moving elongate ‘‘something’’ as a snake likely requires much more analysis than does a two-choice reaction-time task, even though humans seem to have evolved to be oversensitive to any moving objects: it is better to overestimate the possible dangers and escape—even when it was unnecessary—than vice versa. Such quick purposeful reactions to complex stimuli challenge the traditional view about the exploration of the visual world, which assumes humans to be ‘‘foveal animals’’ who sample the world in a discontinuous snapshot manner by bringing objects and other targets into foveal vision and thereby into the center of attention. Vision is lost between the fixations because of saccadic suppression. Thus, one is assumed to be viewing and seeing the world through a hole the size of the fovea, cumulating the percept of the whole scene from sequential pinhole images, each collected during a fixation lasting a few hundred milliseconds. Thus, the experienced richness of the visual scene is surprising, Similarly, the immediate feeling of the ambience of a new place, for example, a cathedral, seems difficult to explain even with a multisensory ‘‘foveal-animal’’ approach, and consequently, the whole experience of the rich ambience has been claimed to be an illusion (Dennett, 1991). However, both behavioral and computational evidence indicate that people can recognize, with a single glance, the ‘‘gist’’—the essence—of the global spatial layout of the surroundings, even when the stimulus is presented only for 30 ms (Oliva and Torralba, 2006). It is unlikely that foveal vision could explain such quick gists that would need several successive fixations, as discussed above. Gathering visual input via fovea only would further implicate that the retina outside the fovea, which is outside the central 2 arc degrees, would be saved mainly for night vision, which sounds wasteful and therefore an improbable solution to serve human behavior. Thus, it is assuring that
increasing evidence indicates the importance of peripheral vision and that people seem to rely on it much more than is commonly acknowledged (Rosenholtz, 2016). Although visual acuity is lower in peripheral than foveal vision, the peripheral visual vision can—by integrating perceptual information over sizeable areas of the visual field—provide powerful summary statistics about various objects and orientations, luminance, and other physical features (Rosenholtz, 2016). For example, summary statistics of shape information from the peripheral retina would explain how a person who fixates the eyes to the center of a screen at the time of a scene-image appearance, tends to direct the first saccade to human faces and other social stimuli located clearly outside the foveal vision (Henriksson et al., 2016). Top-down effects—e.g., from the frontal lobe (Bar et al., 2006)—are expected to help the detection and recognition of relevant objects. How Complex Is Our Environment? Both the brain and the environment are extremely complex, with several hierarchical temporospatial levels of organization. Therefore, the neuroscientists’ attitude in trying to understand how the brain works has been to first explore the simple and small-scale (microscopic or neuronal level) principles and then (one day) to derive from them the large-scale behavioral principles. One example is the description of natural images as Gabor pyramids that provide unambiguous representations of natural images by specifying all possible spatial frequencies and orientations at each pixel of the scene image. Such computational representations of natural images have been successfully used in the decoding of vision-related brain states (Kay et al., 2008), with implicit assumptions that the percepts might rely on a corresponding summed activity of feature-specific neurons. The neurophysiological generalizations from the Gabor pyramid approaches would thus assume that we can—and should—build complex functions from simple elements. This is certainly a feasible approach, but trying to understand behavior as the summed activity of single neurons and action potentials seems equally difficult as it is to explain the formation of sand dunes on the basis of the properties of the single sand grains or to explain the behavior of gas on the basis of molecular dynamics only, without introducing the (emergent) property of pressure. Importantly, at each new emergence level—be it in the brain or in the environment—the description will get simpler (and not more complex) than at the lower level (compare computations with gas pressure versus the associated molecular dynamics). Indeed, the dimensionality of a naturalistic scene is lower than that of the corresponding pixel-by-pixel Gabor pyramid representation because of the clear structure (e.g., continuities, labels, objects, buildings, trees, etc.) of the natural environment. In other words, the natural environment is not random and the neighboring points of the scene are not independent (Chandler and Field, 2007) in contrast to what is assumed in the Gabor pyramid descriptions of scenes. Moreover, the objects of the real world move according to the laws of physics that people are accustomed to (such as gravitation, object movement, and behavior of light). That the abundant pre-information of the environment affects the perception becomes evident from many Neuron 94, June 7, 2017 1035
Neuron
Perspective A
B
C
Figure 2. The Idea of the Caricature Worlds Three caricature face images are presented as if they would be examples of caricature worlds of three different viewers. The images (A)–(C) are distorted in different manner but may still allow the viewer to identify the person who has inspired them.
illusions where, for example, the shading is interpreted to indicate concave or convex shapes, in accordance with light coming from above (from the sun). Thus, we can conclude that although the world is very complex, it is well structured and thus not so difficult to understand as it would be if comprised of totally independent elements. Dimension reduction—description in terms of a few summarizing features—can strikingly decrease the complexity of both the physical world and brain functions (Pang et al., 2016). Since ecologically valid stimuli often elicit stronger brain responses than do artificial and static stimuli, the difficulties of using naturalistic stimuli in laboratories of human brain imaging may have been overemphasized. Moreover, much of the data obtained in strict laboratory setting are not ecologically valid and thus may not fulfill the expectations in elucidating brain function. Living in Caricature Worlds One mechanism of the rapid gist of the environment could be that each individual constructs his/her own, individually distorted ‘‘caricature world,’’ where the ecologically most important features are accentuated. Considerable individual variability of the constructed caricatures of the reality is to be expected because the percepts and experiences depend on the previous experience and on many other individual features, although, at the receptor level, identical physical stimuli elicit quite similar responses across people. The caricature worlds could arise whenever sparse sensory information is added to predictive top-down effects so that features that most differ from the average (expected) scene are emphasized. We already know that people can make advanced inferences about the world and other people on the basis of very sparse information. For example, a set of about 15 moving pointlight dots recorded with a motion-capture camera can unravel whether the person is female or male, whether she/he is happy or depressed, etc. (for a review, see Blake and Shiffrar, 2007). Thus, people may, for action, need only a small subset of all the rich environmental information. Emphasis of such sparse features of course does not downplay the importance of the 1036 Neuron 94, June 7, 2017
rich information that reaches the brain and can be decoded even from single-trial electrophysiological activity (e.g., Ramkumar et al., 2013). Figure 2 illustrates the idea of the caricature world by means of three face caricatures that, despite their differences, can still be recognized as pictures of the same individual. The individually distorted caricatures here, and in the caricature world in general, do not contradict or match the reality but are similar enough to be shared and communicated with others. In other words, the categorization of stimuli matches across subjects well enough for people to share the common world by discussing and comparing percepts and experiences and by behaving coherently and interactingly in the world. Although the faces are here used only as a simplification of the whole caricature world, it is interesting to note that some face cells in the monkey middle temporal lobe react best to extreme deviations of facial features from the mean (Freiwald et al., 2009). In general, people react strongly to changes and to discrepancies between the expectations and the sensory input, and such deviations from the expected world would be salient in the individual caricature realities. A good analogy for this kind of behavior is video compression: because of redundancy (repeated scenes of almost identical content), it is most efficient to code only the changes between successive frames, which leads to considerable data compression. Discrepancies between predictions and the realized outcomes of actions or the sensory feedback are, in the Bayesian framework of brain processing (Friston et al., 2014), also assumed to be the main reasons for the organism to (re)act and update the internal models. When people learn to see and perceive their environment, discriminate objects, make distinctions between figure and ground, and stabilize ambiguous images, the elements of their worlds become labeled (i.e., can be automatically named) the more completely the more familiar the environment is. Consequently, the environment (caricature world) will be reacted upon in a more automatic and efficient manner than previously.
Neuron
Perspective Thus, it is beneficial during ontogeny to simplify the very rich sensory input and to learn to react only to the behaviorally relevant and most salient features. One well-studied example of such perceptual narrowing, including categorization of input and adaptation to the local sensory environment, is tuning to phonemes of own language (Kuhl, 1994). However, the backside is that the learned habits (skills) and the automatized behavior cannot be easily suppressed, and people thus ‘‘fly on autopilot.’’ For example, reading skill, once learned, cannot be switched off, and any seen text will be automatically read (think, for example, light advertisements in the evening city). Other examples of similarly rutted behavior are many adults’ drawing skills that are stuck to the level of 7 to 10 year olds. One possible explanation is that—without special motivation and proper teaching—the adults cannot any more perceive the building blocks of the environment to which they have learned to react automatically. Artwork can sometimes break this automaticity of perception, as many artists have explicitly commented. For example, Monet has been cited (Sacks, 1995; p. 197) to say that ‘‘whenever you go to paint, try to forget what objects you have in front of you.’’ and he continues by instructing the painter to instead focus on colors and shape. From a neuroscientist’s viewpoint, these are clear advises to avoid the automatized routines, or habits, of viewing and perceiving the world. Along similar lines, art students are advised not to focus on the object when drawing it but rather on the surrounding ‘‘empty spaces’’ that cannot be labeled or named. Moreover, copying a drawing upside down will keep the proportions better, likely because one then cannot label the parts of the scene as hands, chairs, etc. (Edwards, 1979). People thus have to make strong efforts to fight against the learned holistic views and reaction patterns to perceive the world in a richer and more detailed manner. Centrality of Social Interaction When considering the most important functional principles of the human brain, we need to include social interaction—after all, other people form our most important environment and are essential for the development of our minds. We are thinking about others most of the time, even when they are not present: what are the others doing, what have they done, and what are they intending to do? What are they thinking (especially of me)? Are they friendly? Can I trust them? Should I ally with them? Do they perceive the world as I do, and do they appreciate similar things? In primates, social interaction in the form of grooming takes much more time than would be needed just for skin care. In humans, social interaction includes, besides carefully regulated physical contact (Suvilehto et al., 2015), much verbal interaction: teaching, chatting, storytelling, and gossiping. Interestingly, the invention of fire and the concomitant prolongation of the day by about 4 hr may have significantly increased sociality, allowing plenty of time for social ‘‘fire talk,’’ which is very different from the verbal communication during daytime (Dunbar, 2014; Wiessner, 2014). Social interaction starts in early infancy, or even in the womb. It includes turn-takings that, in their most primitive form, can be seen during breast feeding. Proto-conversations between the
caregiver and the infant are already well-structured gestural and vocational interchanges that evolve into more advanced and synchronized infant-parent interactions that please both the baby and the caregiver, thus resulting in better care. Temporal synchrony and interaction are thus needed from early on, as is also evident from still-face experiments, where the infant becomes anxious very soon after the mother freezes her face in front of the infant. In later life, accurate inter-subject synchrony and successful social interaction are closely related, and being in sync can create affiliation (Lakin et al., 2003). Preference for time-accurate, often synchronous (in-phase or outphase) movements with others continues throughout the adult life, and people align at many levels, even in their speaking rhythms without any instructions (Himberg et al., 2015). People very easily sense the engagement of another person by means of timing of the interaction; delayed responses or highly varying response times are felt annoying. One may also think that the omnipresent joy of being moved— both physically and mentally—by music is related to the tendency to be in sync with others. Music-induced synchrony and turn-taking may also have therapeutic effects, improving communication skills in autistic children (Spiro and Himberg, 2016). We can read others’ mental states, including intentions, only on the basis of their motor outputs, including speech and other utterances. During face-to-face interaction, the actionperception loops between two persons fuse into a figure-ofeight loop, which incorporates mutually regulated dynamic coupling and co-adaptation and is affected by emotions (Hari et al., 2015). Temporally accurate alignment occurs at many levels simultaneously and goes beyond mirroring (Hasson and Frith, 2016). Many complex joint tasks—such as carrying a heavy object together, playing in a musical ensemble, or having a conversation with accurate turn-takings—succeed surprisingly smoothly, with mutual alignment, adaptation, adjustments, repairs, and repetition, despite the fragmentary and improvisational nature of the interaction. Social interaction is a characteristic of a dyadic system, depending on both partners involved in the dynamical interaction, without being the property of either of them (De Jaegher et al., 2016). In other words, social interaction has an autonomous and emergent character. We can thus conclude that social interaction is not just a context for social cognition and that it does not need explicit leaders and followers. Most importantly, social interaction is not like a table tennis game with sending (and coding) and receiving (and decoding) messages but rather like a threelegged race where the participants are partly bound together because they have a common goal that they can reach only together (Cummins, 2011; Hari et al., 2016). It is to be noted that even during successful interaction, the moments of togetherness, ‘‘being in the zone,’’ last only for a fraction (say, 15%) of the total time (Noy et al., 2015), and the ability to smoothly tune in and exit the inter-individual synchrony characterize successful interaction. Social interaction has been suggested to be the default that enables social cognition and understanding (Di Paolo and De Jaegher, 2012). Such primacy of the social interaction is hinted Neuron 94, June 7, 2017 1037
Neuron
Perspective for by many pieces of evidence, including the developmental arguments presented above, but more experiments are needed with setups that can capture the rapid-paced social interaction. Importantly, the systemic properties of the social interaction are absent in the typical ‘‘spectator-science’’ setups, where various social stimuli are presented to a single person who is assumed to be just a passive stimulus spectator and response producer. However, during real social interaction, people are engaged interactors that could be studied in setups of two-person neuroscience, where brain activity is simultaneously recorded from both partners (Hari et al., 2015). In such settings, both the partners’ behavior and the co-regulation of their interaction should be taken into account. How to Proceed? The experimental data of brain function increase with exponential speed but are in part contradictory and too often poorly replicable. Moreover, the (fragmentary) data are frequently not connected to any theoretical framework. Thus, the understanding of brain function accumulates much more slowly than could be expected (and hoped for) on the basis of the amount of data and the efforts. Theoretical frameworks are therefore needed, and the current paper presented some general principles that may be worth paying attention to in further research. Specifically, more emphasis should be put to brain dynamics, both with respect to the external world and in relation to one’s own and other people’s behavior. Future studies should explore social interaction in natural settings—that are not so complex as commonly thought—and to include characterization of behavior with reasonable granularity. There is need to further explore the centrality, or even primacy, of social interaction in human brain function by considering the interaction an autonomous property of the dyadic system. Such studies would implicate new experimental setups, as has been recently argued (Hari et al., 2015). One novel approach worth exploring is to implement social interaction as a latent variable (Bollen, 2002) into the modeling and analysis of interactional setups. Consequently, more clever research setups could be designed with falsifiable hypotheses, and one may then not only ask ‘‘how does the human brain work,’’ but also ‘‘how do the brains of a dyad work.’’ To improve our understanding of the behaviorally relevant principles of human brain function, it will be necessary to carefully examine the brain–environment relationship, as well as the smooth and temporally accurate behavior based on the combination of top-down processing and the coarse sensory information that single glances of the surroundings, including other people, can provide. ACKNOWLEDGMENTS This work has been financially supported by the Finnish Cultural Foundation (Eminentia grant from Ester and Uuno Kokki Fund) and by an award from the Louis Jeantet Foundation (Switzerland).
Bar, M., Kassam, K.S., Ghuman, A.S., Boshyan, J., Schmid, A.M., Dale, A.M., €ma € la €inen, M.S., Marinkovic, K., Schacter, D.L., Rosen, B.R., and Halgren, E. Ha (2006). Top-down facilitation of visual recognition. Proc. Natl. Acad. Sci. USA 103, 449–454. Barton, R.A., and Venditti, C. (2014). Rapid evolution of the cerebellum in humans and other great apes. Curr. Biol. 24, 2440–2444. Blake, R., and Shiffrar, M. (2007). Perception of human motion. Annu. Rev. Psychol. 58, 47–73. Bollen, K.A. (2002). Latent variables in psychology and the social sciences. Annu. Rev. Psychol. 53, 605–634. Buzsa´ki, G., Logothetis, N., and Singer, W. (2013). Scaling brain size, keeping timing: evolutionary preservation of brain rhythms. Neuron 80, 751–764. Chandler, D.M., and Field, D.J. (2007). Estimates of the information content and dimensionality of natural scenes from proximity distributions. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 24, 922–941. Cummins, F. (2011). Periodic and aperiodic synchronization in skilled action. Front. Hum. Neurosci. 5, 170. €kyla €, A., and Stevanovic, M. (2016). The co-creation of De Jaegher, H., Pera meaningful action: bridging enaction and interactional sociology. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150378. Dennett, D. (1991). Consciousness Explained (Penguin Books). Di Paolo, E., and De Jaegher, H. (2012). The interactive brain hypothesis. Front. Hum. Neurosci. 6, 163. Dunbar, R.I. (2014). How conversations around campfires came to be. Proc. Natl. Acad. Sci. USA 111, 14013–14014. Edwards, B. (1979). Drawing on the Right Side of the Brain (J.P. Tarcher). Freiwald, W.A., Tsao, D.Y., and Livingstone, M.S. (2009). A face feature space in the macaque temporal lobe. Nat. Neurosci. 12, 1187–1196. Friston, K.J., Stephan, K.E., Montague, R., and Dolan, R.J. (2014). Computational psychiatry: the brain as a phantastic organ. Lancet Psychiatry 1, 148–158. Hari, R., and Kujala, M.V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Physiol. Rev. 89, 453–479. Hari, R., and Parkkonen, L. (2015). The brain timewise: how timing shapes and supports brain function. Philos. Trans. R. Soc. Lond. B Biol. Sci. 370, 20140170. Hari, R., Parkkonen, L., and Nangini, C. (2010). The brain in time: insights from neuromagnetic recordings. Ann. N Y Acad. Sci. 1191, 89–109. Hari, R., Henriksson, L., Malinen, S., and Parkkonen, L. (2015). Centrality of social interaction in human brain function. Neuron 88, 181–193. Hari, R., Sams, M., and Nummenmaa, L. (2016). Attending to and neglecting people: bridging neuroscience, psychology and sociology. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 1–9. Hasson, U., and Frith, C.D. (2016). Mirroring and beyond: coupled dynamics as a generalized framework for modelling social interactions. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150366. Hasson, U., Yang, E., Vallines, I., Heeger, D.J., and Rubin, N. (2008). A hierarchy of temporal receptive windows in human cortex. J. Neurosci. 28, 2539–2550. Henriksson, L., O¨lander, K., and Hari, R. (2016). Cortical dynamics of saccadetarget selection during free-viewing of natural scenes. bioRxiv. http://dx.doi. org/10.1101/075929.
REFERENCES
Hesslow, G. (2002). Conscious thought as simulation of behaviour and perception. Trends Cogn. Sci. 6, 242–247.
Aboitiz, F., Scheibel, A.B., Fisher, R.S., and Zaidel, E. (1992). Fiber composition of the human corpus callosum. Brain Res. 598, 143–153.
Himberg, T., Hirvenkari, L., Mandel, A., and Hari, R. (2015). Word-by-word entrainment of speech rhythm during joint story building. Front. Psychol. 6, 797.
1038 Neuron 94, June 7, 2017
Neuron
Perspective Iriki, A., Tanaka, M., and Iwamura, Y. (1996). Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7, 2325–2330. Jonas, E., and Kording, K.P. (2017). Could a neuroscientist understand a microprocessor? PLOS Comput. Biol. 13, e1005268. Kay, K.N., Naselaris, T., Prenger, R.J., and Gallant, J.L. (2008). Identifying natural images from human brain activity. Nature 452, 352–355.
Ramkumar, P., Jas, M., Pannasch, S., Hari, R., and Parkkonen, L. (2013). Feature-specific information processing precedes concerted activation in human visual cortex. J. Neurosci. 33, 7691–7699. Ringo, J.L., Doty, R.W., Demeter, S., and Simard, P.Y. (1994). Time is of the essence: a conjecture that hemispheric specialization arises from interhemispheric conduction delay. Cereb. Cortex 4, 331–343.
Kuhl, P.K. (1994). Learning and representation in speech and language. Curr. Opin. Neurobiol. 4, 812–822.
Rizzolatti, G., and Craighero, L. (2004). The mirror-neuron system. Annu. Rev. Neurosci. 27, 169–192.
Lakin, J.L., Jefferis, V.E., Cheng, C.M., and Chartrand, T.L. (2003). The chameleon effect as social glue: evidence for the evolutionary significance of nonconscious mimicry. J. Nonverbal Behav. 27, 145–162.
Rosenholtz, R. (2016). Capabilities and limitations of peripheral vision. Annu. Rev. Vis. Sci. 2, 437–457. Sacks, O. (1995). An Anthropologist On Mars (Picador).
Lazebnik, Y. (2002). Can a biologist fix a radio?–Or, what I learned while studying apoptosis. Cancer Cell 2, 179–182. Lerner, Y., Honey, C.J., Silbert, L.J., and Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915. Noy, L., Levit-Binun, N., and Golland, Y. (2015). Being in the zone: physiological markers of togetherness in joint improvisation. Front. Hum. Neurosci. 9, 187.
Spiro, N., and Himberg, T. (2016). Analysing change in music therapy interactions of children with communication difficulties. Philos. Trans. R. Soc. Lond. B Biol. Sci. 371, 20150374. Suvilehto, J.T., Glerean, E., Dunbar, R.I., Hari, R., and Nummenmaa, L. (2015). Topography of social touching depends on emotional bonds between humans. Proc. Natl. Acad. Sci. USA 112, 13811–13816.
Oliva, A., and Torralba, A. (2006). Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36.
Wang, S.S., Shultz, J.R., Burish, M.J., Harrison, K.H., Hof, P.R., Towns, L.C., Wagers, M.W., and Wyatt, K.D. (2008). Functional trade-offs in white matter axonal scaling. J. Neurosci. 28, 4047–4056.
Pang, R., Lansdell, B.J., and Fairhall, A.L. (2016). Dimensionality reduction in neuroscience. Curr. Biol. 26, R656–R660.
Wiessner, P.W. (2014). Embers of society: Firelight talk among the Ju/’hoansi Bushmen. Proc. Natl. Acad. Sci. USA 111, 14027–14035.
Neuron 94, June 7, 2017 1039