- Email: [email protected]
Vision: Dialogues between Deep Networks and the Brain
Current Biology
Dispatches number of oocysts that colonize a midgut is generally believed to be lower than in laboratory conditions. Additionally, whatever parasite-killing mechanism may be at play, it is inhibited during oogenesis. Once this mechanism is identified, it will be critical to test whether it also impacts mosquito oogenesis. If not, any anti-malarial strategy employing this mechanism would reduce Plasmodium numbers without accelerating P. falciparum transmission. On the other hand, an anti-malarial strategy targeting the mosquito’s oogenesis program per se may come at a cost, as it would lead to faster parasite development and earlier transmission. This is particularly relevant because female mosquitoes have a short life span, not more than 2–3 weeks in the field, and so any accelerated growth by the parasite (which usually takes 10–14 days to develop) would lead to increased transmission even if numbers of transmissible parasites are lower. Faster growth occurs because, when egg development is impaired, Plasmodium oocysts can use excess mosquito resources. Importantly, although it accelerates growth, an excess of nutrients does not increase the number of sporozoites produced per oocyst, suggesting that Plasmodium oocysts have a pre-determined number of sporozoites they can generate, and that this number cannot be exceeded. This is likely a consequence of 30 million years of co-evolution [9], which have shaped the oocyst’s developmental process to be as silent as possible, so as not to affect the mosquito’s reproductive fitness and to consequently maintain the population of vectors needed for the perpetuation of the parasite’s life cycle. Ultimately, Plasmodium oocysts act less as parasites and more as commensal organisms. These results are surprising because it has long been suggested that parasitic infections of vectors would impose a reproductive cost for the mosquito [4]. With these results, the authors show that the number of Plasmodium parasites and the development of mosquito eggs are linked in a commensal relationship [5]. The mosquito does not lose its reproductive fitness in spite of the presence of Plasmodium parasites, which benefit twofold from the oogenesis process — for protection and for resources.
Although, by definition, parasitism means that one species — the parasite — exploits a second species — the host — to obtain food, housing, or protection while causing harm to the host, it is not in the parasite’s best interest to weaken its host to a point that it cannot support the parasite anymore. The study by Werling, Shaw, Itoe et al. suggests that Plasmodium– Anopheles interactions are the product of an evolutionary process that may have progressed from a typical parasitic association into a commensal one.
mosquito shapes within-host malaria infectivity and virulence. Nat. Commun. 9, 3474. 4. Schwenke, R.A., Lazzaro, B.P., and Wolfner, M.F. (2015). Reproduction–immunity trade-offs in insects. Annu. Rev. Entomol. 61, 239–256. 5. Werling, K., Shaw, W.R., Itoe, M.A., Westervelt, K.A., Marcenac, P., Paton, D.G., Peng, D., Singh, N., Smidler, A.L., South, A., et al. (2019). Steroid hormone function controls noncompetitive Plasmodium development in Anopheles. Cell 177, 315–325. 6. Bai, H., Gelman, D.B., and Palli, S.R. (2010). Mode of action of methoprene in affecting female reproduction in the African malaria mosquito, Anopheles gambiae. Pest Manag. Sci. 66, 936–943.
REFERENCES 1. Bennink, S., Kiesow, M.J., and Pradel, G. (2016). The development of malaria parasites in the mosquito midgut. Cell. Microbiol. 18, 905–918. 2. Mitchell, S.N., and Catteruccia, F. (2017). Anopheline reproductive biology: Impacts on vectorial capacity and potential avenues for malaria control. Cold Spring Harb. Perspect. Med. 7, 14. 3. Costa, G., Gildenhard, M., Eldering, M., Lindquist, R.L., Hauser, A.E., Sauerwein, R., Goosmann, C., Brinkmann, V., CarrilloBustamante, P., and Levashina, E.A. (2018). Non-competitive resource exploitation within
7. Atella, G.C., Silva-Neto, M.A.C., Golodne, D.M., Arefin, S., and Shahabuddin, M. (2006). Anopheles gambiae lipophorin: Characterization and role in lipid transport to developing oocyte. Insect Biochem. Mol. Biol. 36, 375–386. 8. Blandin, S., Shiao, S.H., Moita, L.F., Janse, C.J., Waters, A.P., Kafatos, F.C., and Levashina, E.A. (2004). Complement-like protein TEP1 is a determinant of vectorial capacity in the malaria vector Anopheles gambiae. Cell 116, 661–670. 9. Hayakawa, T., Culleton, R., Otani, H., Horii, T., and Tanabe, K. (2008). Big bang in the evolution of extant malaria parasites. Mol. Biol. Evol. 25, 2233–2239.
Vision: Dialogues between Deep Networks and the Brain Charles E. Connor Department of Neuroscience, Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, MD 21218, USA Correspondence: [email protected] https://doi.org/10.1016/j.cub.2019.05.072
Two new papers show how deep neural networks interacting with the brain can generate visual images that drive surprisingly strong neural responses. These images are tantalizing reflections of visual information in the brain. Artificial intelligence (AI) has been revolutionized in recent years by ‘deep networks’: artificial neural networks, realized in computers, with multiple layers — which define the network ‘depth’ — of processing units that transform input information into output solutions [1]. They learn to produce desired outputs through back-
R634 Current Biology 29, R623–R646, July 8, 2019 ª 2019 Elsevier Ltd.
propagation, a method of adjusting connection weights between units to minimize output errors. Deep networks have been around for a long time [2], but recent advances in processing power have allowed them to smash through longstanding AI challenges like object recognition and master-level Go play. One of the most powerful learning
Current Biology
Dispatches methods is to put deep networks into dialogues with each other. AlphaGo and its successors learned to beat human Go masters in part by playing games against themselves [3] (professional players marvel at the novel strategies evolved in these self-play games). Generative adversarial networks (see below) that learn to ‘fool’ discriminative networks have been used to create realistic scenes, objects, and even human faces that cannot be distinguished from real people [4]. Adversarial attacks also expose the fragility of deep network vision systems by showing how they can be tricked with invisibly subtle pixel changes. What then would happen if you put a biological neural network — made of real neurons, connected by axons and dendrites — into one of these dialogues? Could the artificial network ‘fool’ the biological one? And, if so, with what kinds of inputs? And what would those inputs reveal about the biological network? These questions are addressed in two recent publications, one by Ponce et al. [5], the other by Bashivan et al. [6]. Ponce et al. [5] used a deep generative network (DGN) developed by Dosovitskiy and Brox [7], coupled with a genetic algorithm driven by neural responses, to evolve stimuli that elicit strong responses from monkey inferotemporal cortex, which comprises the final stages in the ventral (object-processing) visual pathway [8]. Each inferotemporal neuron expresses, in electrical spikes, the output from a huge network of neurons in the lower-level ventral pathway and other brain regions with which it connects. This network is trained, through evolution and development, to extract information needed by the brain from visual images. In this sense, it is a very direct counterpart to visually trained deep networks. The DGN was previously trained to produce naturalistic images to fool an adversary network discriminating between natural and artificial images. Thus, the DGN contains a wealth of implicit information about images from the real world. DGNs effectively invert the operations of a deep neural network, taking as inputs ‘codes’, or activation patterns of units in internal layers, and transforming them into images that could produce those activation patterns. Genetic algorithms select ancestor stimuli based on response
strength (the evolutionary ‘‘fitness’’ criterion) and then mutate and/or recombine their elements (in this case, the activation codes) to create new stimuli, which successively converge through many generations toward the neuron’s tuning region. Genetic algorithms have been used previously to study ventral pathway coding with parametrically defined textures and three-dimensional shapes [9–11]. Here, by combining genetic algorithms with DGNs, Ponce et al. [5] were able to evolve stimuli at the most basic level — pixel patterns in images. Each experiment began with 40 codes designed to produce quasi-random textures, a very agnostic starting point (Figure 1). The responses of the neuron (or multi-neuron group) to these images were measured. The next generation comprised the 10 highest response image codes from the previous generation plus 30 new codes created by mutating and recombining parent codes selected probabilistically based on response strength. Evolution continued for up to 250 generations comprising 10,000 synthetic stimuli. The resulting maximum response synthetic stimuli were impressive, though distorted, approximations to natural faces and bodies (when recording from face and body patches in inferotemporal cortex) that typically exceeded maximum responses to natural stimuli (though these were sampled less intensively and without evolution). This is the most compelling evidence yet that cortical face patches are specialized for faces per se, as opposed to something more general, like round objects or visual expertise. Face-like stimuli evolve, improbably, from meaningless noise, with no input or constraint apart from the neural responses themselves. Previous experiments on face patches have always relied on experimenter-selected stimuli. In this new work by Ponce et al. [5], the experimenter input was essentially random, and the DGN/genetic algorithm system could have produced anything else in the astronomical domain of naturalistic images. At the same time, however, the evolved stimuli were complex, abstract, and distorted. They seem to hint mysteriously at some unknown dimensionality underlying face,
body, and object selectivity. Their high responses suggest they could be more fundamental to understanding the visual brain than natural stimuli themselves. Or, they may drive high responses precisely because of their blurred, confused, distorted nature. A number of previous studies [12–14] have shown how ventral pathway neurons respond more strongly to blurred or jumbled stimuli, which may put the visual system into a prolonged state of error resolution. Bashivan et al. [6] studied V4, an earlier stage of the ventral pathway, where receptive fields are smaller and response properties are simpler (no face cells for example) [8]. Their method began with fitting a network-based predictive model of V4 single- or multiunit responses to naturalistic images. The model was a weighted sum of units in a middle layer of a deep network model trained to label natural images. They used this model as an efficient shortcut — rather than evolving stimuli that evoked high V4 responses, they evolved stimuli with high predicted responses from the model. The critical result was that these high-prediction synthetic images did indeed evoke high responses from the V4 neuron itself, typically higher than those evoked by the best naturalistic images, and far higher than the best parametric shape stimuli, providing the strongest possible validation of the model. But, importantly, these synthetic stimuli contained obvious contour and spatial frequency elements in common with high response naturalistic images and shape stimuli. A common complaint about visual neuroscience is that everyone uses different stimuli that cannot readily be compared. Bashivan et al. [6] should be commended for bridging across parametric, naturalistic, and networkgenerated stimuli, and congratulated for observing correlated results. The high predicted and observed responses, and the correlations with other stimulus methods, are the most resounding confirmation to date of how valid and accurate network-based models of ventral pathway function can be. In addition to their practical applications, deep networks, which derive their basic architecture from the visual system, have
Current Biology 29, R623–R646, July 8, 2019 R635
Current Biology
Dispatches Gen 1
Gen 2
Gen 7
Gen 33 Gen 98 Gen145
distortions themselves. The repetitive patterns in Bashivan et al. [8] may drive stronger V4 responses by giving neurons more of what they like. But scientific tests of such hypotheses will require parameterization of the novel images that emerge from deep network/brain dialogues.
REFERENCES
Current Biology
Figure 1. Diagramatic representation of a DGN/genetic algorithm experiment as in Ponce et al. [5]. In generation 1, quasi-random activation patterns (codes, represented here by gray-level pixel patterns) are fed into the DGN, which generates texture-like stimuli. The stimuli are presented to a fixating monkey, and electrical spike responses to the stimuli (represented by the vertical lines along each of four stimulus presentation periods) are recorded from a neuron in inferotemporal cortex. Higher response stimuli (evoking many electrical spikes) contribute to generation 2 via mutation (top two rows) and/or recombination (bottom row). The mutations and recombinations occur at the level of the codes, producing new, partially similar codes, which are again fed into the network to produce images, which are presented as visual stimuli, which in turn evoke spiking responses. This process is repeated through many generations, which in this case gradually evolve toward face-like stimuli. The images shown here derive from real experiments and were provided by Margaret Livingstone, the senior author in [5].
been touted as powerful models and tools for understanding how vision works in the brain [15]. Ponce et al. [5] and Bashivan et al. [6] point to how these models can begin to pay off. As in Ponce et al. [5], the current endpoint is the images themselves. (Bashivan et al. [6] also explored how synthesized images can be used to control the activity of multiple V4 sites, a result not covered here.) But in this case, due to smaller receptive fields and simpler response properties in V4, the images appear much more analytically tractable. They contain unmistakable straight and curved contour/spatial frequency elements for which there are standard methods of quantitative characterization. And, the example results shown in the paper uniformly contain repeating patterns of these elements. This suggests that the network-based models have captured spatially repeating motifs in the biological neural networks underlying V4 responses. These would be a natural basis for the previously observed consistency of shape responses across multiple positions in V4 receptive fields [16]. To return to the questions posed above: (1) Could the artificial network ‘‘fool’’ the biological one? Yes, if evoking
strong responses with artificially constructed images counts as fooling. (2) With what kinds of inputs? With abstract or semi-abstract images that apparently share some characteristics with high response natural and parametric stimuli. (3a) What would those inputs reveal about the biological network? To the extent these images resemble natural and parametric stimuli, they confirm, in a powerful way, previous conclusions based on such stimuli. Deep network dialogues with face patch neurons, even though they start with noise, reliably evolve face-like images, the strongest confirmation yet that these patches are inarguably dedicated to face processing. Reversing network-based models of V4 neurons produces straight and curved contours and spatial frequency patches of the type previously defined with other stimuli [16–19]. (3b) But what new things do they reveal, especially given that they evoke stronger responses than previous stimuli? Here, we run up against the frequently fatal obstacle that images are so complex and highdimensional as to defy easy description or parameterization. The distorted facelike images in Ponce et al. [5] contain some kind of information that elicits unusually strong responses, perhaps the
R636 Current Biology 29, R623–R646, July 8, 2019
1. Kriegeskorte, N., and Golan, T. (2019). Neural network models and deep learning. Curr. Biol. 29, R225–R240. 2. North, G. (1987). A celebration of connectionism. Nature 328, 107. 3. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489. 4. Karras, T., Laine, S., and Aila, T. (2018). A stylebased generator architecture for generative adversarial networks. arXiv, preprint arXiv:1812.04948. 5. Ponce, C.R., Xiao, W., Schade, P.F., Hartmann, T.S., Kreiman, G., and Livingstone, M.S. (2019). Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177, 999–1009.e10. 6. Bashivan, P., Kar, K., and DiCarlo, J.J. (2019). Neural population control via deep image synthesis. Science 364, eaav9436. 7. Dosovitskiy, A., and Brox, T. (2016). Generating images with perceptual similarity metrics based on deep networks. Advances in Neural Information Processing, poster abstract. 8. Kravitz, D.J., Saleem, K.S., Baker, C.I., Ungerleider, L.G., and Mishkin, M. (2013). The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cog. Sci. 17, 26–49. 9. Okazawa, G., Tajima, S., and Komatsu, H. (2015). Image statistics underlying natural texture selectivity of neurons in macaque V4. Proc. Natl. Acad. Sci. USA 112, E351–E360. 10. Yamane, Y., Carlson, E.T., Bowman, K.C., Wang, Z., and Connor, C.E. (2008). A neural code for three-dimensional object shape in macaque inferotemporal cortex. Nat. Neurosci. 11, 1352–1360. 11. Hung, C.C., Carlson, E.T., and Connor, C.E. (2012). Medial axis shape coding in macaque inferotemporal cortex. Neuron 74, 1099–1113. 12. Oleskiw, T.D., Nowack, A., and Pasupathy, A. (2018). Joint coding of shape and blur in area V4. Nat. Commun. 9, 466. 13. Schwiedrzik, C.M., and Freiwald, W.A. (2017). High-level prediction signals in a low-level area of the macaque face-processing hierarchy. Neuron 96, 89–97.
Current Biology
Dispatches 14. Issa, E.B., Cadieu, C.F., and DiCarlo, J.J. (2018). Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals. eLife 7, e42870.
16. Pasupathy, A., and Connor, C.E. (2001). Shape representation in area V4: position-specific tuning for boundary conformation. J. Neurophysiol. 86, 2505–2519.
15. Yamins, D.L., and DiCarlo, J.J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365.
17. Pasupathy, A., and Connor, C.E. (1999). Responses to contour features in macaque area V4. J. Neurophysiol. 82, 2490–2502.
18. Pasupathy, A., and Connor, C.E. (2002). Population coding of shape in area V4. Nat. Neurosci. 5, 1332–1338. 19. Gallant, J.L., Braun, J., and Van Essen, D.C. (1993). Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex. Science 259, 100–103.
Cell Polarity: Getting the PARty Started Mike Boxem and Sander van den Heuvel Developmental Biology, Department of Biology, Faculty of Science, Utrecht University, Padualaan 8, 3584 CH, Utrecht, The Netherlands Correspondence: [email protected] (M.B.), [email protected] (S.v.d.H.) https://doi.org/10.1016/j.cub.2019.05.032
Polarity establishment is a key developmental process, but what determines its timing is poorly understood. New research in Caenorhabditis elegans demonstrates that the PAR polarity system extensively reconfigures before becoming competent to polarize. By inhibiting membrane localization of anterior PAR proteins, AIR-1 (aurora A) and PLK-1 (polo kinase) prevent premature polarization.
Polarity, the asymmetric organization of cellular components, is a near universal aspect of animal cells. Yet, we all started life as a single cell zygote, without a defined front or back and lacking predetermined structures. This poses a clear question: when and how does polarity arise during development? Much insight has come from studies in Caenorhabditis elegans, which revealed a self-organizing molecular network that polarizes the zygote by creating two opposing PAR (partitioning defective) protein domains [1,2]. In mammals, PAR protein asymmetry arises when apical and basolateral cellular domains are formed, following compaction of the 8-cell embryo [3]. In contrast, the C. elegans oocyte polarizes shortly after fertilization, triggered by a symmetrybreaking cue from the mature spermderived centrosome pair. This determines the anterior–posterior body axis however, C. elegans mutants that arrest in meiosis without centrosome maturation eventually polarize with a reverse orientation [4–7]. What, then, controls the correct developmental timing and cue for polarization? A new study from Reich et al., recently published in Current Biology, shows that part of the answer lies in the PAR system itself, which has to go through a maturation phase before
becoming competent to polarize the embryo [8]. PAR proteins and many aspects of their functional interactions have been discovered through groundbreaking genetic studies of early C. elegans embryos by Ken Kemphues and colleagues [9]. This work revealed two classes of PAR proteins — one needed for anterior identity and one for posterior identity. The localization of PAR proteins in the one-cell embryo corresponds with these functions, but only after polarity establishment. Following fertilization, the oocyte pronucleus first needs to complete meiosis I and II. At the end of this process, the anterior PAR (aPAR) proteins PAR-3, PAR-6 and PKC-3 occupy the entire cell membrane or underlying cortex, while the posterior PAR (pPAR) proteins PAR-1 and PAR-2 are spread through the cytoplasm. Shortly after completion of meiosis, this configuration suddenly changes (symmetry breaking) to one where the aPAR and pPAR proteins form two opposing domains at the cell cortex (Figure 1). How the PAR proteins achieve their polarized localization has been intensively studied. Reorganization of the actomyosin cytoskeleton beneath the
cell membrane contributes an important mechanism [2]. The proximity of the paternal centrosome pair locally inhibits actomyosin contractility, which triggers an anterior-directed cortical actomyosin flow and removes the aPAR proteins from the posterior pole through advective transport. This allows the pPAR proteins to load onto the cortex. Mutually inhibitory interactions between the two groups of PAR proteins then establish and maintain a stable polarized state [2]. In the absence of cortical flows, a secondary mechanism for symmetry breaking becomes apparent, in which centrosomal microtubules locally protect PAR-2 from inhibitory phosphorylation by PKC-3 [10]. While not essential, this mechanism does serve to accelerate the process of polarization in wild-type embryos. The timing of symmetry breaking depends on maturation of the centrosome, and an experimentally induced delay or block in centrosome maturation causes a delay or failure in polarization [5,11,12]. These observations led to a model in which maturation of the paternal centrosome provides a temporal cue in the formation of a single axis of polarity upon completion of meiosis. However, the reverse polarity in meiotically-arrested
Current Biology 29, R623–R646, July 8, 2019 ª 2019 Elsevier Ltd. R637
Dispatches number of oocysts that colonize a midgut is generally believed to be lower than in laboratory conditions. Additionally, whatever parasite-killing mechanism may be at play, it is inhibited during oogenesis. Once this mechanism is identified, it will be critical to test whether it also impacts mosquito oogenesis. If not, any anti-malarial strategy employing this mechanism would reduce Plasmodium numbers without accelerating P. falciparum transmission. On the other hand, an anti-malarial strategy targeting the mosquito’s oogenesis program per se may come at a cost, as it would lead to faster parasite development and earlier transmission. This is particularly relevant because female mosquitoes have a short life span, not more than 2–3 weeks in the field, and so any accelerated growth by the parasite (which usually takes 10–14 days to develop) would lead to increased transmission even if numbers of transmissible parasites are lower. Faster growth occurs because, when egg development is impaired, Plasmodium oocysts can use excess mosquito resources. Importantly, although it accelerates growth, an excess of nutrients does not increase the number of sporozoites produced per oocyst, suggesting that Plasmodium oocysts have a pre-determined number of sporozoites they can generate, and that this number cannot be exceeded. This is likely a consequence of 30 million years of co-evolution [9], which have shaped the oocyst’s developmental process to be as silent as possible, so as not to affect the mosquito’s reproductive fitness and to consequently maintain the population of vectors needed for the perpetuation of the parasite’s life cycle. Ultimately, Plasmodium oocysts act less as parasites and more as commensal organisms. These results are surprising because it has long been suggested that parasitic infections of vectors would impose a reproductive cost for the mosquito [4]. With these results, the authors show that the number of Plasmodium parasites and the development of mosquito eggs are linked in a commensal relationship [5]. The mosquito does not lose its reproductive fitness in spite of the presence of Plasmodium parasites, which benefit twofold from the oogenesis process — for protection and for resources.
Although, by definition, parasitism means that one species — the parasite — exploits a second species — the host — to obtain food, housing, or protection while causing harm to the host, it is not in the parasite’s best interest to weaken its host to a point that it cannot support the parasite anymore. The study by Werling, Shaw, Itoe et al. suggests that Plasmodium– Anopheles interactions are the product of an evolutionary process that may have progressed from a typical parasitic association into a commensal one.
mosquito shapes within-host malaria infectivity and virulence. Nat. Commun. 9, 3474. 4. Schwenke, R.A., Lazzaro, B.P., and Wolfner, M.F. (2015). Reproduction–immunity trade-offs in insects. Annu. Rev. Entomol. 61, 239–256. 5. Werling, K., Shaw, W.R., Itoe, M.A., Westervelt, K.A., Marcenac, P., Paton, D.G., Peng, D., Singh, N., Smidler, A.L., South, A., et al. (2019). Steroid hormone function controls noncompetitive Plasmodium development in Anopheles. Cell 177, 315–325. 6. Bai, H., Gelman, D.B., and Palli, S.R. (2010). Mode of action of methoprene in affecting female reproduction in the African malaria mosquito, Anopheles gambiae. Pest Manag. Sci. 66, 936–943.
REFERENCES 1. Bennink, S., Kiesow, M.J., and Pradel, G. (2016). The development of malaria parasites in the mosquito midgut. Cell. Microbiol. 18, 905–918. 2. Mitchell, S.N., and Catteruccia, F. (2017). Anopheline reproductive biology: Impacts on vectorial capacity and potential avenues for malaria control. Cold Spring Harb. Perspect. Med. 7, 14. 3. Costa, G., Gildenhard, M., Eldering, M., Lindquist, R.L., Hauser, A.E., Sauerwein, R., Goosmann, C., Brinkmann, V., CarrilloBustamante, P., and Levashina, E.A. (2018). Non-competitive resource exploitation within
7. Atella, G.C., Silva-Neto, M.A.C., Golodne, D.M., Arefin, S., and Shahabuddin, M. (2006). Anopheles gambiae lipophorin: Characterization and role in lipid transport to developing oocyte. Insect Biochem. Mol. Biol. 36, 375–386. 8. Blandin, S., Shiao, S.H., Moita, L.F., Janse, C.J., Waters, A.P., Kafatos, F.C., and Levashina, E.A. (2004). Complement-like protein TEP1 is a determinant of vectorial capacity in the malaria vector Anopheles gambiae. Cell 116, 661–670. 9. Hayakawa, T., Culleton, R., Otani, H., Horii, T., and Tanabe, K. (2008). Big bang in the evolution of extant malaria parasites. Mol. Biol. Evol. 25, 2233–2239.
Vision: Dialogues between Deep Networks and the Brain Charles E. Connor Department of Neuroscience, Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, MD 21218, USA Correspondence: [email protected] https://doi.org/10.1016/j.cub.2019.05.072
Two new papers show how deep neural networks interacting with the brain can generate visual images that drive surprisingly strong neural responses. These images are tantalizing reflections of visual information in the brain. Artificial intelligence (AI) has been revolutionized in recent years by ‘deep networks’: artificial neural networks, realized in computers, with multiple layers — which define the network ‘depth’ — of processing units that transform input information into output solutions [1]. They learn to produce desired outputs through back-
R634 Current Biology 29, R623–R646, July 8, 2019 ª 2019 Elsevier Ltd.
propagation, a method of adjusting connection weights between units to minimize output errors. Deep networks have been around for a long time [2], but recent advances in processing power have allowed them to smash through longstanding AI challenges like object recognition and master-level Go play. One of the most powerful learning
Current Biology
Dispatches methods is to put deep networks into dialogues with each other. AlphaGo and its successors learned to beat human Go masters in part by playing games against themselves [3] (professional players marvel at the novel strategies evolved in these self-play games). Generative adversarial networks (see below) that learn to ‘fool’ discriminative networks have been used to create realistic scenes, objects, and even human faces that cannot be distinguished from real people [4]. Adversarial attacks also expose the fragility of deep network vision systems by showing how they can be tricked with invisibly subtle pixel changes. What then would happen if you put a biological neural network — made of real neurons, connected by axons and dendrites — into one of these dialogues? Could the artificial network ‘fool’ the biological one? And, if so, with what kinds of inputs? And what would those inputs reveal about the biological network? These questions are addressed in two recent publications, one by Ponce et al. [5], the other by Bashivan et al. [6]. Ponce et al. [5] used a deep generative network (DGN) developed by Dosovitskiy and Brox [7], coupled with a genetic algorithm driven by neural responses, to evolve stimuli that elicit strong responses from monkey inferotemporal cortex, which comprises the final stages in the ventral (object-processing) visual pathway [8]. Each inferotemporal neuron expresses, in electrical spikes, the output from a huge network of neurons in the lower-level ventral pathway and other brain regions with which it connects. This network is trained, through evolution and development, to extract information needed by the brain from visual images. In this sense, it is a very direct counterpart to visually trained deep networks. The DGN was previously trained to produce naturalistic images to fool an adversary network discriminating between natural and artificial images. Thus, the DGN contains a wealth of implicit information about images from the real world. DGNs effectively invert the operations of a deep neural network, taking as inputs ‘codes’, or activation patterns of units in internal layers, and transforming them into images that could produce those activation patterns. Genetic algorithms select ancestor stimuli based on response
strength (the evolutionary ‘‘fitness’’ criterion) and then mutate and/or recombine their elements (in this case, the activation codes) to create new stimuli, which successively converge through many generations toward the neuron’s tuning region. Genetic algorithms have been used previously to study ventral pathway coding with parametrically defined textures and three-dimensional shapes [9–11]. Here, by combining genetic algorithms with DGNs, Ponce et al. [5] were able to evolve stimuli at the most basic level — pixel patterns in images. Each experiment began with 40 codes designed to produce quasi-random textures, a very agnostic starting point (Figure 1). The responses of the neuron (or multi-neuron group) to these images were measured. The next generation comprised the 10 highest response image codes from the previous generation plus 30 new codes created by mutating and recombining parent codes selected probabilistically based on response strength. Evolution continued for up to 250 generations comprising 10,000 synthetic stimuli. The resulting maximum response synthetic stimuli were impressive, though distorted, approximations to natural faces and bodies (when recording from face and body patches in inferotemporal cortex) that typically exceeded maximum responses to natural stimuli (though these were sampled less intensively and without evolution). This is the most compelling evidence yet that cortical face patches are specialized for faces per se, as opposed to something more general, like round objects or visual expertise. Face-like stimuli evolve, improbably, from meaningless noise, with no input or constraint apart from the neural responses themselves. Previous experiments on face patches have always relied on experimenter-selected stimuli. In this new work by Ponce et al. [5], the experimenter input was essentially random, and the DGN/genetic algorithm system could have produced anything else in the astronomical domain of naturalistic images. At the same time, however, the evolved stimuli were complex, abstract, and distorted. They seem to hint mysteriously at some unknown dimensionality underlying face,
body, and object selectivity. Their high responses suggest they could be more fundamental to understanding the visual brain than natural stimuli themselves. Or, they may drive high responses precisely because of their blurred, confused, distorted nature. A number of previous studies [12–14] have shown how ventral pathway neurons respond more strongly to blurred or jumbled stimuli, which may put the visual system into a prolonged state of error resolution. Bashivan et al. [6] studied V4, an earlier stage of the ventral pathway, where receptive fields are smaller and response properties are simpler (no face cells for example) [8]. Their method began with fitting a network-based predictive model of V4 single- or multiunit responses to naturalistic images. The model was a weighted sum of units in a middle layer of a deep network model trained to label natural images. They used this model as an efficient shortcut — rather than evolving stimuli that evoked high V4 responses, they evolved stimuli with high predicted responses from the model. The critical result was that these high-prediction synthetic images did indeed evoke high responses from the V4 neuron itself, typically higher than those evoked by the best naturalistic images, and far higher than the best parametric shape stimuli, providing the strongest possible validation of the model. But, importantly, these synthetic stimuli contained obvious contour and spatial frequency elements in common with high response naturalistic images and shape stimuli. A common complaint about visual neuroscience is that everyone uses different stimuli that cannot readily be compared. Bashivan et al. [6] should be commended for bridging across parametric, naturalistic, and networkgenerated stimuli, and congratulated for observing correlated results. The high predicted and observed responses, and the correlations with other stimulus methods, are the most resounding confirmation to date of how valid and accurate network-based models of ventral pathway function can be. In addition to their practical applications, deep networks, which derive their basic architecture from the visual system, have
Current Biology 29, R623–R646, July 8, 2019 R635
Current Biology
Dispatches Gen 1
Gen 2
Gen 7
Gen 33 Gen 98 Gen145
distortions themselves. The repetitive patterns in Bashivan et al. [8] may drive stronger V4 responses by giving neurons more of what they like. But scientific tests of such hypotheses will require parameterization of the novel images that emerge from deep network/brain dialogues.
REFERENCES
Current Biology
Figure 1. Diagramatic representation of a DGN/genetic algorithm experiment as in Ponce et al. [5]. In generation 1, quasi-random activation patterns (codes, represented here by gray-level pixel patterns) are fed into the DGN, which generates texture-like stimuli. The stimuli are presented to a fixating monkey, and electrical spike responses to the stimuli (represented by the vertical lines along each of four stimulus presentation periods) are recorded from a neuron in inferotemporal cortex. Higher response stimuli (evoking many electrical spikes) contribute to generation 2 via mutation (top two rows) and/or recombination (bottom row). The mutations and recombinations occur at the level of the codes, producing new, partially similar codes, which are again fed into the network to produce images, which are presented as visual stimuli, which in turn evoke spiking responses. This process is repeated through many generations, which in this case gradually evolve toward face-like stimuli. The images shown here derive from real experiments and were provided by Margaret Livingstone, the senior author in [5].
been touted as powerful models and tools for understanding how vision works in the brain [15]. Ponce et al. [5] and Bashivan et al. [6] point to how these models can begin to pay off. As in Ponce et al. [5], the current endpoint is the images themselves. (Bashivan et al. [6] also explored how synthesized images can be used to control the activity of multiple V4 sites, a result not covered here.) But in this case, due to smaller receptive fields and simpler response properties in V4, the images appear much more analytically tractable. They contain unmistakable straight and curved contour/spatial frequency elements for which there are standard methods of quantitative characterization. And, the example results shown in the paper uniformly contain repeating patterns of these elements. This suggests that the network-based models have captured spatially repeating motifs in the biological neural networks underlying V4 responses. These would be a natural basis for the previously observed consistency of shape responses across multiple positions in V4 receptive fields [16]. To return to the questions posed above: (1) Could the artificial network ‘‘fool’’ the biological one? Yes, if evoking
strong responses with artificially constructed images counts as fooling. (2) With what kinds of inputs? With abstract or semi-abstract images that apparently share some characteristics with high response natural and parametric stimuli. (3a) What would those inputs reveal about the biological network? To the extent these images resemble natural and parametric stimuli, they confirm, in a powerful way, previous conclusions based on such stimuli. Deep network dialogues with face patch neurons, even though they start with noise, reliably evolve face-like images, the strongest confirmation yet that these patches are inarguably dedicated to face processing. Reversing network-based models of V4 neurons produces straight and curved contours and spatial frequency patches of the type previously defined with other stimuli [16–19]. (3b) But what new things do they reveal, especially given that they evoke stronger responses than previous stimuli? Here, we run up against the frequently fatal obstacle that images are so complex and highdimensional as to defy easy description or parameterization. The distorted facelike images in Ponce et al. [5] contain some kind of information that elicits unusually strong responses, perhaps the
R636 Current Biology 29, R623–R646, July 8, 2019
1. Kriegeskorte, N., and Golan, T. (2019). Neural network models and deep learning. Curr. Biol. 29, R225–R240. 2. North, G. (1987). A celebration of connectionism. Nature 328, 107. 3. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489. 4. Karras, T., Laine, S., and Aila, T. (2018). A stylebased generator architecture for generative adversarial networks. arXiv, preprint arXiv:1812.04948. 5. Ponce, C.R., Xiao, W., Schade, P.F., Hartmann, T.S., Kreiman, G., and Livingstone, M.S. (2019). Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177, 999–1009.e10. 6. Bashivan, P., Kar, K., and DiCarlo, J.J. (2019). Neural population control via deep image synthesis. Science 364, eaav9436. 7. Dosovitskiy, A., and Brox, T. (2016). Generating images with perceptual similarity metrics based on deep networks. Advances in Neural Information Processing, poster abstract. 8. Kravitz, D.J., Saleem, K.S., Baker, C.I., Ungerleider, L.G., and Mishkin, M. (2013). The ventral visual pathway: an expanded neural framework for the processing of object quality. Trends Cog. Sci. 17, 26–49. 9. Okazawa, G., Tajima, S., and Komatsu, H. (2015). Image statistics underlying natural texture selectivity of neurons in macaque V4. Proc. Natl. Acad. Sci. USA 112, E351–E360. 10. Yamane, Y., Carlson, E.T., Bowman, K.C., Wang, Z., and Connor, C.E. (2008). A neural code for three-dimensional object shape in macaque inferotemporal cortex. Nat. Neurosci. 11, 1352–1360. 11. Hung, C.C., Carlson, E.T., and Connor, C.E. (2012). Medial axis shape coding in macaque inferotemporal cortex. Neuron 74, 1099–1113. 12. Oleskiw, T.D., Nowack, A., and Pasupathy, A. (2018). Joint coding of shape and blur in area V4. Nat. Commun. 9, 466. 13. Schwiedrzik, C.M., and Freiwald, W.A. (2017). High-level prediction signals in a low-level area of the macaque face-processing hierarchy. Neuron 96, 89–97.
Current Biology
Dispatches 14. Issa, E.B., Cadieu, C.F., and DiCarlo, J.J. (2018). Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals. eLife 7, e42870.
16. Pasupathy, A., and Connor, C.E. (2001). Shape representation in area V4: position-specific tuning for boundary conformation. J. Neurophysiol. 86, 2505–2519.
15. Yamins, D.L., and DiCarlo, J.J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365.
17. Pasupathy, A., and Connor, C.E. (1999). Responses to contour features in macaque area V4. J. Neurophysiol. 82, 2490–2502.
18. Pasupathy, A., and Connor, C.E. (2002). Population coding of shape in area V4. Nat. Neurosci. 5, 1332–1338. 19. Gallant, J.L., Braun, J., and Van Essen, D.C. (1993). Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex. Science 259, 100–103.
Cell Polarity: Getting the PARty Started Mike Boxem and Sander van den Heuvel Developmental Biology, Department of Biology, Faculty of Science, Utrecht University, Padualaan 8, 3584 CH, Utrecht, The Netherlands Correspondence: [email protected] (M.B.), [email protected] (S.v.d.H.) https://doi.org/10.1016/j.cub.2019.05.032
Polarity establishment is a key developmental process, but what determines its timing is poorly understood. New research in Caenorhabditis elegans demonstrates that the PAR polarity system extensively reconfigures before becoming competent to polarize. By inhibiting membrane localization of anterior PAR proteins, AIR-1 (aurora A) and PLK-1 (polo kinase) prevent premature polarization.
Polarity, the asymmetric organization of cellular components, is a near universal aspect of animal cells. Yet, we all started life as a single cell zygote, without a defined front or back and lacking predetermined structures. This poses a clear question: when and how does polarity arise during development? Much insight has come from studies in Caenorhabditis elegans, which revealed a self-organizing molecular network that polarizes the zygote by creating two opposing PAR (partitioning defective) protein domains [1,2]. In mammals, PAR protein asymmetry arises when apical and basolateral cellular domains are formed, following compaction of the 8-cell embryo [3]. In contrast, the C. elegans oocyte polarizes shortly after fertilization, triggered by a symmetrybreaking cue from the mature spermderived centrosome pair. This determines the anterior–posterior body axis however, C. elegans mutants that arrest in meiosis without centrosome maturation eventually polarize with a reverse orientation [4–7]. What, then, controls the correct developmental timing and cue for polarization? A new study from Reich et al., recently published in Current Biology, shows that part of the answer lies in the PAR system itself, which has to go through a maturation phase before
becoming competent to polarize the embryo [8]. PAR proteins and many aspects of their functional interactions have been discovered through groundbreaking genetic studies of early C. elegans embryos by Ken Kemphues and colleagues [9]. This work revealed two classes of PAR proteins — one needed for anterior identity and one for posterior identity. The localization of PAR proteins in the one-cell embryo corresponds with these functions, but only after polarity establishment. Following fertilization, the oocyte pronucleus first needs to complete meiosis I and II. At the end of this process, the anterior PAR (aPAR) proteins PAR-3, PAR-6 and PKC-3 occupy the entire cell membrane or underlying cortex, while the posterior PAR (pPAR) proteins PAR-1 and PAR-2 are spread through the cytoplasm. Shortly after completion of meiosis, this configuration suddenly changes (symmetry breaking) to one where the aPAR and pPAR proteins form two opposing domains at the cell cortex (Figure 1). How the PAR proteins achieve their polarized localization has been intensively studied. Reorganization of the actomyosin cytoskeleton beneath the
cell membrane contributes an important mechanism [2]. The proximity of the paternal centrosome pair locally inhibits actomyosin contractility, which triggers an anterior-directed cortical actomyosin flow and removes the aPAR proteins from the posterior pole through advective transport. This allows the pPAR proteins to load onto the cortex. Mutually inhibitory interactions between the two groups of PAR proteins then establish and maintain a stable polarized state [2]. In the absence of cortical flows, a secondary mechanism for symmetry breaking becomes apparent, in which centrosomal microtubules locally protect PAR-2 from inhibitory phosphorylation by PKC-3 [10]. While not essential, this mechanism does serve to accelerate the process of polarization in wild-type embryos. The timing of symmetry breaking depends on maturation of the centrosome, and an experimentally induced delay or block in centrosome maturation causes a delay or failure in polarization [5,11,12]. These observations led to a model in which maturation of the paternal centrosome provides a temporal cue in the formation of a single axis of polarity upon completion of meiosis. However, the reverse polarity in meiotically-arrested
Current Biology 29, R623–R646, July 8, 2019 ª 2019 Elsevier Ltd. R637