- Email: [email protected]
Visualizations of Digital Interaction in Daily Life
Available online at www.sciencedirect.com
Computers and Composition 28 (2011) 224–234
Visualizations of Digital Interaction in Daily Life Andrew Morrison a,∗ , Timo Arnall b a
Research Coordinator, Institute of Design, Oslo School of Architecture and Design (AHO), Pb 6768 St. Olavsplass, 0130 Oslo, Norway b Institute of Design, Oslo School of Architecture and Design (AHO), Pb 6768 St. Olavsplass, 0130 Oslo, Norway
Abstract We inquire into how visual signage may make aspects of ubiquitous computing technologies visible and how digital tools and platforms impact that visual design and semiosis. We explore how visual interfaces have been designed and mediated within a design-research project that looked at technologies for mediating between digital media and the physical environment. Referring to visual and media theory, and to social semiotics, we problematize the role of interfaces and interaction with respect to advertising, services, and signage in urban settings. We link our interpretation to critical reflection on designed symbols that express potentials and implications of an emerging technology. This is elaborated as pointing towards a kind of research-by-design practice that we call “discursive design.” © 2011 Elsevier Inc. All rights reserved.
Norwegian Abstract I denne artikkelen undersøker vi hvordan allestedsnærværende datateknologi synliggjør, visuelle tegn og hvordan gjennomgripende digitale verktøy og plattformer innvirker på visuell design og semiose. Vi undersøker hvordan visuelle grensesnit er designet og formidlet innen et designforskningsprosjekt som studerte teknologier for formidling mellom digitale medier og det fysiske miljø. Med utgangspunkt både i medieteorier, sosial semiotikk og visuell teori, problematiserer vi rollen som grensesnitt og samhandling/bruk har innen reklame, tjenester og tegn i urbane omgivelser. Videre er vår kritiske refleksjon om designede symbolers muligheter og konsekvenser for teknologi under utvikling knyttet til en form for design- og forskningspraksis vi refererer til som ‘diskursiv design’. 1. Framings 1.1. Communication, context, and technology Walking through a Tokyo, Japan Metro subway station, a commuter may point a camera phone at an advertising billboard (see Figure 1), and in a few seconds the image is registered and directs the viewer to a specially designed website for Northwest Airlines. The black and white pattern is a matrix barcode similar to those that airline passengers now find on their electronic boarding passes. In this instance of public display in the Tokyo Metro, the visual code has been shifted from its intended procedural context of use to a persuasive one—that of digital advertising. This bold ∗
Corresponding author. E-mail addresses: [email protected] (A. Morrison), [email protected] (T. Arnall). URLs: http://www.aho.no/no/User-pages/Fag/A/Andrew-Morrison/ (A. Morrison), http://www.yourban.no (A. Morrison), http://www.aho.no/no/User-pages/Fag/T/Timo-Arnall/ (T. Arnall), http://www.elasticspace.com/ (T. Arnall). 8755-4615/$ – see front matter © 2011 Elsevier Inc. All rights reserved. doi:10.1016/j.compcom.2011.07.004
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
225
Figures 1 and 2. An advert on Tokyo, Japan subway system, applying a QR code as symbol wrapped along a series of pillars (Image © Mattias Berg, used with permission) and a Tokyo subway station with colored matrix barcodes arranged into pixels (Image used with Creative Commons license, courtesy Jan Chipchase, used with permission).
display of technical imagery, its actual meaning invisible to the human eye, plays on a cultural curiosity for novel technology. Graphic mediations such as this one signify means of access to electronic content and semiosis. The “markings” embody the layering of digital services, communications, and media in physical products and urban spaces as part of the changing practices of everyday life. The markings point to situations in which mobile devices and networks are essential in reading, interpreting, and having agency over contemporary meanings and interactions in public settings. This is apparent in a second example of Tokyo Metro advertising for an Animé conference (see Figure 2) in which popular cultural characters and matrix bar codes are co-presented. The impetus for creating ubiquitous, “seamless” technologies that link the digital with the physical emerged from a perceived gap between physical spaces/objects and mobile devices (e.g., Want, Harrison, Fishkinto, & Gujar, 1999). Following the initial propositions of Mark Wesier (1991), ubiquitous computing or “ubicomp” is premised upon notions of invisibility, ambience, and seamless access through all-pervasive wireless networks and embedded computation. Technologies such as Radio Frequency Identification (RFID) and Bluetooth are built using electromagnetic fields through which information and data flows. These fields are not visible to the human eye, so indexical, visual “markups” or signs for these technologies need to be designed and interpreted. Little research has been carried out on these technologies from a communication and interaction design perspective that studies the technologies’ means of transmission and modes of mediation. In this article, we take up such research with reference to social semiotics and “discursive design.” In so doing, we connect approaches originating in multimodal applied linguistics and design studies and practice. As city dwellers, consumers, and mobile communicators, we are already actively involved in using these technologies in our daily lives. How they are designed and how they may be understood through making their technological affordances visible opens communication and interaction design to new modes of visualization and meaning making. 1.2. Research focus, formats, and methods The main questions we address are how visual signage may manifest aspects of ubicomp technologies and how pervasive digital tools and platforms impact on that visual design and related semiosis. We take up this bi-directional process through discursive design. We elaborate on this term to help situate perspectives on designing with digital materials and their related analyses via social semiotics with its focus on remediation, discourse practice, and multimodal communication. Such perspectives are often overshadowed by more functional perspectives in mainstream Human Computer Interaction (HCI), where the focus is on the technology or its use. In design and in analysis, social semiotics and ubiquitous computing are seldom connected and usually do not feature in one another’s experimental processes of inquiry or research publications. We have linked these two domains within a multi-level, collaborative, interdisciplinary project called Touch (). The Touch project examined the mobile phone as a platform for mediating between peo-
226
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 3 & 4. Two RFID (Radio Frequency Identification) interfaces situated in public space. Touching these active areas with an RFID-enabled device will initiate transactions such as paying for groceries (left) or a activating a subway ticket (right).
ple and their physical environment and included practice-based design research experiments. First, we studied examples of visual mark-up of ubicomp technologies in the world. We needed to consider how examples of technologies such as barcodes, RFID (see Figures 3 and 4) and Bluetooth systems are mediated in everyday use at a contextual, cultural, and communicative level. Second, we conducted design experiments that explored invisible wireless technology by prototyping and visualizing potential products, services, and communications material. We see two main developments in the digital mediation of information in everyday public life. First, digital information is increasingly being mediated on physical surfaces, from massive screens to small swipe cards. Second, there are now elaborate and invisible layers of infrastructure that wirelessly convey media to mobile devices. In this article, therefore, we illustrate the cultural and communicative contexts of interaction design by connecting design practice and research; multimodal discourse studies and social semiotics; and visual analysis that relates to such developments and their uptake in advertising, media, popular culture, and the city. Methodologically, a non-directive visual ethnographic study was carried out over four years, spanning 14 countries. This study included roaming streets and buildings to photograph and observe the manifestations of ubicomp technologies in various cities. Images of these manifestations were archived on the photo sharing website Flickr at
1.3. Discursive Design & Social Semiotics 1.3.1. The discursive in design Design research is an amalgam of disciplines and professions, and it includes practice-based modes of inquiry. Often labeled Research by Design (e.g. Morrison & Sevaldson, 2010), design research is conducted in a reflexive and iterative interplay between materials, experimentation, and use that entail creativity, innovation, commercialization, and, increasingly, collaboration. In addition to these aspects, the mediation of design processes, material explorations, and the potential roles and uses of products and services demands attention at a communicative level. “Discursive design” bridges the technical and the semiotic in the domains of digital design and communication. Timo Arnall and Einar Sneve Martinussen (2010) refer to the art-centered design and research of Dunne Anthony (1999) called “critical design” as an important antecedent to discursive design. Andrew Morrison and Henry Mainsah (2011, this issue) argue that “A discursive design view is oriented toward the manifestation of designs that aim to reach beyond . . . providing solutions to problems.” Morrison and Mainsah argue that the object of discursive design analysis is communication informed by research and developments in social semiotics and multimodality, as well as from more technical domains, we would add, such as ubicomp and Human Computer Interaction (HCI).
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
227
1.3.2. Social semiotics, media, and design research Discursive design is made up of mixed modes, materials, and mediations. Kress Gunther and van Leeuwen Theo (2001) proposed that discourse be studied as multimodal, that, is in relation to different modes (e.g. written, gestural) and to the increasing role of media in contemporary communication. In this view, meaning is made through the realization of modes and media in contexts of cultural use. For Gunther Kress (2010), signs are made with very many different means, in many different modes. They are the expression of the interest of socially formed individuals, who with these signs, realize–give outward expression to–their meanings, using culturally available semiotic resources, which have been shaped by the members of social groups and their cultures (p. 10). Referring to the high level of abstraction of linguistics and semiotics, Kress (2010) asserts that “By entire contrast, the study of modes in multimodal social semiotics focuses on the material, the specific, the making of signs now, in this environment for this occasion” (p. 13, emphasis original). He further argues that we make and remake signs in our semiotic interaction, an interaction that is ongoing and embodied. Liu Yu and O’Halloran Kay (2009) see the analysis of multimodal discourse as undergoing two main shifts, first from grammar to non-verbal modes and semiotic resources, and second, to meaning making, modes, and texts that are composed and enacted by multiple semiotic systems and signs (pp. 367-368). We suggest a third move that better connects the multi-semiotic and the technical in regard to design and mediated meaning making. Andrew Morrison, Even Westvang, and Simen Skogsrud (2010) have framed such an approach as communication design that examines the composition of multimodal articulations, that is, what it is possible to say (utterance) and how it is mediated via a mix of discourse modes in which the technological is also given weight. This accords with the notion of multiple mediation advanced by Susannne Bødker and Peter Bøgh Andersen (2005). Textual analyses of emerging phenomena also benefit from being conducted alongside rich design cases that take up the prospective, communicative potential of digital tools and technologies. Paul Prior and Julie Hengst (2010) argue, however, that this shaping and mediation cannot only be seen in terms of different discourse modes. An emphasis on semiotic remediation in discourse practice is also needed. Yet this discourse practice does not lie only in patterns and parameters of use; it is also embodied within textual artifacts and their visual character and relations to seemingly seamless technologies. This is what we take up in joining a macro level social semiotics with discursive design projects, examples, and analyses that relate to the speculative, emergent, and exploratory as they exist in, and through, design. Thus far, little social semiotic analysis has been closely carried out between designers, computer scientists, and media and humanities scholars (Morrison, 2010). As a step in this direction, below we identify examples of indexicality; the graphic and iconic qualities of markers of identification; and specific symbolic calls to meaningful action in the context of the “mediated city.” Making material from ubicomp technology, visualizing the invisible, and highlighting these mediations accentuates a communication design that offers new means to interaction. However, it also challenges us as designers, educators, and researchers to unpack the seemingly seamless and to see how ubicomp technology can be put to work in our daily urban lives. 1.3.3. Digital advertising and the city Urban spaces are increasingly media spaces (McQuire, 2008) with large urban screens that offer audio-visual, and interactive, situated content. In dense urban environments and within transit infrastructures like airports, signage contributes strongly to a sense of place (Baines & Dixon, 2003). Signage systems, often derived from codes created in Otto Neurath’s (2010) “Isotype” work, offer navigational aids that are strongly implemented in architectural space, often with instructional and symbolic imagery that are concurrent with written language. The manifestations of ubicomp technologies are seldom linked to research into digital media and advertising, nor are they connected through interaction design research, to studies of media, technology, and the city. Nick Couldry and Anna McCarthy (2004) have argued that we need see cities as cultural and mediated spaces. In Urban Studies, David Shane and Brian McGrath (2005) argues today’s cities are viewed as networked and mediated environments that are “senseable” (Zardini, 2005) and in which mobile media exchanges and seamless interaction feature. Amanda Williams and Paul Dourish (2006) argue that ubicomp technologies need to be situated in relation to cultural frames, not just technical ones. In contrast to earlier research on print advertising (e.g., Cook, 2001) and more recent multimodal discursive analyses, little research has been carried out on the cultural and mediational significations of digital advertising. Rarely does social semiotic analysis of electronic adverts contain any substantial references to technological aspects of mediations and the “remediation” of earlier media and messages into different forms and channels (Bolter & Grusin, 1999). Andrew
228
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Morrison and Synne Skjulstad (2010) have argued, however, for a communication design focus on digital advertising that is located in a sociocultural approach to mediated meaning making that includes relations between the mediational and the technical and that connects communication, context, and culture. 1.3.4. Surfaces and layers We see two significant shifts occurring in the digital mediation of information in everyday public life. First, physical surfaces are increasingly intended to mediate digital information, whether by directing people to the World Wide Web, providing digital display through large urban screens, or digitally connecting urban spaces and the World Wide Web through situated interfaces such as matrix barcodes, WiFi, or RFID. As the Web became a marketing tool in the mid1990s, advertising was increasingly appended with URL addresses pointing to websites. As the Web became culturally accepted, meaning and content often disappeared from physical advertising, with a call to action to “complete” the experience by going online. More recently, the emphasis has shifted to directing an audience toward product names or keywords that can be more easily discovered through search and other forms of unique identification, such as references to usernames or hashtags on the social media site Twitter.1 The increasing use of signage and urban screens to point toward online digital media shows the “digitalization” of surfaces in everyday environments. In addition, everyday spaces have been saturated with mostly invisible layers of infrastructure to deliver digital media to mobile devices across wireless networks. Personal devices are increasingly able to deliver digital information through small screens or to interact with the physical world through sensors such as cameras, accelerometers, and various location-sensing wireless interfaces. This has led to an explosion of media that is situated in physical environments, from mapping services such as Google Maps supports a variety of online communicative practices and introduced new namespaces to describe people and subjects. Users are referred to using the @ symbol, so @username is used for authorship and attribution while the # (hash) symbol is used to refer to subjects, places, or concepts (Boyd et al., 2010). These symbols have begun to filter into printed material and advertising. For example, see
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
229
Figures 5 & 6. At left, a sticker on street pole in Slovenia using barcode as a graphic element. At right, a matrix code used to advertise mobile banking in Belgium.
as social code (Dawkins, 1976)—is present in the name for an event. Second, the iconic barcode is transformed into an image of a zebra. Third, at the bottom of the sticker, a URL is given as context and as a source for information. As discursive design, visualizations such as this one are symbolic as well as informational markers. They signal not only their functional relations but also a set of connotations of about technology, identity, and identification. In cases where these visual codes have been “re-semioticized” from technical to persuasive uses, their appearance may provoke curiosity or connote a brand’s cultural image as “cutting edge.” Optically, visual codes create what Edward Tufte (1983) would call “unintentional optical art” that vibrate and distract as patterns interact with the physiology of our optical nerves (p. 107). In a study of users in Finland, Kaj Mäkelä, Sara Belt, Dan Greenblatt, and Jonna Häkkilä (2007) report that barcodes look disturbingly “technological” or “mathematical” for many people. In Figure 6, for example, the novelty of a mobile banking service is symbolized on a glass window by a matrix code. This code provides access to banking services when scanned with a mobile phone; the technical is externalized indexically in the visual pattern, connoting a new, high-tech, and “current” service. As Prior and Hengst (2010) argue, this is an instance of semiotic remediation, where a rhetorical persuasive move is achieved through the promotion of the sign of technical code. Maya Lotan attempted to connect computational codes and aesthetic cultural codes from fashion in a project called Urban Seeder. This connection was based on uniquely identifiable visual codes that were generated from organic, abstract, and floral patterns. These patterns were generated through an online social networking site where each user generated unique codes that could be printed on personal items ranging from business cards to clothing (see Figures 7 and 8). Users of the social network would then seek out and scan potential Urban Seeder patterns with cameras embedded in mobile phones. Participation in the service became a new way of observing the world through
Figures 7 & 8. At left, a t-shirt with a textile “Urban Seeder” pattern being identified with a camera phone. At right, a belt with the same Urban Seeder pattern.
230
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 9 & 10. At left, SMS messaging is used to advertise access to a real estate property brochure in Oslo, Norway. At right, SMS instructions printed on a conference badge in Amsterdam, Netherlands.
the discovery of new meanings embedded in otherwise ordinary patterns, which is similar to Kress’ notion that we make and remake signs in our semiotic interaction. 1.4.2. Unique codes The Internet is built upon the ability to find individual devices with unique codes, such as an IP addresses for particular devices on a network, International Standard Book Numbers for books (ISBN), Digital Object Identifiers for published academic articles (DOI), and protocols for networked devices such as the Internet Protocol version 6 (IPv6). How, then, might we identify objects and places? The mass-ubiquity of short-message-service (SMS)-capable mobile phones has led to the implementation of identification systems to layer information, annotations, or communications onto the physical surfaces of everyday life. Research has addressed the communicative and commercial potential of SMS, for example in popular culture and youth studies as well as in electronic commerce. Little research has covered the uses of these forms as symbolic and communicative mark-up that is a part of a persuasive visual, digital public advertising discourse. In Figure 9, a new housing development in central Oslo advertises the availability of its real estate brochure (“prospekt”) by asking members of the public to send the code SCHOUS to the mobile phone number 2012. Rather than printing standard real estate information on the billboard, the advertisers invite potential customers to actively ask for information on their personal devices. Here the practices of social media are embedded in the visually displayed, just as they are in the message about SMS on a conference event badge in Figure 10. In these instances, we see semiotic remediation as means to participation in mobile information, entertainment, and communication. For example, with its distinctive symbol, Yellow Arrow () is about marking things and places (see Figures 11 and 12) with personal stories. By submitting a story through the Yellow Arrow website or by
Figures 11 & 12. At left, a yellow arrow sticker pasted on street furniture. At right, the storefront for Aste 90 in Helsinki, Finland, with a marker to creative stories by Finnish designers (Image from
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
231
Figures 13 & 14. At left, a message on a fluffy pink hat reads, “Hi, I’m thing 9877TBJ, check me out on thinglink.org”. At right, a Thing-linked object tagged with an RFID tag that can be explored by touching the RFID tag with a mobile phone.
SMS, each submission is given a unique shortcode that can then be discovered by others using the same communications channels. The yellow arrow provides a recognizable visual code that builds upon associative relations to signage, the strong connotative elements of which allows the service to spread semiotically through cultural means. In a globalized mass production market, hand-made, crafted, or customized things that express greater individual identity have gained popularity through websites such as
232
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 15 & 16. At left, a Bluetooth beacon asks passers-by to turn on their Bluetooth in order to be sent marketing material for new mobile phones. At right, a sign indicating the position of an RFID reader with a dashed line.
Figure 17. Speculative icons for RFID interaction from spatial properties of RFID fields (far left) to specific applications that might be accessed through an RFID-enabled mobile phone (far right). (Source: The Touch Project).
a mobile phones to read and write to RFID chips embedded in the environment, products, or things. RFID only works over a distance of about ten centimeters, which means that RFID often works via interactional “hotspots” that must be marked on the top of subway turnstiles; next to the doors on buses, trams, and trains; and on “smart” posters (see Figure 16). These interfaces must signal to users that RFID interfaces are available—at hand, to be used—but may also indicate interactional potential through perceptual information and symbols. Both the physical affordances of RFID systems and their symbolic, communicative potential remain largely unexplored in design and technology research. A central theme in the Touch project was to use visual design and communicative media as a means to unpack the qualities, potentials, and implications of RFID technology. A series of workshops between interaction designers, product designers, and graphic designers alongside an ethnographer and technologists were conducted. These sessions included the use ofmethods such as sketching, visual diagramming, modeling, and prototyping. A rich set of visual signs were developed from these workshops (a selection is seen in Figure 17) and speculating on various mediational means in RFID interfaces, from indexical, denotative representations of radio fields to connotative and metaphorical explorations of interactions and applications of RFID. The circular dashed line (see Figure 17, far left) became a shared way of visually representing RFID throughout all of the Touch project’s communicative material. This built upon existing visual tropes for representing hidden features and folding lines that also connote further immaterial properties, such as a notional boundary on a map or a “line of sight” in a comic strip. As a permeable threshold, the dashed line denotes a space of possible interaction and offered a clarity and ease of graphic adaptation that could not be found in other visual semiotic means. Other graphic experiments played with the mobile phone intersecting with RFID fields, visually describing the gesture needed to interact between a device and an RFID tag embedded in the world (Nordby, 2010). Visual signs for applications such as opening a door or paying with an RFID mobile phone were also developed, playing with culturally-shared (and Isotype-inspired) graphic codes for keys, money, and gestures (Figure 17). These signs were developed from the need to communicate about potentials in the technology and to express new modes of interaction to other designers, engineers, and users at a macro level of communication with mobiles and RFID and as signals of micro-level enactments of exchanges. As discursive designs, they signal both the functional relations of RFID and a set of connotations about technology, applications, transactions, and risk.
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
233
These interactive technologies rely on an immaterial substrate through which information and data flows. This only becomes “visible” or “interactional” through a designer’s intervention in a product or environment, in physical or visual expression. The visual “mark-up” of the technology becomes an important discursive design manifestation that communicatively uncovers hidden materials as well as potentials and implications at the level of utterance. 2. Conclusion The composition and mediation of the technologies of ubiquitous computing geared toward our daily lives need to be closely considered because they are embodied via complex and often invisible materials. Discursive design allows interaction designers, teachers, and researchers to place mediational and discursive aspects at the center of their design-related activities when working with digital technologies and emerging situated socio-cultural practices. For design research, this implies that we should emphasize the bindings and distinctions between design processes and visual mediations, and symbols and signs, in engaging with emerging technologies as material for creative and communicative composition. Social semiotics, informed by research in ubicomp and design allows inquiry into interaction and communication design to address cultural, connotative, mediated meaning-making. We have shown how the computational and compositional are already visually layered in urban environments symbolically via indexical and iconic reference and through persuasive and exhibitory means. The settings and illustrations discussed above show that there are possibilities for furthering practice-oriented views of discursive design research that investigate, analyze, and propose new communicative artifacts that link products, services, and mediations with necessary reflections in and on action. To do so, we need to better connect modes, media, and materials in exploring multiple, multimodal mediations in which the visualization of ubiquitous technologies need to be made manifest. It is important that students, designers, educators, and researchers examine further the changing character and contexts of mediated composition and communication as it migrates into the spaces and places of our daily urban lives. We have shown that speculations about the implications of emerging technologies may be informed not only by a “near future” view from ubicomp but also by critical social semiotic analyses of compositional and communicative practices of the “recent present.” Acknowledgments The Touch project (
234
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Morrison, Andrew, & Mainsah, Henry. (2011). African clouds over the Oslo opera. Computers and Composition, 28(3), this issue. Morrison, Andrew, & Sevaldson, Birger. (2010). Getting going: Research by design. Special issue on Research by Design. FORMakademisk, 3(1). Retrieved from http://www.formakademisk.org/index.php/formakademisk/article/view/63 Morrison, Andrew, & Skjulstad, Synne. (2010). Unreal estate: Digital design and mediation in marketing urban residency. In Ina Wagner, Tone Bratteteig, & Dagny Stuedahl (Eds.), Exploring digital design: Multi-disciplinary design practices (pp. 189–220). Vienna: Springer. Morrison, Andrew, Westvang, Even, & Skogsrud, Simen. (2010). Whisperings in the undergrowth: Performativity, collaborative design and online social networking. In Ina Wagner, Tone Bratteteig, & Dagny Stuedahl (Eds.), Exploring digital design: Multi-disciplinary design practices (pp. 221–260). Vienna: Springer. Neurath, Otto, with Eve, Matthew, & Burke, Christopher. (2010). (Eds). From hieroglyphics to Isotype: A visual autobiography. London: Hyphen Press. Nordby, Kjetil. (2010a). Conceptual designing and technology: short-range RFID as design material. International Journal of Design, 4(1). Retrieved from: http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/625 Nordby, Kjetil (2010b). Multi-field relations in designing for short-range RFID. Personal and Ubiquitous Computing. Online FirstTM , 8 April 2010. Retrieved from http://www.springerlink.com/content/x5138828j6j6q102/ Prior, Paul, & Hengst, Julie. (2010). Introduction: Exploring semiotic remediation. In Paul Prior, & Julie Hengst (Eds.), Exploring semiotic remediation as discourse practice (pp. 1–23). Basingstoke, UK: Palgrave Macmillan. Shane, David, & McGrath, Brian. (2005). Sensing the 21st-century city. London: John Wiley & Sons. Sharifi, Matthew, Payne, Terry, & David, Esther. (2006). Public display advertising based on Bluetooth device presence. In Enrico Rukzio, Massimo Paolucci, Tim Finin, Paul Wisner, & Terry Payne (Eds.), Proceedings of the Workshop Mobile Interaction with the Real World (MIRW 2006) September 12, Espoo, Finland, (pp. 52–55). Tufte, Edward. (1983). Visual display of quantitative information. Cheshire, CT: Graphics Press. Want, Roy, Harrison, Beverly, Fishkinto, Kenneth, & Gujar, Anuj. (1999). Bridging physical and virtual worlds with electronic tags. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: The CHI is the Limit, May 15-20, Pittsburgh, PA, (pp. 370–377). Weiser, Mark. (1991 September). The computer for the twenty-first century. Scientific American, 94–104. Williams, Amanda, & Dourish, Paul. (2006). Imagining the city: The cultural dimensions of urban computing. Computer, 39(9), 38–43. Zardini, Mirko (Ed.). (2005). Sense of the city. Baden/Montreal: Lars Muller Publishers/CCA.
Computers and Composition 28 (2011) 224–234
Visualizations of Digital Interaction in Daily Life Andrew Morrison a,∗ , Timo Arnall b a
Research Coordinator, Institute of Design, Oslo School of Architecture and Design (AHO), Pb 6768 St. Olavsplass, 0130 Oslo, Norway b Institute of Design, Oslo School of Architecture and Design (AHO), Pb 6768 St. Olavsplass, 0130 Oslo, Norway
Abstract We inquire into how visual signage may make aspects of ubiquitous computing technologies visible and how digital tools and platforms impact that visual design and semiosis. We explore how visual interfaces have been designed and mediated within a design-research project that looked at technologies for mediating between digital media and the physical environment. Referring to visual and media theory, and to social semiotics, we problematize the role of interfaces and interaction with respect to advertising, services, and signage in urban settings. We link our interpretation to critical reflection on designed symbols that express potentials and implications of an emerging technology. This is elaborated as pointing towards a kind of research-by-design practice that we call “discursive design.” © 2011 Elsevier Inc. All rights reserved.
Norwegian Abstract I denne artikkelen undersøker vi hvordan allestedsnærværende datateknologi synliggjør, visuelle tegn og hvordan gjennomgripende digitale verktøy og plattformer innvirker på visuell design og semiose. Vi undersøker hvordan visuelle grensesnit er designet og formidlet innen et designforskningsprosjekt som studerte teknologier for formidling mellom digitale medier og det fysiske miljø. Med utgangspunkt både i medieteorier, sosial semiotikk og visuell teori, problematiserer vi rollen som grensesnitt og samhandling/bruk har innen reklame, tjenester og tegn i urbane omgivelser. Videre er vår kritiske refleksjon om designede symbolers muligheter og konsekvenser for teknologi under utvikling knyttet til en form for design- og forskningspraksis vi refererer til som ‘diskursiv design’. 1. Framings 1.1. Communication, context, and technology Walking through a Tokyo, Japan Metro subway station, a commuter may point a camera phone at an advertising billboard (see Figure 1), and in a few seconds the image is registered and directs the viewer to a specially designed website for Northwest Airlines. The black and white pattern is a matrix barcode similar to those that airline passengers now find on their electronic boarding passes. In this instance of public display in the Tokyo Metro, the visual code has been shifted from its intended procedural context of use to a persuasive one—that of digital advertising. This bold ∗
Corresponding author. E-mail addresses: [email protected] (A. Morrison), [email protected] (T. Arnall). URLs: http://www.aho.no/no/User-pages/Fag/A/Andrew-Morrison/ (A. Morrison), http://www.yourban.no (A. Morrison), http://www.aho.no/no/User-pages/Fag/T/Timo-Arnall/ (T. Arnall), http://www.elasticspace.com/ (T. Arnall). 8755-4615/$ – see front matter © 2011 Elsevier Inc. All rights reserved. doi:10.1016/j.compcom.2011.07.004
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
225
Figures 1 and 2. An advert on Tokyo, Japan subway system, applying a QR code as symbol wrapped along a series of pillars (Image © Mattias Berg, used with permission) and a Tokyo subway station with colored matrix barcodes arranged into pixels (Image used with Creative Commons license, courtesy Jan Chipchase
display of technical imagery, its actual meaning invisible to the human eye, plays on a cultural curiosity for novel technology. Graphic mediations such as this one signify means of access to electronic content and semiosis. The “markings” embody the layering of digital services, communications, and media in physical products and urban spaces as part of the changing practices of everyday life. The markings point to situations in which mobile devices and networks are essential in reading, interpreting, and having agency over contemporary meanings and interactions in public settings. This is apparent in a second example of Tokyo Metro advertising for an Animé conference (see Figure 2) in which popular cultural characters and matrix bar codes are co-presented. The impetus for creating ubiquitous, “seamless” technologies that link the digital with the physical emerged from a perceived gap between physical spaces/objects and mobile devices (e.g., Want, Harrison, Fishkinto, & Gujar, 1999). Following the initial propositions of Mark Wesier (1991), ubiquitous computing or “ubicomp” is premised upon notions of invisibility, ambience, and seamless access through all-pervasive wireless networks and embedded computation. Technologies such as Radio Frequency Identification (RFID) and Bluetooth are built using electromagnetic fields through which information and data flows. These fields are not visible to the human eye, so indexical, visual “markups” or signs for these technologies need to be designed and interpreted. Little research has been carried out on these technologies from a communication and interaction design perspective that studies the technologies’ means of transmission and modes of mediation. In this article, we take up such research with reference to social semiotics and “discursive design.” In so doing, we connect approaches originating in multimodal applied linguistics and design studies and practice. As city dwellers, consumers, and mobile communicators, we are already actively involved in using these technologies in our daily lives. How they are designed and how they may be understood through making their technological affordances visible opens communication and interaction design to new modes of visualization and meaning making. 1.2. Research focus, formats, and methods The main questions we address are how visual signage may manifest aspects of ubicomp technologies and how pervasive digital tools and platforms impact on that visual design and related semiosis. We take up this bi-directional process through discursive design. We elaborate on this term to help situate perspectives on designing with digital materials and their related analyses via social semiotics with its focus on remediation, discourse practice, and multimodal communication. Such perspectives are often overshadowed by more functional perspectives in mainstream Human Computer Interaction (HCI), where the focus is on the technology or its use. In design and in analysis, social semiotics and ubiquitous computing are seldom connected and usually do not feature in one another’s experimental processes of inquiry or research publications. We have linked these two domains within a multi-level, collaborative, interdisciplinary project called Touch (
226
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 3 & 4. Two RFID (Radio Frequency Identification) interfaces situated in public space. Touching these active areas with an RFID-enabled device will initiate transactions such as paying for groceries (left) or a activating a subway ticket (right).
ple and their physical environment and included practice-based design research experiments. First, we studied examples of visual mark-up of ubicomp technologies in the world. We needed to consider how examples of technologies such as barcodes, RFID (see Figures 3 and 4) and Bluetooth systems are mediated in everyday use at a contextual, cultural, and communicative level. Second, we conducted design experiments that explored invisible wireless technology by prototyping and visualizing potential products, services, and communications material. We see two main developments in the digital mediation of information in everyday public life. First, digital information is increasingly being mediated on physical surfaces, from massive screens to small swipe cards. Second, there are now elaborate and invisible layers of infrastructure that wirelessly convey media to mobile devices. In this article, therefore, we illustrate the cultural and communicative contexts of interaction design by connecting design practice and research; multimodal discourse studies and social semiotics; and visual analysis that relates to such developments and their uptake in advertising, media, popular culture, and the city. Methodologically, a non-directive visual ethnographic study was carried out over four years, spanning 14 countries. This study included roaming streets and buildings to photograph and observe the manifestations of ubicomp technologies in various cities. Images of these manifestations were archived on the photo sharing website Flickr at
1.3. Discursive Design & Social Semiotics 1.3.1. The discursive in design Design research is an amalgam of disciplines and professions, and it includes practice-based modes of inquiry. Often labeled Research by Design (e.g. Morrison & Sevaldson, 2010), design research is conducted in a reflexive and iterative interplay between materials, experimentation, and use that entail creativity, innovation, commercialization, and, increasingly, collaboration. In addition to these aspects, the mediation of design processes, material explorations, and the potential roles and uses of products and services demands attention at a communicative level. “Discursive design” bridges the technical and the semiotic in the domains of digital design and communication. Timo Arnall and Einar Sneve Martinussen (2010) refer to the art-centered design and research of Dunne Anthony (1999) called “critical design” as an important antecedent to discursive design. Andrew Morrison and Henry Mainsah (2011, this issue) argue that “A discursive design view is oriented toward the manifestation of designs that aim to reach beyond . . . providing solutions to problems.” Morrison and Mainsah argue that the object of discursive design analysis is communication informed by research and developments in social semiotics and multimodality, as well as from more technical domains, we would add, such as ubicomp and Human Computer Interaction (HCI).
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
227
1.3.2. Social semiotics, media, and design research Discursive design is made up of mixed modes, materials, and mediations. Kress Gunther and van Leeuwen Theo (2001) proposed that discourse be studied as multimodal, that, is in relation to different modes (e.g. written, gestural) and to the increasing role of media in contemporary communication. In this view, meaning is made through the realization of modes and media in contexts of cultural use. For Gunther Kress (2010), signs are made with very many different means, in many different modes. They are the expression of the interest of socially formed individuals, who with these signs, realize–give outward expression to–their meanings, using culturally available semiotic resources, which have been shaped by the members of social groups and their cultures (p. 10). Referring to the high level of abstraction of linguistics and semiotics, Kress (2010) asserts that “By entire contrast, the study of modes in multimodal social semiotics focuses on the material, the specific, the making of signs now, in this environment for this occasion” (p. 13, emphasis original). He further argues that we make and remake signs in our semiotic interaction, an interaction that is ongoing and embodied. Liu Yu and O’Halloran Kay (2009) see the analysis of multimodal discourse as undergoing two main shifts, first from grammar to non-verbal modes and semiotic resources, and second, to meaning making, modes, and texts that are composed and enacted by multiple semiotic systems and signs (pp. 367-368). We suggest a third move that better connects the multi-semiotic and the technical in regard to design and mediated meaning making. Andrew Morrison, Even Westvang, and Simen Skogsrud (2010) have framed such an approach as communication design that examines the composition of multimodal articulations, that is, what it is possible to say (utterance) and how it is mediated via a mix of discourse modes in which the technological is also given weight. This accords with the notion of multiple mediation advanced by Susannne Bødker and Peter Bøgh Andersen (2005). Textual analyses of emerging phenomena also benefit from being conducted alongside rich design cases that take up the prospective, communicative potential of digital tools and technologies. Paul Prior and Julie Hengst (2010) argue, however, that this shaping and mediation cannot only be seen in terms of different discourse modes. An emphasis on semiotic remediation in discourse practice is also needed. Yet this discourse practice does not lie only in patterns and parameters of use; it is also embodied within textual artifacts and their visual character and relations to seemingly seamless technologies. This is what we take up in joining a macro level social semiotics with discursive design projects, examples, and analyses that relate to the speculative, emergent, and exploratory as they exist in, and through, design. Thus far, little social semiotic analysis has been closely carried out between designers, computer scientists, and media and humanities scholars (Morrison, 2010). As a step in this direction, below we identify examples of indexicality; the graphic and iconic qualities of markers of identification; and specific symbolic calls to meaningful action in the context of the “mediated city.” Making material from ubicomp technology, visualizing the invisible, and highlighting these mediations accentuates a communication design that offers new means to interaction. However, it also challenges us as designers, educators, and researchers to unpack the seemingly seamless and to see how ubicomp technology can be put to work in our daily urban lives. 1.3.3. Digital advertising and the city Urban spaces are increasingly media spaces (McQuire, 2008) with large urban screens that offer audio-visual, and interactive, situated content. In dense urban environments and within transit infrastructures like airports, signage contributes strongly to a sense of place (Baines & Dixon, 2003). Signage systems, often derived from codes created in Otto Neurath’s (2010) “Isotype” work, offer navigational aids that are strongly implemented in architectural space, often with instructional and symbolic imagery that are concurrent with written language. The manifestations of ubicomp technologies are seldom linked to research into digital media and advertising, nor are they connected through interaction design research, to studies of media, technology, and the city. Nick Couldry and Anna McCarthy (2004) have argued that we need see cities as cultural and mediated spaces. In Urban Studies, David Shane and Brian McGrath (2005) argues today’s cities are viewed as networked and mediated environments that are “senseable” (Zardini, 2005) and in which mobile media exchanges and seamless interaction feature. Amanda Williams and Paul Dourish (2006) argue that ubicomp technologies need to be situated in relation to cultural frames, not just technical ones. In contrast to earlier research on print advertising (e.g., Cook, 2001) and more recent multimodal discursive analyses, little research has been carried out on the cultural and mediational significations of digital advertising. Rarely does social semiotic analysis of electronic adverts contain any substantial references to technological aspects of mediations and the “remediation” of earlier media and messages into different forms and channels (Bolter & Grusin, 1999). Andrew
228
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Morrison and Synne Skjulstad (2010) have argued, however, for a communication design focus on digital advertising that is located in a sociocultural approach to mediated meaning making that includes relations between the mediational and the technical and that connects communication, context, and culture. 1.3.4. Surfaces and layers We see two significant shifts occurring in the digital mediation of information in everyday public life. First, physical surfaces are increasingly intended to mediate digital information, whether by directing people to the World Wide Web, providing digital display through large urban screens, or digitally connecting urban spaces and the World Wide Web through situated interfaces such as matrix barcodes, WiFi, or RFID. As the Web became a marketing tool in the mid1990s, advertising was increasingly appended with URL addresses pointing to websites. As the Web became culturally accepted, meaning and content often disappeared from physical advertising, with a call to action to “complete” the experience by going online. More recently, the emphasis has shifted to directing an audience toward product names or keywords that can be more easily discovered through search and other forms of unique identification, such as references to usernames or hashtags on the social media site Twitter.1 The increasing use of signage and urban screens to point toward online digital media shows the “digitalization” of surfaces in everyday environments. In addition, everyday spaces have been saturated with mostly invisible layers of infrastructure to deliver digital media to mobile devices across wireless networks. Personal devices are increasingly able to deliver digital information through small screens or to interact with the physical world through sensors such as cameras, accelerometers, and various location-sensing wireless interfaces. This has led to an explosion of media that is situated in physical environments, from mapping services such as Google Maps
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
229
Figures 5 & 6. At left, a sticker on street pole in Slovenia using barcode as a graphic element. At right, a matrix code used to advertise mobile banking in Belgium.
as social code (Dawkins, 1976)—is present in the name for an event. Second, the iconic barcode is transformed into an image of a zebra. Third, at the bottom of the sticker, a URL is given as context and as a source for information. As discursive design, visualizations such as this one are symbolic as well as informational markers. They signal not only their functional relations but also a set of connotations of about technology, identity, and identification. In cases where these visual codes have been “re-semioticized” from technical to persuasive uses, their appearance may provoke curiosity or connote a brand’s cultural image as “cutting edge.” Optically, visual codes create what Edward Tufte (1983) would call “unintentional optical art” that vibrate and distract as patterns interact with the physiology of our optical nerves (p. 107). In a study of users in Finland, Kaj Mäkelä, Sara Belt, Dan Greenblatt, and Jonna Häkkilä (2007) report that barcodes look disturbingly “technological” or “mathematical” for many people. In Figure 6, for example, the novelty of a mobile banking service is symbolized on a glass window by a matrix code. This code provides access to banking services when scanned with a mobile phone; the technical is externalized indexically in the visual pattern, connoting a new, high-tech, and “current” service. As Prior and Hengst (2010) argue, this is an instance of semiotic remediation, where a rhetorical persuasive move is achieved through the promotion of the sign of technical code. Maya Lotan attempted to connect computational codes and aesthetic cultural codes from fashion in a project called Urban Seeder. This connection was based on uniquely identifiable visual codes that were generated from organic, abstract, and floral patterns. These patterns were generated through an online social networking site where each user generated unique codes that could be printed on personal items ranging from business cards to clothing (see Figures 7 and 8). Users of the social network would then seek out and scan potential Urban Seeder patterns with cameras embedded in mobile phones. Participation in the service became a new way of observing the world through
Figures 7 & 8. At left, a t-shirt with a textile “Urban Seeder” pattern being identified with a camera phone. At right, a belt with the same Urban Seeder pattern.
230
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 9 & 10. At left, SMS messaging is used to advertise access to a real estate property brochure in Oslo, Norway. At right, SMS instructions printed on a conference badge in Amsterdam, Netherlands.
the discovery of new meanings embedded in otherwise ordinary patterns, which is similar to Kress’ notion that we make and remake signs in our semiotic interaction. 1.4.2. Unique codes The Internet is built upon the ability to find individual devices with unique codes, such as an IP addresses for particular devices on a network, International Standard Book Numbers for books (ISBN), Digital Object Identifiers for published academic articles (DOI), and protocols for networked devices such as the Internet Protocol version 6 (IPv6). How, then, might we identify objects and places? The mass-ubiquity of short-message-service (SMS)-capable mobile phones has led to the implementation of identification systems to layer information, annotations, or communications onto the physical surfaces of everyday life. Research has addressed the communicative and commercial potential of SMS, for example in popular culture and youth studies as well as in electronic commerce. Little research has covered the uses of these forms as symbolic and communicative mark-up that is a part of a persuasive visual, digital public advertising discourse. In Figure 9, a new housing development in central Oslo advertises the availability of its real estate brochure (“prospekt”) by asking members of the public to send the code SCHOUS to the mobile phone number 2012. Rather than printing standard real estate information on the billboard, the advertisers invite potential customers to actively ask for information on their personal devices. Here the practices of social media are embedded in the visually displayed, just as they are in the message about SMS on a conference event badge in Figure 10. In these instances, we see semiotic remediation as means to participation in mobile information, entertainment, and communication. For example, with its distinctive symbol, Yellow Arrow (
Figures 11 & 12. At left, a yellow arrow sticker pasted on street furniture. At right, the storefront for Aste 90 in Helsinki, Finland, with a marker to creative stories by Finnish designers (Image from
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
231
Figures 13 & 14. At left, a message on a fluffy pink hat reads, “Hi, I’m thing 9877TBJ, check me out on thinglink.org”. At right, a Thing-linked object tagged with an RFID tag that can be explored by touching the RFID tag with a mobile phone.
SMS, each submission is given a unique shortcode that can then be discovered by others using the same communications channels. The yellow arrow provides a recognizable visual code that builds upon associative relations to signage, the strong connotative elements of which allows the service to spread semiotically through cultural means. In a globalized mass production market, hand-made, crafted, or customized things that express greater individual identity have gained popularity through websites such as
232
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Figures 15 & 16. At left, a Bluetooth beacon asks passers-by to turn on their Bluetooth in order to be sent marketing material for new mobile phones. At right, a sign indicating the position of an RFID reader with a dashed line.
Figure 17. Speculative icons for RFID interaction from spatial properties of RFID fields (far left) to specific applications that might be accessed through an RFID-enabled mobile phone (far right). (Source: The Touch Project).
a mobile phones to read and write to RFID chips embedded in the environment, products, or things. RFID only works over a distance of about ten centimeters, which means that RFID often works via interactional “hotspots” that must be marked on the top of subway turnstiles; next to the doors on buses, trams, and trains; and on “smart” posters (see Figure 16). These interfaces must signal to users that RFID interfaces are available—at hand, to be used—but may also indicate interactional potential through perceptual information and symbols. Both the physical affordances of RFID systems and their symbolic, communicative potential remain largely unexplored in design and technology research. A central theme in the Touch project was to use visual design and communicative media as a means to unpack the qualities, potentials, and implications of RFID technology. A series of workshops between interaction designers, product designers, and graphic designers alongside an ethnographer and technologists were conducted. These sessions included the use ofmethods such as sketching, visual diagramming, modeling, and prototyping. A rich set of visual signs were developed from these workshops (a selection is seen in Figure 17) and speculating on various mediational means in RFID interfaces, from indexical, denotative representations of radio fields to connotative and metaphorical explorations of interactions and applications of RFID. The circular dashed line (see Figure 17, far left) became a shared way of visually representing RFID throughout all of the Touch project’s communicative material. This built upon existing visual tropes for representing hidden features and folding lines that also connote further immaterial properties, such as a notional boundary on a map or a “line of sight” in a comic strip. As a permeable threshold, the dashed line denotes a space of possible interaction and offered a clarity and ease of graphic adaptation that could not be found in other visual semiotic means. Other graphic experiments played with the mobile phone intersecting with RFID fields, visually describing the gesture needed to interact between a device and an RFID tag embedded in the world (Nordby, 2010). Visual signs for applications such as opening a door or paying with an RFID mobile phone were also developed, playing with culturally-shared (and Isotype-inspired) graphic codes for keys, money, and gestures (Figure 17). These signs were developed from the need to communicate about potentials in the technology and to express new modes of interaction to other designers, engineers, and users at a macro level of communication with mobiles and RFID and as signals of micro-level enactments of exchanges. As discursive designs, they signal both the functional relations of RFID and a set of connotations about technology, applications, transactions, and risk.
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
233
These interactive technologies rely on an immaterial substrate through which information and data flows. This only becomes “visible” or “interactional” through a designer’s intervention in a product or environment, in physical or visual expression. The visual “mark-up” of the technology becomes an important discursive design manifestation that communicatively uncovers hidden materials as well as potentials and implications at the level of utterance. 2. Conclusion The composition and mediation of the technologies of ubiquitous computing geared toward our daily lives need to be closely considered because they are embodied via complex and often invisible materials. Discursive design allows interaction designers, teachers, and researchers to place mediational and discursive aspects at the center of their design-related activities when working with digital technologies and emerging situated socio-cultural practices. For design research, this implies that we should emphasize the bindings and distinctions between design processes and visual mediations, and symbols and signs, in engaging with emerging technologies as material for creative and communicative composition. Social semiotics, informed by research in ubicomp and design allows inquiry into interaction and communication design to address cultural, connotative, mediated meaning-making. We have shown how the computational and compositional are already visually layered in urban environments symbolically via indexical and iconic reference and through persuasive and exhibitory means. The settings and illustrations discussed above show that there are possibilities for furthering practice-oriented views of discursive design research that investigate, analyze, and propose new communicative artifacts that link products, services, and mediations with necessary reflections in and on action. To do so, we need to better connect modes, media, and materials in exploring multiple, multimodal mediations in which the visualization of ubiquitous technologies need to be made manifest. It is important that students, designers, educators, and researchers examine further the changing character and contexts of mediated composition and communication as it migrates into the spaces and places of our daily urban lives. We have shown that speculations about the implications of emerging technologies may be informed not only by a “near future” view from ubicomp but also by critical social semiotic analyses of compositional and communicative practices of the “recent present.” Acknowledgments The Touch project (
234
A. Morrison, T. Arnall / Computers and Composition 28 (2011) 224–234
Morrison, Andrew, & Mainsah, Henry. (2011). African clouds over the Oslo opera. Computers and Composition, 28(3), this issue. Morrison, Andrew, & Sevaldson, Birger. (2010). Getting going: Research by design. Special issue on Research by Design. FORMakademisk, 3(1). Retrieved from http://www.formakademisk.org/index.php/formakademisk/article/view/63 Morrison, Andrew, & Skjulstad, Synne. (2010). Unreal estate: Digital design and mediation in marketing urban residency. In Ina Wagner, Tone Bratteteig, & Dagny Stuedahl (Eds.), Exploring digital design: Multi-disciplinary design practices (pp. 189–220). Vienna: Springer. Morrison, Andrew, Westvang, Even, & Skogsrud, Simen. (2010). Whisperings in the undergrowth: Performativity, collaborative design and online social networking. In Ina Wagner, Tone Bratteteig, & Dagny Stuedahl (Eds.), Exploring digital design: Multi-disciplinary design practices (pp. 221–260). Vienna: Springer. Neurath, Otto, with Eve, Matthew, & Burke, Christopher. (2010). (Eds). From hieroglyphics to Isotype: A visual autobiography. London: Hyphen Press. Nordby, Kjetil. (2010a). Conceptual designing and technology: short-range RFID as design material. International Journal of Design, 4(1). Retrieved from: http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/625 Nordby, Kjetil (2010b). Multi-field relations in designing for short-range RFID. Personal and Ubiquitous Computing. Online FirstTM , 8 April 2010. Retrieved from http://www.springerlink.com/content/x5138828j6j6q102/ Prior, Paul, & Hengst, Julie. (2010). Introduction: Exploring semiotic remediation. In Paul Prior, & Julie Hengst (Eds.), Exploring semiotic remediation as discourse practice (pp. 1–23). Basingstoke, UK: Palgrave Macmillan. Shane, David, & McGrath, Brian. (2005). Sensing the 21st-century city. London: John Wiley & Sons. Sharifi, Matthew, Payne, Terry, & David, Esther. (2006). Public display advertising based on Bluetooth device presence. In Enrico Rukzio, Massimo Paolucci, Tim Finin, Paul Wisner, & Terry Payne (Eds.), Proceedings of the Workshop Mobile Interaction with the Real World (MIRW 2006) September 12, Espoo, Finland, (pp. 52–55). Tufte, Edward. (1983). Visual display of quantitative information. Cheshire, CT: Graphics Press. Want, Roy, Harrison, Beverly, Fishkinto, Kenneth, & Gujar, Anuj. (1999). Bridging physical and virtual worlds with electronic tags. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: The CHI is the Limit, May 15-20, Pittsburgh, PA, (pp. 370–377). Weiser, Mark. (1991 September). The computer for the twenty-first century. Scientific American, 94–104. Williams, Amanda, & Dourish, Paul. (2006). Imagining the city: The cultural dimensions of urban computing. Computer, 39(9), 38–43. Zardini, Mirko (Ed.). (2005). Sense of the city. Baden/Montreal: Lars Muller Publishers/CCA.