Geovisual evaluation of public participation in decision making: The grapevine

Journal of Visual Languages and Computing 22 (2011) 305–321

Contents lists available at ScienceDirect

Journal of Visual Languages and Computing journal homepage: www.elsevier.com/locate/jvlc

Geovisual evaluation of public participation in decision making: The grapevine$ Robert Aguirre n, Timothy Nyerges Department of Geography, University of Washington, Box 353550, Seattle, WA 98195, USA

a r t i c l e i n f o

abstract

Article history: Received 18 December 2008 Received in revised form 11 December 2010 Accepted 27 December 2010 Available online 1 January 2011

This article reports on a three-dimensional (time–space) geovisual analytic called a ‘‘grapevine.’’ People often use metaphors to describe the temporal and spatial structure of online discussions, e.g., ‘‘threads’’ growing as a result of message exchanges. We created a visualization to evaluate the temporal and spatial structure of online message exchanges based on the shape of a grapevine naturally cultivated in a vineyard. Our grapevine visualization extends up through time with features like buds, nodes, tendrils, and leaves produced as a result of message posting, replying, and voting. Using a rotatable and fully interactive three-dimensional GIS (Geographic Information System) environment, a geovisual analyst can evaluate the quality of deliberation in the grapevine visualization by looking for productive patterns in fine-grained human– computer–human interaction (HCHI) data and then sub-sampling the productive parts for content analysis. We present an example of how we used the technique in a study of participatory interactions during an online field experiment about improving transportation in the central Puget Sound region of Washington called the Let’s Improve Transportation (LIT) Challenge. We conclude with insights about how our grapevine could be applied as a general purpose technique for evaluation of any participatory learning, thinking, or decision making situation. & 2010 Elsevier Ltd. All rights reserved.

Keywords: Grapevine Geovisual analytics Public participation Decision making Spatio-temporal events Human–computer–human interaction

1. Introduction The last decade and a half has seen significant progress in tool building for online participatory interaction. Cyberinfrastructure tools now exist that are capable of supporting large numbers of people over wide areas in participatory thinking, learning, and decision making activities [1–6]. Especially promising is the potential use of cyberinfrastructure to scale the ‘‘analytic–deliberative’’ decision making process, as advocated by the National Research Council (NRC), to larger numbers of people participating from wider regional areas [1–3]. Analysis, $

This paper has been recommended for acceptance by S.-K. Chang. Corresponding author. E-mail addresses: [email protected] (R. Aguirre), [email protected] (T. Nyerges). n

1045-926X/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.jvlc.2010.12.004

the systematic application of specific theories and methods for interpreting data and drawing conclusions about phenomena, is one way of knowing what course of action to take in decision making. Deliberation, any process for communicating or raising and collectively considering issues, is another way of knowing what course of action to take. In analytic–deliberative decision making, analysis and deliberation are used together as complementary ways of knowing. Over the same decade and a half period of time that has seen significant progress in tool building for online participatory interaction, social scientists evaluating public participation in decision making topics like transportation improvement have not been reporting similar levels of progress when it comes to the quality of interaction. Evaluators have not always found meaningful and diverse interaction when non-experts communicate with experts

306

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

and executives in a public meeting, and the skepticism is not limited to use of online tools [3]. If online systems can be built to scale participatory interactions out to greater numbers of people over a wider regional area, how can they also be built to enhance the quality and the outcome of those participatory interactions? A common explanation for the lack of quality interaction has to do with how public meetings are convened, or how carefully meetings have been structured to ensure meaningful participation. A surprising assumption one often hears is that people are not as engaged online as they are in a face-to-face situation, presumably no matter how carefully public participation has been structured. A number of findings from long-term surveys like the Digital Future Report suggest otherwise, demonstrating significant increase in Internet use by American households engaged in online communities dedicated to social or political issues or in learning outside of the classroom [7]. For example, about 80 percent of Americans surveyed use the Internet, one of the highest rates in the world. A large and growing percentage of those surveyed were members of online communities related to social causes, although it should be noted that relatively low percentages believed the Internet was a tool for public influence in terms of giving people more of a say in what the government does. Still, about one-third of those surveyed in the Digital Future Report agree or strongly agree that by using the Internet they could have more political power, and a large and growing percentage say that going online can help people better understand politics. We tend to view the question whether to structure an online or a face-to-face situation as merely one of methodological preference. Online tools that can work consistently and repeatedly to support representative and broadly based samples of people in structured public participation situations rather than conventional public meetings can certainly be made useful. The fundamental research question is really about exactly what ‘kinds’ of online situation work better than others. A seldom given explanation for the lack of quality interaction during public participation situations has to do with evaluation itself. Are the evaluation methods used by social scientists able to distinguish productive from unproductive interactions at a fine-grained level? In any given participatory situation there may have been many short or isolated periods in which a productive deliberation emerged, but the observations and methods used for evaluation, e.g., summative self-report measures like questionnaires or post-situation interviews, were not able to pick them out [2,3]. Probably the most overlooked advantage in selecting an online system to convene a participatory learning, thinking, or decision making situation is that the system itself can act as a fine-grained data collection tool. An online system can unobtrusively log almost everything that voluntary participants are doing with the system. Ideally, these observations are triangulated with appropriately sub-sampled self-report data from online questionnaires and interviews. With this kind of mixedmethod approach using event log data, evaluators are better prepared to find productive interactions hidden

amongst generally unproductive ones, a pattern that may be symptomatic of situations where large groups of lay people from a wide regional area are asked to voluntarily work together to make choices about highly technical information. There is also a strategic advantage in using online systems to convene participatory interactions. By tying social science evaluation more closely into participant interaction data recorded in a system event log, it ties the needs of evaluators more closely into the world of the designers and developers of the original tool, tightening the feedback loop so to speak. Designers and developers, in turn, might learn to use evaluation results to decide which features were useful and which were not. Even if the results of closer interaction between developers, users, and evaluators results in a series of disagreements about who controls whose needs or whose work drives the whole enterprise we see that as progress, and the results can certainly have a stimulating effect. We expect that entangling the data and methods of social science evaluators into the choices that developers make and the needs that users have will not only result in more advanced systems but measurably better ones. In addressing the fundamental question of how to scale public participation to larger numbers of people over wider regional areas using cyberinfrastructure there are at least two major challenges. One major challenge is improving evaluation. Valid social science research methods are not always followed especially when it comes to sampling. Self-report assessments collected from a participant by the actual convener of the public participation situation at the end of the process, which depending on the research design are vulnerable to the biases or sympathies of the conveners themselves, might simply be unable to distinguish exactly where a deliberation was suddenly productive amidst uneven or declining activity. Another major challenge is to make sure that the results of evaluation, both positive and negative, get back to the designers and developers to improve the system itself. There should be, but seldom is, meaningful interactive feedback between the people who designed and developed an online system, the people who use the system in ways that work best for his or her own specific purposes, and the people who evaluate the process and the outcomes. Investigating the use of cyberinfrastructure to improve the quality and scale of analytic–deliberative decision making is an inherently interdisciplinary project synthesizing three very different domains of research including system design and development, participant use, and social science evaluation. The three different domains of research work together like a ‘‘virtual’’ organization, whether the different groups of people recognize it or not. 1.1. Designing and developing an online tool For projects like ours with the resources to build a custom application, we started from scratch in the domain of system design and development using a conceptual model of ‘‘best process’’ for structuring public participation in decision making [1–3]. A tool was designed and

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

developed by the Participatory GIS for Transportation (PGIST) project using recommendations from the National Research Council about broadly based analytic–deliberative decision making [1–3]. The base software platform has proven flexible enough for multiple uses and has since been modified for a Voicing Climate Concerns (VCC) project about the regional impacts of global climate change on the Oregon coast [8]. The Web portal houses a collection of participatory tools and instructions organized in a sequence with a workflow engine. The design and development team built the Web portal so that participants would have access to a certain set of tools at a certain time to complete an objective or ‘‘Step.’’ The LIT Challenge was designed as a month-long agenda of five steps. Each step contained two or more sub-steps, including twelve in all. For brevity, we focus on our expectations and instructions to participants at the step level. Each step in the LIT Web portal had either a generally ‘‘deliberative’’ or ‘‘analytic’’ objective. To start, participants enter information about his or her travel path and then voice values and concerns about improving transportation in the central Puget Sound region. A moderator performs a synthesis of concerns using a special online tool by grouping them into a set of common themes. Participants review the common themes and then vote on whether they agree that the themes adequately represent the original concerns (LIT Step 1). After voicing his or her concerns and voting, participants review and weigh different factors useful as criteria for selecting a transportation improvement package (LIT Step 2). Each participant then creates a unique package with a set of geospatial analysis tools, first selecting projects from a large spatial inventory of proposed projects all over the central Puget Sound region and then selecting funding mechanisms to cover the cost (LIT Step 3). After each participant creates a transportation improvement package, an off-line process is used to synthesize all of the participant’s contributions and identify six diverse packages. Participants are asked to deliberate about the six packages and then vote on the six packages in order of preference (LIT Step 4). Finally, after the most preferred package is selected, participants review and endorse a final report to agency decision makers and technical specialists about the outcomes of the decision making process and the final package recommendation (LIT Step 5).

307

$20+ billions of dollars for the central Puget Sound region. The research question motivating the LIT online field experiment was, ‘‘What Internet platform designs and capabilities, particularly including GIS technology, can improve public participation in ‘analytic–deliberative’ transportation decision making within large groups?’’ The project used a quasi-experimental research design to test use of the LIT Web portal, crossing a ‘field’ study with a laboratory ‘experiment’ [9]. Online field experiments balance the advantage of a controlled experimental situation with the advantage of observing people interacting in a natural situation over a long period of time, equally important for validity [9–11]. Current federal and state transportation laws mandate public participation in decisions about long-range planning, capital improvement programming, and major investment studies. Thus, it was given that public participation in decision making about improving transportation was both necessary and desirable, something that admittedly may not apply in every participatory situation. As a result, the project enjoyed key collaboration with public agencies in the central Puget Sound region in developing the transportation improvement programming substance of the experiment. The month-long LIT Challenge experimental decision process (15 October 2007–13 November 2007) was timed to coincide with a 6 November 2007 ballot initiative asking voters to support a $17.8 billion regional transportation improvement package for the central Puget Sound region of Washington State, including King, Snohomish and Pierce counties. Fig. 1 displays the unfiltered grapevine visualization georeferenced to the three-county area of the online field experiment. Further below we explain how we used a rotatable and fully interactive version of the three-dimensional time–space grapevine visualization to evaluate the quality of deliberative interactions during the LIT Challenge. In Fig. 1, a 2D top down static view of the grapevine shows the central location of the LIT Web portal server on the University of Washington campus relative to the locations of all the registered participants throughout the central Puget Sound region. Participant locations were

1.2. Convening an online field experiment The LIT Challenge was a month-long, online and asynchronous decision making situation in late 2007 involving about 200+ community participants from a three county area around Seattle, WA who were asked to be part of a citizen advisory group. In total, 246 participants registered for the experiment. Of the 246 registrants, 179 qualified for payment based on geographic criteria, representing the group we call our quota participants. On average, only about half of the 179 quota participants were active in the LIT Challenge at any one time, ranging from a high of 60 percent to a low of 40 percent by the end of the experiment. The participants’ collaborative task was to decide on the best transportation improvement package involving

Fig. 1. Side view and top down view of the grapevine visualization shows the central geographic location of the PPGIS server to participant locations.

308

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

self-reported by informed and consenting voluntary human subjects as either a geocoded home address or home zip code. Based upon online questionnaire responses, most of our participants were actively interested in the topic of improving transportation and felt comfortable using online tools. In other words, we were convening a fairly natural participant use situation and expected high levels of activity. 1.3. Evaluating an online field experiment We evaluated the quality and scale of public participation in the online field experiment using a combination of client–server event log data; online questionnaires administered before, during, and after the experiment itself; a sample of voluntary screen recordings; and finally, a sample of in-depth face-to-face interviews. Over the course of the month-long LIT Challenge, the LIT Web portal logged over 120,000 client–server interaction events. We downloaded the Web portal event database, coded the interaction events by whether they were the result of analytic or deliberative HCHI activities, and then applied our grapevine technique using 3D GIS software. For the remainder of the paper we turn to the development and use of the three-dimensional time–space visualization itself, organized as follows. In Section 2, we preface our explanation of the grapevine with some background about the origins of the three-dimensional geovisualization. We then provide a brief literature review outlining how our grapevine technique synthesized three methodologies including sequential data analysis, social network analysis, and time–space geography. In Section 3, we describe the grapevine and all its organiclooking features, which were designed to be like the anatomy of an actual grapevine plant. We break the complex visual structure into its component features including a main stem, nodes, buds, tendrils, and leaves. We also explain the three different types of human– computer–human interaction events that can generate the component features of the grapevine and why they are important in a visual representation of the quality of deliberation. In Section 4, we report our findings using the grapevine technique in terms of filtering and analyzing productive clusters of deliberation in the LIT Challenge. We then suggest how the grapevine can be used for wider application to any participatory situation. Finally, in Section 5 we conclude by considering how the results of evaluation using geovisual analytic techniques like the grapevine can provide feedback on best practices for system design and system use to improve the quality and scale of public participation in decision making. 2. The grapevine as an evaluation method The purpose in creating a three-dimensional space– time visualization was to balance the power of computing to process fine-grained interaction event data with the human process of spatial thinking [12] using a GIS, under conditions where it was difficult or undesirable to use powerful parametric statistical techniques. An unanticipated outcome of our evaluation phase was that despite

our best efforts we could not use parametric statistical techniques because some basic assumptions could not be met. Most of the difficulties stemmed from the fact that with well over 200 participants and almost as many different event types, the majority of data sets for input into statistical packages were large and sparse matrices with unknown data category frequency distributions, notoriously difficult for parametric statistical methods to deal with. One interesting non-parametric statistical technique we discovered was a categorical analysis called ‘‘configural frequency analysis,’’ a method designed to focus on people rather than variables [13]. With this method we could analyze categorical profiles of participants based on differences in interactions with the LIT Web portal or differences based on self-reported data from questionnaires and interviews. The problem was that the primary research question motivating the LIT Challenge research design did not match an exploratory investigation into how different personal or demographic characteristics generally affected the quality and scale of online participation, even if suggestive. Given our expertise using GIS software, we felt that we could develop a geovisualization for analysis of human– computer–human interaction data. Our geovisualization would be capable of displaying very large amounts of data in a form that human spatial thinking could make sense of without unanticipated or unintended confusion about patterns. To come up with the grapevine idea, we looked at three different bodies of research including exploratory sequential data analysis [14], social network analysis [15,16], and time–space or time geography emphasizing visualization [17–23]. 2.1. Sequential: exploratory sequential data analysis (ESDA) The analysis of sequences has a long past and there have been many statistical techniques offered. For instance, a geographic treatment by Getis [24] described a quantitative method for exploring adjacent categories of things that he suggests could be used to study land use types as observed from the window of a moving train as easily as it could be used to study soil type cross-sections in physical geography. Contemporary sequential analysis techniques have become quite sophisticated. For example, Magnusson’s [25] ‘‘T-patterns’’ method can find recurring sequential patterns of event types that are not necessarily adjacent but repeat in the same order hidden in the midst of other event types. Exploratory Sequential Data Analysis (ESDA) represents another set of techniques for characterizing the sequential structure of events in time and it is widely used in human–computer interaction. The side view graphic in Fig. 1 represents a sequential structure of events, but our emphasis is as much on the location of the interaction events in geographic space as it is on their occurrence in time. ESDA is explained in much greater detail in Sanderson and Fisher [14]. The general purpose HCI spectrum for ESDA is composed of eight event types, ranging from a fine-grained mouse-click or keyboard press that lasts a millisecond, to coarse-grained social

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

activities lasting hours, to project activities lasting months. Depending on the theoretical research question, one researcher might tend to code every single mouseclick and keyboard press whereas another researcher might want to lump fine-grained activity into a single long act like ‘‘using the computer.’’ Scientists who start out preferring to ask cognitive, behavioral or social theoretical questions will generally choose different ranges of the HCI activity spectrum to code. Cognitive and behavioral researchers prefer high-frequency and short-interval observations like eye movements or mouse-clicks. Social researchers prefer low-frequency and long-interval observations like deliberating or coming to a consensus. The HCI spectrum does not simplify every important difference between HCI researchers to that of merely theoretically recognizing one part of the HCI spectrum while being blind or deaf to others, since there are statistical and grammatical method differences as well [26]. However, there are trade-offs and overlaps between behavioral, cognitive and social theoretical approaches to the study of HCI. A spectrum of HCI time intervals has been also discussed by other authors [27,28], who list micro-scale acts affecting human cognitive and behavioral performance with visual displays of information on a computer. We used a similar idea in thinking about how to infer client–server events against a continuous spectrum of more ‘‘analytic’’ versus more ‘‘deliberative’’ HCHI activities. Researchers in fields ranging from computer science and communications to geography and sociology have used ESDA techniques to interpret coded sequences of HCI events in single user, group face-to-face, and group online experimental settings. ESDA techniques have also been incorporated into a software application called MacShapa [29]. In the late 1990s, Jankowski and Nyerges [30] used MacShapa to analyze dozens of hours of videotaped group decision making by systematically coding consecutive 30-s intervals of time into discrete categories of decision making activity, and then seeing if certain expected sequences of coded activity tend to occur more frequently. Researchers have applied sequential techniques to verbal protocol and client-side interaction log data in order to analyze small group use of a geovisualization tool for the purpose of usability and cognitive testing [31,32]. Other researchers like Tanimoto, Hubbard, and Winn [33] and Keel [34] have reported on the design and development of visualizations to support group sensemaking and learning using computational agents to unobtrusively gather data about sequences of individual user activities, and then perform an analysis and return the results back to participants as a form of instant feedback. 2.2. Social: social network analysis Social network analysis has a long past in sociometry and has become a popular method for analyzing relationships between people, data, computers, and concepts on the Internet. The early origins of social network analysis in sociometry and sociology are discussed elsewhere [15,16]. A social network analysis can infer social roles (e.g., centrality in communication, broker, leader, follower,

309

etc.) based on the frequency of who communicates with whom and for how long. Researchers in a number of computer fields outside of sociology have used social network analysis of event log data for process mining in business enterprise systems because event logs are ‘‘process-aware’’ [15,16]. Unlike e-mail traffic, event logs record what step within a structured process an event took place, e.g., whether participants exchanged a message while working on LIT Step 1 or LIT Step 2. Process mining is a method for exploring how people can use the same tool but work together in very unique sequences of work, based on actual executions by human users as opposed to simply describing an expected process based on a formal workflow diagram. A process mining approach to social network analysis visualizes not only the frequency of interpersonal relationships but also the roles of people, data, computers, and concepts within the context of tasks in a ‘‘process.’’ Process mining with event logs can illustrate how users working together tend to deviate from the normal workflow, which is important information when a system has been designed to be flexible. In a sense, process mining combines an exploratory sequential data analysis of an unfolding process of interaction with a social network analysis of emerging social roles and relationships. Although the identification of roles in a process can provide useful insights, the influence of temporal and spatial context is largely unaccounted for when charting social networks. Social network analyses rest on a mere snap shot of social relations across a featureless physiographic space called a ‘‘sociogram.’’ Geographic constraints affecting the exchange of information and messages over the Internet might seem irrelevant, or at least not nearly as relevant as geographic constraints affecting the movement of people and goods. However, in a social network analysis the context for decision making situations like transportation improvement, which include constraints and opportunities associated with the geographic features of the terrain itself (e.g., geographic locations, neighborhoods with certain social or political characteristics, ease of access to public transportation), are ignored. It is impossible to investigate durable, unavoidable, or persistent geographic factors behind emergent online social networks when the participants are removed from a real social and geographic context. Zook et al. [35] surveyed current literature on the role of geographic locations in the study of the Internet and makes a supportive case for the role of geography. More than a decade earlier, Wallace [36] had expressed the need to adapt social network analysis to real geographic space as ‘‘sociogeographic networks,’’ which he defines as spatially focused nets of social interaction. Despite these and other exceptions, we find it curious that it is difficult to find a lot of published research in the field of human–computer interaction that infers a sociogeographic network based on the physical locations of server and client computers. Our reading of the social network analysis literature, particularly social network analysis based on event logs, led us to consider how to factor real geographic locations into social networks that emerge during shared processes of decision making work. We considered the advantages

310

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

and disadvantages of plotting proximity based on frequency and type of interaction as in social network analysis, versus proximity based on geographic location, and concluded that we would still be able to use social network analysis techniques with our grapevine technique by sub-sampling interactions at various cross-sections of the grapevine, which suggest not only actual time but also step in a structured process. For instance, the top down view in Fig. 1 conveys the idea of a ‘‘network’’ and could mean a network of people interacting through a Web portal, a network of computers interacting as clients and servers, a network of spatial locations based on proximity and geographic context, and most importantly, all three simultaneously. Advanced development of the grapevine technique in a 3D GIS might involve a procedure or set of calculations to visually ‘‘reproject’’ or warp geographic relationships based on an evolving social network space, allowing the analyst to visually toggle back and forth between geographic and social space. 2.3. Space–time: time geography ¨ Hagerstrand [19] introduced the classical space–time (actually ‘‘time–space’’) concept of time geography during a 1969 presidential address to the European Congress of the Regional Science Association. He visualized the time– space concept by plotting an individual’s X and Y coordinate location and Z timestamp in a time–space volume in order to trace hypothetical movement across geographic space and up through time. On the basis of patterns in such 4D visualizations, he believed inferences could be made about constraints on mobility, goal-oriented behavior, avoidance-oriented behavior, or the social and cognitive influence of new information on mobility-related decisions. Visualization of the time–space concept was, ¨ according to Hagerstrand [19], a technique serving a larger ‘‘point of view’’ that focused on the disaggregate fate of individual human beings in complex systems. ¨ Hagerstrand’s time–space concept has experienced an empirically motivated rebirth over the last decade [20–23,37]. Unlike Pred’s [38] theoretical interest in time geography as a complement to structuration theory, recent interest in time–space geography is related to the widespread availability of Global Positioning System (GPS) devices. A recent compilation of research using ¨ Hagerstrand’s time geography can be found in an edited volume by Miller [22]. An important factor in the recent empirical turn in time geography has been the ability to collect travel paths with mobile GPS devices and then plot the data in three-dimensional space using GIS software. In this respect, our capability to use the LIT Web portal to capture hundreds of thousands of unobtrusive observations of client–server interactions is similar to the ability to collect unobtrusive observations of participant’s locations using GPS. An emerging investigation by geographers using time geography is how the use of mobile devices, the Internet, or geospatial information technology can alter the choices that an individual has to make about where they need to go next. Mobile devices and information technologies do not erase unavoidable constraints of geography and time

as much as they simply remove the individual’s need to confront them in the first place. For instance, with a mobile device like a cell phone a person is free to choose not to make a long trip and instead communicate remotely while traveling to another destination. Yu and Shaw [23] designed a set of adjusted time–space prisms to visualize new constraints that apply when an individual mixes physical and virtual activities together. Although the grapevine side view in Fig. 1 resembles ¨ Hagerstrand’s classical time–space visualizations in terms of extruded time data referenced to geographic space, time geography concepts were limited as a basis for analyzing the movement and exchange of information between computers. The measurable units of analysis in time geography studies are people moving in geographic space over continuous periods of time, not packets of information moving in cyberspace between people and computers at mostly fixed locations over long but intermittent periods of time. Contemporary time geography research has not been able to specify much about the behavioral, cognitive, or social influence of information events across space and through time beyond how it specifically influences travel behavior. In addition, with the exception of animal movement studies that use GPS data from multiple animals to visualize time–space patterns and infer social and behavioral concepts [39], time geography has not been used to investigate very large groups over long periods of time spanning months or years. 3. Creating a grapevine with event data The client–server interaction ‘‘event’’ represented our proxy for analytic or deliberative HCHI ‘‘activity’’ (see Table 1). In any investigation, there are units of analysis that simply carry data and then there are units of analysis pertinent to theory. In geospatial data modeling, an event is a data element standing for something that exists in geographic space but only for a limited amount of time [40–44]. Yuan and Hornsby [43] suggest that just as data ‘‘objects’’ represent static geographic ‘‘entities,’’ data events represent ephemeral geographic ‘‘occurrents’’ that happen in space for a certain period of time and then go away. An event represents a transaction or exchange of information via the Internet between a client browser application on a computer somewhere and the LIT server on the University of Washington campus. Thus the LIT Web portal event represents an interaction between computers, one that we use to infer the occurrence of HCHI between people in real geographic space and time. All observations of client–server interaction events are recorded by the LIT Web portal itself in the system event log. Unlike conventional server event logs, nearly everything that a participant requested a browser to do had to be executed directly by the server. We tested this using multiple user screen recordings of every step in the LIT Web portal. Thus, we felt comfortable that client–server interaction events were being reliably recorded by the LIT Web portal log. The main challenge was examining whether client–server events were a meaningful proxy for the analytic and deliberative HCHI activities of actual

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

Table 1 A representative example of how we made inferences about HCHI activity based on client–server event data logged by the LIT Web portal. When an individual ‘‘event’’ is logged by the LIT Web portal, it is tagged with additional information at multiple levels of granularity, helping us to infer specifically wherein the analytic–deliberative process the event occurred. Six levels of granularity in event data logged by the LIT Web Portal CCTAgent.setCommentVotingId = 1087375 1. ‘‘Event’’ The server logs a record that it successfully executed a key script and method for a specific client identified by a User ID, giving it a unique Event ID and including additional information about the target and any user-generated content as a result of executing the script and method. Based on the fact that the user generated vote content as a ‘‘1’’ and not a ‘‘0,’’ and because the log includes the target of the vote, we can infer that a specific human participant voted to ‘‘agree’’ with an another participant’s comment, identified as comment No. 1087375. c0

e1 = number:1086435

2. ‘‘Paired-Event’’ Now we also know that the target of the voting event was a comment event that had been logged earlier by the server as Event ID 1086435, from which we can find out the specific user ID of the participant who posted the original message, when they posted it, and where they posted it from (i.e., their self-reported home address or zip code). ioid = 1107097 3. ‘‘Technique’’ Because the event was logged with the information object ID (ioid) 1107097, we know the participant voted while doing a sequence of events associated with browsing all of the messages in Step 1 that the moderator felt fit into a ‘‘Governance and Funding’’ theme. activityId = 1078294 4. ‘‘Method’’ Activity ID is 1078294 also tells us that the participant voted while working within ‘‘Step 1c: Review summaries.’’ If browsing the ‘‘Governance and Funding’’ theme were possible in more than one LIT Step or sub-Step, we would be able to distinguish where the participant was working when they voted. contextId = 1078302 5. ‘‘Session’’ The Web portal supports multiple steps within an experiment. Context ID 1078302 tells us that the participant voted to agree with while working within ‘‘Step 1: Discuss concerns.’’ workflowId = 1078232 6. ‘‘Situation’’ The LIT Web portal can support multiple experiments. Workflow ID 1078232 tells us that the participant voted in the ‘‘Final LIT’’ experiment within LIT Web portal.

participants in real space and time, pertinent to the theory that participation in an analytic–deliberative process of interaction can improve decision making. The LIT Web portal logged 120,396 client–server interaction events during the LIT Challenge. Client applications of registered participants logged 120 different categories of events, which we call event types. Every record of a client–server interaction event is logged with a unique identifier as well as other attributes including the time, the registered user ID, the specific script and method (event type) called, the LIT Step and sub-Step in the Web portal where the script and method was requested, the unique ID of whatever content the client requested from the server, and the unique ID of any usergenerated content the client posted to the server (see Table 1). The frequency of the 120 participant event types

311

varied widely. For example, the event type indicating a participant deleted his or her own transportation concern occurred only once whereas the event type indicating a participant voted on a post or a reply occurred over 2000 times. Having distinguished 120 different participant event types, we then categorized event types by whether we felt they were the result of an analytic or deliberative HCHI activity or sequence of acts. To match our theoretical expectations, we relied primarily on the definitions of analysis, deliberation, and broadly based deliberation provided by the NRC [1,2]. In addition, we looked at findings in HCI, experimental psychology, and other fields of study to try to understand a cognitive basis of difference between analytic and deliberative HCHI activity in terms of what the human mind may be doing in perceiving, processing, and making sense of incoming auditory/ verbal or visual/pictorial information [45–47]. A sequence of HCHI acts for sending or exchanging a message usually in the form of words is a deliberative activity. For instance, an event type indicating a participant used the concern tool to write several sentences about the social inequities of using tolls to pay for an expensive transportation improvement project falls on the deliberative side of the spectrum. On the other hand, a sequence of HCHI acts like clicking a link to look at a map of a transportation improvement project is an analytic activity. We focused first on distinguishing the most important and clearly deliberative HCHI activities and identified five. One deliberative HCHI activity was passively viewing someone else’s message and presumably reading it, which was somewhat problematic to infer from client–server events and thus was not used to a great extent in our grapevine technique. The other four HCHI activities included actively sending a message using one of four different tools including (1) type your concern, (2) type your comment on someone else’s concern, (3) type your post, and finally, (4) type a reply to someone else’s post. Even though we elaborated criteria for distinguishing analytic and deliberative HCHI activities, we did not at this stage belabor the distinction by looking into the subtleties of each and every HCHI activity possible with the tools of the LIT Web portal. For instance, using the tool for adding a transportation-related term into the LIT glossary or using the tool for tagging one’s own concern with a set of keywords might be a little more on the deliberative side of the HCHI spectrum, because they involve communicating to others with text content. However, for simplicity, we decided that any event type other than the five most clearly deliberative types, with the exception of voting, was an ‘‘analytic’’ activity. Voting to agree or disagree with a message represented the only popular HCHI activity in the LIT Web portal that seemed both analytic and deliberative simultaneously. In the deliberative democracy literature, a vote is generally considered an analytic act [6]. The content of a vote is a binary or rank variable like one would expect as an analytic judgment of a phenomena. However, like a deliberative activity, voting on a message can reinforce a shared understanding about values and concerns, or

312

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

simply act as optional shorthand for the text message, ‘‘I agree with what you are saying.’’ Therefore, we decided to think of event types indicating voting on concerns, concern comments, posts, or replies in the LIT Challenge as deliberative activities that uniquely straddled both the analytic and deliberative side of the HCHI spectrum.

3.1. The grapevine metaphor Software designers and online moderators achieve an analytic–deliberative balance by getting participants to deliberate with each other about a specific set of analytical procedures, essentially making the participants latch their attention onto the details of the analysis itself and in the process attempting to discourage tangential conversations. By continually adjusting and encouraging certain kinds of activity, a moderator can move participatory interactions as a whole towards generating productive ‘‘clusters’’ of deliberation about the analysis used for decision making. The grapevine visualization uses the familiarity of a recognizable plant shape in nature to help the human eye evaluate large amounts of fine-grained data about the quality and scale of the analytic–deliberative process in public participation decision making. In nature, grapevines can often be found growing wild up the sides of fences or just about any free-standing structure. However, the grapes that these wild natural vines produce are not preferable for the purposes of human consumption. As opposed to a wild grapevine, cultivated grapevines in a vineyard are situated so that they naturally latch onto a specifically arranged set of support structures like a metal cable or wooden stakes. Then the grapevine plant itself is continually pruned and coerced to grow in a certain way that will generate productive grape clusters for harvesting. Thus the challenge for the viticulturist is to balance vegetative growth (energy spent to spread out new stems and leaves) with reproductive growth (energy spent to produce a certain abundance of grape clusters). The viticulturist achieves this balance by training the grapevine and constantly pruning its growth. The grapevine visualization is based on the metaphor of grapes in nature not only because of the familiar plantlike shape but also because the metaphor captures the ‘‘spirit’’ of participatory interaction [49]. Wild grapevines in nature are like the deliberations that can unfold in any given online community. They can grow off in every direction based upon whatever happens to be of interest to whomever the participants happen to be at the time. However, let us assume a decision has to be made about courses of action aimed at changing existing situations into preferred ones, in other words a ‘‘design’’ goal, but in this case a goal that affects the population of a particular area [48]. The challenge in convening a productive broadly based analytic–deliberative decision making process about design is to balance participant breadth of deliberation (energy spent broadening a diverse spectrum of topics, like the vegetative growth of a grapevine in nature) with participant’s depth of focused deliberative insight (energy spent addressing simplifying assumptions

or omissions in a particular analysis, like the reproductive growth of a grapevine in nature). 3.2. A supporting analytic structure In considering the grapevine visualization the first thing to understand is that in an analytic–deliberative process the deliberative activities are supposed to focus on analysis, which supports the nature of the discussion. In other words, deliberation is not just a deliberation about anything that comes to mind. The analytic support structure is a sequence of analytical activities. Core sequences of analytic activity like browsing and selecting GIS maps thus provide a structure so that deliberative activities like posting messages can have something specific to talk about. For the LIT online field experiment, the developers created a set of GIS-based transportation planning analysis steps for participants to browse and select. Participants create a package by selecting from a set of improvement projects and then adding a preferred funding mechanism. Participants were given a choice of 19 major categories of proposed road or transit improvement projects and 15 major categories of funding options. Participants were assisted in what choice to make with a ‘‘Tax Calculator’’ tool that opened in a new tab in a browser. For example, participants could browse the funding option category ‘‘Gas tax increase’’ and select from five options including ‘‘2 cents per gallon, 6 cents per gallon, 12 cents per gallon, 16 cents per gallon, and 20 cents per gallon.’’ Using the ‘‘Tax Calculator,’’ participants could enter personal travel and household financial information in order to estimate how much money they would be responsible for on a yearly basis given the choice of funding options. After doing this analysis in a step by step way using the tools available in the LIT online system, participants were then asked to deliberate about the transportation improvement programming analysis itself; in order to then agree upon one preferred set of projects and funding mechanisms. The dense structure in blue in Fig. 2 represents the participant’s analytic interactions, i.e., browsing and selecting information for the purpose of performing some

Fig. 2. A static representation of the grapevine displaying all analytic– deliberative activity (A), analytic activity only (B), and deliberative activity only (C). A visual analyst would use a rotatable 3D display and be able to zoom in and out of the grapevine.

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

operation. The analytic interactions with the LIT Web portal in Fig. 2 represent the support structure without which participants would have nothing to latch onto in terms of discussion. The organic looking grapevine itself represents deliberative message exchange and is described in more detail in Table 2 (see also Fig. 3). The green grapevine structure in Fig. 2 represents messaging activity and does not include analytic activities like

313

browsing and then selecting messages. The grapevine features in Figs. 1–5 were processed and displayed in a fully interactive 3D GIS environment, ESRI’s ArcGIS 3D Analyst or ArcScene. The complexity of any grapevine structure comes from different combinations and properties of five features (Table 2) including: (1) a main stem; (2) nodes and (3) buds that grow along its main stem; (4) tendrils that grow

Table 2 Seven main features of the grapevine, in terms of what it represents as a visualization of event-based HCHI activity and expected patterns in productive versus unproductive growth. See also Fig. 3. Feature

What it represents

Productive

Unproductive

A. Main stem

A running average of the locations of the last 10 participants who generated a message.

Stem twists back and forth because of rapid message turn-taking from participants at different locations.

B. Node and internode

A message added along the main stem from a particular location and point in time. Nodes can generate buds if there is a reply. A message that at least one other participant replied to with their own message. Buds generate shoots and leaves. A vote to agree or disagree with the message in a node, bud, or leaf. A tendril grows from a node, bud, or leaf to the specific time and location of the voting participant. A reply to a bud. A shoot grows from a bud and ends in a leaf at the time and location of the responding participant. A message sent as a reply. A leaf is generated from a bud and exists at the end of a shoot. A cluster of shared understanding, the proverbial fruits of an analytic– deliberative process. A synthesis of sense and meaning in message exchange, best harvested from productive areas of a grapevine.

Many large nodes with short internodes, because participants are rapidly posting messages and voting to agree or disagree. Many large buds are generated because many participants are replying to each other’s messages.

Main stem grows straight up with little twisting because of a lack of rapid message exchange or lack of geographic diversity. Few or mostly small nodes are generated, because participants are not posting messages or voting on each other’s messages. Few mostly small buds, or a greater proportion of nodes to buds, because participants are not replying to each other’s messages. Nodes with a few short tendrils branching out in only a few directions at a relatively high angle, indicating delayed and non-geographically diverse voting responses. Few or no shoots branching out in only a few directions at a high angle relative to the bud. Few or small leaves, because few participants voted to agree or disagree with a reply. Participants spend too much discussion energy posting their own messages about unrelated topics, rather than replying to others or discussing their shared understanding about something.

C. Bud

D. Tendril

E. Shoot

F. Leaf

G. Cluster

Nodes with many tendrils, both short and long branching out in all directions at a relatively low angle, indicating rapid and geographically diverse voting responses. Many shoots both short and long branching out in all directions at a relatively low angle to the bud. Many large leaves, because participants voted to agree or disagree with a reply. Participants balance their discussion energies between posting their own messages or new topics, with replying to each other’s messages and focusing on their shared understanding about something in particular.

Fig. 3. The features of a geovisual grapevine in comparison to the anatomy of the natural grapevine it was specifically designed to look like, including the main stem (A), nodes (B), buds (C), tendrils (D), shoots (E), leaves (F), and clusters (G) (see Table 2).

314

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

Fig. 4. Another static display of the grapevine showing the main stem when all other features are turned off (A), nodes turned on (B), nodes displayed proportional to the number of votes (C), and buds displayed proportional to the number of replies (D). Dashed line in blue represents end of LIT Step 1. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 5. More static displays of the grapevine showing the stem with nodes displayed proportional to the number of votes (A), node tendril features turned on (B), and nodes, leaves and leaf tendril features turned on (C).

out of nodes and latch onto the analytic support structure; and finally (5) shoots that grow from buds and end in a leaf (Fig. 3). By recognizing productive patterns in nodes, buds, leaves, main stem, and tendrils with visual cues, a visual analyst can find the most productive clusters of shared understanding to ‘‘harvest’’ for further analysis. One can get a feel for the grapevine by zooming in and rotating it in an interactive three-dimensional environment. Using the fully interactive environment, any visual analyst will be able to visually recognize and rank sections of the grapevine if they know what patterns to look for, that is, guided by visual cues as a mental picture of what productive versus unproductive activity looks like. In addition, in order to validate visual analyst rankings of each visual cue, described below, we developed a unique computer calculation for each (Table 3).

3.3. Nodes connected by a main stem The first major features of the grapevine are nodes on a main stem. A node represents when a user posts a message and the main stem of the grapevine grows from one node to the next (Fig. 4). Whenever a participant posts a new message it creates a new node. The node is a dynamic point event that we plotted in three-dimensional time–space in ESRI’s ArcScene using the selfreported location of the participant in latitude and longitude coordinates and the time (Pacific Standard Time) that the participant posted the message. However, the grapevine’s main stem is not just a plot of the location of new messages in time and space wherever they occurred. The main stem was something that we generated to represent the changing center of gravity where the

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

315

Table 3 Results of using the grapevine technique comparing visual cue ranks based on human spatial thinking skills (in bold) versus computer calculations (in italics, bold italics). Messages exchanged during the highlighted days, representing the top 12 most productive clusters, were selected for further content analysis. Date

Cluster

Cue 1

12-Nov 11-Nov 10-Nov 9-Nov 8-Nov 7-Nov 6-Nov 5-Nov 4-Nov 3-Nov 2-Nov 1-Nov 31-Oct 30-Oct 29-Oct 28-Oct 27-Oct 26-Oct 25-Oct 24-Oct 23-Oct 22-Oct 21-Oct 20-Oct 19-Oct 18-Oct 17-Oct 16-Oct 15-Oct 14-Oct

33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4

29 21 28 26 8 20 14 18 25 11 7 10 9 30 4 24 19 15 16 27 23 6 17 22 2 1 12 13 3 5

Cue 2 25 20 27 26 8 21 14 15 24 13 7 16 10 NA 5 28 18 12 17 29 23 6 19 22 2 1 9 11 4 3

29 21 27 28 12 17 15 14 24 10 11 23 22 30 13 25 18 16 20 26 9 4 8 19 2 1 7 6 3 5

Cue 3 27 23 30 26 11 19 15 17 22 12 13 25 20 29 7 28 16 14 21 24 8 5 9 18 2 1 10 4 3 6

29 27 28 24 14 9 18 15 19 7 4 25 17 30 10 22 26 16 21 23 12 8 13 11 1 3 20 5 2 6

Cue 4 NA NA NA 24 16 10 19 14 21 8 5 NA 20 NA 6 23 NA 11 18 22 13 9 17 15 2 1 12 4 3 7

most recent messages were coming from. In other words, the latitude and longitude locations of nodes along the main stem are a running average of the locations from which participants added a message. We purposely made the twisting and coiling of the main stem ‘moderately’ sensitive to message turn-taking behavior by calculating each new node based on an average of the latitudes and longitudes of the last 10 messages (Fig. 4). So, for instance, if participants from exactly the same location (e.g., a zip code centroid) generated 15 posts in a row, by the time of the tenth post the main stem would have drifted up and over in time–space until it was directly over that zip code location. If participants interact with rapid turn-taking from locations that are widely dispersed or on opposite sides of a population center, the main stem would twist and turn back and forth with a dense collection of nodes (Fig. 4). If participants were not interacting at all, the grapevine would appear unproductive and display a barren and straight main stem with only a few nodes separated in time by long internode gaps. Over a wide regional area, without controls on who is participating, one could imagine that the main stem would hover over the most populated areas. On the other hand, if there was some effort at recruiting participants from less populated areas and capping registration from the most populated areas, then when the main stem

30 28 29 22 16 10 18 15 23 3 4 24 17 27 6 26 25 14 19 21 11 7 13 12 1 2 20 9 5 8

Cue 5 29 28 27 26 15 12 7 6 22 10 9 25 20 30 5 21 24 4 17 23 18 13 19 14 2 1 16 11 3 8

27 19 25 26 9 17 16 18 23 4 8 21 20 28 6 24 17 14 18 22 11 5 12 15 1 2 10 13 3 7

Cue 6 26 19 27 28 6 21 13 14 22 16 9 23 20 NA 7 29 25 4 15 12 18 5 10 24 2 1 11 17 3 8

5b 5a 5a 5a 5a 5a 4b 4a 4a 4a 4a 3c 3c 3b 3c 3b 3b 3a 2b 2b 2b 1c 1c 1c 1b 1b 1b 1b 1b 1b

Mean 5a 5a 5a 5a 5a 4a 4a 4a 4a 4a 3c 3c 3c 3c 3c 3a 3a 3a 2b 2b 2b 1c 1c 1c 1c 1b 1b 1b 1b 1a/b

28.8 23.2 27.4 25.2 11.8 14.6 16.2 16.0 22.8 7.0 6.8 20.6 17.0 29.0 7.8 24.2 21.0 15.0 18.8 23.8 13.2 6.0 12.6 15.8 1.4 1.8 13.8 9.2 3.2 6.2

Diff. 26.8 22.5 27.8 26.0 11.2 16.6 13.6 13.2 22.2 11.8 8.6 22.3 18.0 29.5 6.0 25.8 20.8 9.0 17.6 22.0 16.0 7.6 14.8 18.6 2.0 1.0 11.6 9.4 3.2 6.4

2.1 0.7 0.4 0.8 0.6 2.0 2.6 2.8 0.6 4.8 1.8 1.7 1.0 0.5 1.8 1.6 0.3 6.0 1.2 1.8 2.8 1.6 2.2 2.8 0.6 0.8 2.2 0.2 0.0 0.2

began to shift from one area to the next it could indicate that the topics being discussed had some sort of ‘‘regional’’ basis. However, the geographic patterns to be expected are probably not as simple as regions. For instance, participants who live near major highways might have a generally different point of view than those who live far from major highways, thus changes in the growth of the main stem due to greater participation by people living near highways would probably require some computational awareness to help a human geovisual analyst recognize when such a pattern was occurring. Nonetheless, simpler geographic patterns such as a back and forth contest over time with respect to the center of gravity of a discussion between participants living near coastal areas versus those living in more inland areas would be relatively easy to pick out, especially if the participant recruitment strategy was specifically designed to look at those differences by balancing differences in population density. A productive grapevine will not only display an abundance of nodes, but also large nodes, all along the main stem, indicating lots of voting activity (Fig. 5). In our 3D GIS, we displayed the size of each node in proportion to how many votes a post or concern received (Fig. 5). In the LIT Web portal, voting on a post or concern meant voting to either agree or disagree. High levels of agreement or disagreement are displayed exactly the same way, as a

316

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

large node, unless we change the color of the node to indicate the ratio of positive to negative voting.

stem indicating that there was relatively sustained reply activity.

3.4. Nodes with tendrils

4. Our technique using the grapevine

Tendrils grow up and out from the site of a node or a leaf to the time and location at which a participant voted. Most nodes and leaves will produce at least one tendril. Participants could also vote on a reply or a concern comment and generate an additional set of tendrils off of leaves (Fig. 5). Any node that no other participant voted on or replied to is essentially a ‘dead’ node. In order to avoid cluttering the grapevine with lines going every which way, we displayed tendrils with a semi-transparent thin green line (Fig. 5). A productive cluster should have a proliferation of tendrils growing in an open pattern branching off at a low angle rather than a high angle to the node and extending out in all different directions (see also Fig. 3). Tendrils that branch off at a low angle indicate a relatively rapid voting response. Overall, a healthy mixture of long and short tendrils branching out at low angles in many different directions from a large node means participants from many different locations voted on a post or concern and did so relatively rapidly.

The first procedure in our method of using the grapevine technique was to filter the overview of deliberative activity down to the bare minimum necessary for each visual cue, by turning off all non-essential layers. Daily activity was a ‘‘natural break’’ in the distribution of nodes and represented the best unit for statistical comparison of events and closer inspection of content. Thus the unit of comparison was a small window in ArcScene centered on a single day of activity, beginning with Day 4 and ending with Day 33, totaling 30 windows. The second procedure was to rank each cluster with the 30 small view windows in ArcScene. The visual analysis task was to visually rearrange all 30 view windows like a puzzle from most productive to least productive according to each visual cue (Fig. 6). The order in which the view windows were arranged for each visual cue became the rankings in Table 3, ranked from 1 to 30. Fig. 6 shows the result of a visual analyst’s work with cue 1, rearranging daily deliberative activity from most productive to least productive looking at the number and size of nodes. In some ways, the grapevine might be considered a more primitive or even unreliable form of evaluation because it depends on human spatial thinking skills; on the other hand, for that very reason it could be considered superior. The spatial processing abilities of the human brain make the visualization work thus it is a synergy between the power of computing and the power of human spatial thinking skills. We expect any visual analyst evaluator with a bit of practice can learn to recognize and rank chosen sections of the rotatable three-dimensional grapevine visualization from least

3.5. Buds with leaves A productive pattern in participant discussion occurs when participants actually reply to each other’s messages. When this occurs we say that the node, the site of the original message, has developed into a bud giving rise to a reply displayed as a shoot with a leaf (Fig. 5). All buds develop from nodes but not all nodes generate buds. We displayed the size of each bud (i.e., post or concern) proportional to how many replies (i.e., reply or concern comment) it received. A productive grapevine will have an abundance of large buds distributed along the main

Fig. 6. A screenshot of ESRI 3D GIS environment illustrating the visual analyst ‘‘game’’ of rearranging views representing daily clusters of deliberation, from most productive at upper left to least productive at lower right then recording the rank order.

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

productive to most productive. The question is, how reliably can a human being interpret patterns in an admittedly complex-looking visualization? For every visual cue that a human analyst would use to rank a section of the grapevine, we developed a simple computer calculation to mimic human judgment. By comparing human rank order with rank order derived from computer calculations, we could validate human spatial thinking performance and determine if there were any significant differences using simple non-parametric statistical tests. A reasonable statistical test of agreement between multiple raters is to use a coefficient of concordance or community of judgment like Kendall’s W-statistic [50]. However, to compare human versus computer-based sets of ratings we decided to use a Marginal Homogeneity (MH) test. The MH test can distinguish if there is a significant difference between two samples. We calculated computer ranks for each visual cue and compared human visual rankings versus computer calculated rankings. The MH test results indicated that the human visual analyst’s rankings and the calculated rankings were not significantly different, and in fact almost identical. Testing visual analyst performance ranking clusters of deliberative activity with the grapevine under different conditions remains an area for further research. After verifying human judgment of productive clusters, we selected content from within the highest ranking clusters.

317

4.2. Results from geovisual analysis We felt that most participant references to transportation-related features, objects, concepts, and occurrences in the central Puget Sound region would be expressed as nouns, either as the subject or object of the sentence or of a prepositional phrase [52,53]. Thus distilling clusters of deliberation into elements of meaning meant filtering messages down to the most frequently occurring noun baseforms used as subjects or objects of meaning. We used the Connexor tags to select nouns and discovered that during the most highly productive discussions participants mentioned 3728 unique nouns representing 1155 noun baseforms. For example, the words ‘‘bike’’ and ‘‘bicycle’’ are just different forms of the same baseform ‘‘bicycle.’’ We considered a number of methodological issues in our content analysis, such as whether pulling subject and object nouns out of the sentence really captured the sense and meaning of participant shared understanding, as well as whether we should have included gerunds since they could also be the subject or object of a sentence. Nonetheless, we decided that a nounbased comparison of deliberative message exchanges was an appropriate start. After distilling the discussion into noun baseforms, we re-examined the original context in which noun baseforms were mentioned and re-read message exchanges to get a sense for what participants meant.

4.3. Results from content analysis 4.1. Recognizing productive clusters of deliberation One of the first things we noticed using the grapevine technique was that nodes were not evenly distributed along the main stem (Fig. 4). There was a higher abundance of nodes associated with activity in LIT Step 1, whereas there was a lot of bare stem after LIT Step 1 indicating declining deliberative activity except for one distinctive surge of activity. Activity increased during LIT sub-Steps 3c and 4a when participants were deciding which projects were best for the central Puget Sound region and which funding options should be used to pay for them. Among quota or paid human subjects (n = 179) the number of people actively participating declined from 60 percent to 40 percent, which gave the late surge that occurred mainly during Steps 3c and 4a (days 23 and 24 of the experiment) a unique importance. Six of the top dozen clusters of deliberation were associated with LIT Step 1 and six were associated with later steps, totaling 209 separate exchanges plus thousands of votes. Relying primarily on the computer calculations of visual cues, we sub-selected 45 above average deliberative exchanges and processed the text content using a demo version of a software tool called Connexor [51]. The Connexor software parsed the 45 messages into 17,145 individual elements of content. Each individual content element was tagged with detailed information including a unique ID, the cluster and day the element was generated by a user, the numerical order of the element in the sentence, its word baseform, and its syntactic relation, syntax, and morphology.

The content analysis indicates a major shift in the qualitative nature of shared understanding after LIT Step 1 when participants did a geospatial analysis. The 99th percentile of noun baseforms in terms of frequency of occurrence (frequency of 30 or more) in the first six clusters before the end of LIT Step 1 included the following ten words (Fig. 7): bicycle (91), bus (73), transit (62), pedestrian (40), car (39), bicyclist (34), transportation (33). people (31), road (31), and traffic (31). The 99th percentile of noun baseforms (frequency of 16 or more) in the last six clusters after the end of LIT Step 1 included the following six words (Fig. 8): toll (36), project (32), package (23), people (20), improvement (16), and tax (16). After re-reading the original messages with the 99th percentile noun baseforms in mind, it became clear that the shift in the frequency of noun baseforms did indicate a shift in the sense and meaning of the discussion. When asked to deliberate about values and concerns for improving transportation during LIT Step 1, participants spent a lot of energy discussing alternate modes of transportation like bikes, buses, or rail in a broad discussion about ways of reducing car traffic thus reducing the need for expensive roads and transit projects. However, when asked to review planning factors, create a regional transportation improvement package using a geospatial analysis, and then deliberate about the results of the analysis to select the best package; participants focused on calling out social and economic inequalities inherent in selecting different funding options, like taxes or tolls, as ways of fairly redistributing the cost of transportation

318

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

Fig. 7. Noun baseforms mentioned by participants during message exchanges in LIT Step 1, in terms of frequency of occurrence. Only noun baseforms above the 90th percentile (4 7 gray line) are shown. Noun baseforms above the 99th percentile ( 430 white line) are highlighted in white.

Fig. 8. Noun baseforms mentioned by participants during message exchanges after LIT Step 1, in terms of frequency of occurrence. Noun baseforms above the 90th percentile ( 45 gray line) are shown. Noun baseforms above the 99th percentile ( 415 orange line) are highlighted in orange. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

improvement among those living and traveling in the central Puget Sound region. The noun baseform ‘‘package’’ was used only once during the most productive deliberative exchanges of LIT Step 1 (Fig. 7). However, ‘‘package’’ was the 3rd most frequently mentioned noun baseform (23 times) during the most productive exchanges after the end of LIT Step 1

(Fig. 8). The dramatic increase in the noun ‘‘package’’ is an indicator that the shift in shared understanding we observed after the end of LIT Step 1 was the result of participant experience with geospatial tools in the LIT Web portal, and not a result of participant experience with things outside of the LIT Web portal. As corroborating evidence, many of the most productive deliberative

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

exchanges contributing to the topical shift in shared understanding occurred simultaneously with an enormous spike in analytic activities of using GIS maps to create a transportation package during LIT Step 3. The content analysis of the most productive deliberative exchanges, identified by grapevine visual analysis, provides evidence that participant use of geospatial analysis tools in later steps of the LIT Web portal improved the quality of deliberation even though it may have ironically decreased the scale of deliberation and participant interest, as compared with other parts of the LIT Web portal like LIT Step 1. We believe that the moderated analytic–deliberative process in later steps of the LIT Challenge successfully ‘‘trained’’ participant deliberative energies to latch onto the analytic support structure provided by the transportation package analysis, particularly Step 3 of the LIT Web portal. By engaging in analytic HCHI activity with the LIT Web portal, participants refocused the growth of deliberation on simplifying assumptions and omissions in the funding options component of the analysis, rather than continuing to contribute to the growth of a broad and unattached discussion about personal values and concerns that had little or nothing to do with the assumptions of the simulated public agency transportation analysis. If we had investigated activity data alone, without combining a content analysis to help us distill sense and meaning, we might have concluded that LIT Step 1 was clearly the most successful part of the entire process and should be emulated in future design and development, since our human subjects spent a great deal of energy deliberating about a broad and largely activist spectrum of concerns. Participant activity as measured by number of participants and time spent declined after Step 1 and even faltered at various points during Steps 2, 3, 4 and 5. However from a certain quality of interaction perspective, even though the earlier LIT Steps were more wildly productive in terms of message posting and exchanging, the geospatial exercises in the LIT Challenge along with persistent moderation trained participant energies to deliberate about the analysis itself despite lower overall productivity as measured by messaging. In many ways, the most important deliberation of the entire LIT Challenge occurred in the midst of declining overall participation near the end of the experiment during LIT Step 4. Natural participant deliberative energies should be nurtured but the growth of discussion should also be pruned and trained so that it does not grow off in every direction and ultimately collapse. Instead, the growth of deliberation should be trained to latch onto and make use of analytic support activities, in order to focus deliberative energy to produce clusters of deliberation about an analysis. By supporting the analytic–deliberative process with geospatial analysis tools, the LIT Web portal seemed to refocus broadly based participant energies on deliberating about the analysis, resulting in a more useful outcome from the standpoint of agency decision makers and technical specialists who use similar analyses in transportation planning. After using a grapevine-based content analysis to identify shifts in shared understanding during a broadly

319

based analytic–deliberative process, we feel we are in a better position to provide public agencies with empirically based evaluations of whether, in the use of analysis to make actual decisions, these agencies may have seriously miscalculated something about the breadth and depth of concern among stakeholders. To most agency planning specialists, the transportation package analysis in the LIT Web portal would probably appear fairly straightforward in terms of its funding options, as it was modeled after the process used by the Puget Sound Regional Council, the metropolitan planning organization of the central Puget Sound region. Evaluation of the analytic–deliberative process in the LIT Web portal provided us with empirical evidence of the breadth and depth of a particular stakeholder concern in the central Puget Sound region about unknown, unintended, or unanticipated social inequities being generated by transportation planning agencies as a result of selection of certain funding options. Further investigation into relationships between words and modifying words using natural language processing, concept map visualizations, or other methods integrating content and discourse analysis would reveal much more about the sense and meaning of qualitative shifts in shared understanding as a result of participant experience with the LIT Web portal. 5. Conclusion Development and exploration of HCHI using the grapevine technique contributes to a growing literature in visual and geovisual analytics about insightful techniques to examine collaborative decision processes with geospatial technology. Thomas and Cook [54] and Andrienko et al. [55] have made calls for more research about the use of visualizations of information to support analysis and deliberation about complex problems. Considering how other geovisualization techniques have been used for exploratory analysis of spatio-temporal data [56–58], we created a custom geovisual analytic technique that balanced the computing power of a GIS to display large amounts of fine-grained event data, with the human process of spatial thinking or visual reasoning to identify the emergent patterns. Creating a grapevine requires an understanding of GIS and some basic effort in data processing. However, the effort required is no more than any other type of GIS analysis in our estimation. The grapevine required no statistical assumptions to use. One of the things we wanted to do with our geovisual analytic technique, quickly and reliably, was to examine and recognize the most productive daily clusters of HCHI activity as areas of focus for content analysis. Human spatial thinking performance in matching and ranking observed patterns based on expected patterns was not statistically different than computer-based ranks. Further testing will help identify where spatial thinking skills are most challenged when it comes to recognizing productive patterns in the grapevine visualization. The grapevine technique also contributes to the literature about the broadly based analytic–deliberative

320

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

decision making process. An important hypothesis behind analytic–deliberative decision making is that decisions are better when they come from a combination of analysis and deliberation rather than from analysis alone. The findings suggest that indeed when deliberation is structured by a specific analysis the two ways of knowing can enhance each other. However, we also found that convening an analytic–deliberative process in order to engage a broadly based lay public with technical information (or with technical specialists and decision-making executives), even when done in a convenient online tool with human moderators and a structured process of public participation, tends to produce neither a continuous nor stable quality of deliberation. The analytic–deliberative process unfolds in fits and starts of productive deliberation mixed within and among relatively unproductive interactions. Though the process may start fast it can undergo measurable decline in terms of participation and interest. For those with experience trying to implement the NRC’s recommendations and convene a broadly based analytic–deliberative process this may appear as an alltoo-familiar and discouraging pattern. Our results with the grapevine suggest that uneven growth and quality of deliberation in an analytic– deliberative process may not be indicative of a systematic failure to engage participants. An analytic–deliberative process that participants generally considered confusing or boring might still have contained sporadic and stimulating episodes of broadly based deliberation useful for improving analysis in decision making about public health, public safety, and the environment. The grapevine method may, therefore, be particularly useful if broadly based analytic–deliberative decision making processes tend to be sporadically productive in ways that only a fine-grained formative type of assessment can distinguish. As further steps, comparative research projects that study the quality and scale of public participation in analytic–deliberative decision making under varying conditions are needed to shed light on largely untested expectations expressed in the participatory democracy literature [3,6,59]. Results that show the effect of different combinations of public participation tools and recruitment strategies in different decision making conditions would be particularly valuable. Specifying the theoretical differences between ‘‘analytic’’ and ‘‘deliberative’’ event types along an analytic–deliberative HCHI spectrum, for the purpose of coding a client–server event log, is also needed. Further investigations might confirm that though broadly based analytic–deliberative processes appear to the untrained eye as unproductive and less than engaging, evaluation of fine-grained interaction data can locate the irregular episodes of exceptional public deliberation about analysis that will improve decision making.

Acknowledgments This research was partially supported by National Science Foundation Information Technology Research

Program Grant no. EIA 0325916, National Science Foundation Geography and Spatial Sciences Grant no. BCS0921688, and National Oceanic and Atmospheric Administration Grant no. NA07OAR4310410. Support of the National Science Foundation and the National Oceanic and Atmospheric Administration is gratefully acknowledged. We would also like to acknowledge our two anonymous reviewers and members of various research teams including Michalis Avraam, Piotr Jankowski, Michael Patrick, Kevin Ramsey, Zhong Wang, Matt Wilson and Guirong Zhou; and especially Tanveer Randhawa and Kanwar Buttar for work in development of automated grapevine applications. The authors are solely responsible for the content. References [1] National Research Council, Understanding Risk: Informing Decisions in a Democratic Society, National Academy Press, Washington, DC, 1996. [2] National Research Council, Decision Making for the Environment: Social and Behavioral Science Research Priorities, National Academy Press, Washington, DC, 2005. [3] National Research Council, Public Participation in Environmental Assessment and Decision Making, National Academy Press, Washington, DC, 2008. [4] T. Nyerges, Scaling-up as a grand challenge for public participation GIS, Directions Magazine (2005) /www.directionsmag.comS. [5] T. Nyerges, P. Jankowski, D. Tuthill, K. Ramsey, Participatory GIS support for collaborative water resource decision making: results of a field experiment, Annals of the Association of American Geographers 96 (4) (2006) 699–725. [6] J. Gastil, P. Levine (Eds.), The Deliberative Democracy Handbook, Jossey-Bass, San Francisco, CA, 2005. [7] Digital Future Report, 2010, /http://www.digitalcenter.orgS. [8] Let’s Improve Transportation, 2007, /www.letsimprovetransporta tion.orgS. [9] D.T. Cook, D.T. Campbell, Quasi-experimental Design: Design and Analysis Issues for Field Settings, Rand McNally, Skokie, IL, 1979. [10] D. Brinberg, J. McGrath, Validity and the Research Process, Sage, Thousand Oaks, 1985. [11] J.A. Konstan, Y. Chen, Online field experiments: lessons from CommunityLab, in: Proceedings of the Third Annual Conference on e-Social Science Conference, Ann Arbor, MI, 2007. [12] National Research Council, Learning to Think Spatially, National Academy Press, Washington, DC, 2006. [13] A. von Eye, Configural Frequency Analysis—A Program for 32 Bit Windows Operating Systems Program Manual, Michigan State University, East Lansing, MI, 2008. [14] P. Sanderson, C. Fisher, Exploratory sequential data analysis: foundations, Human–Computer Interaction 9 (1994) 251–317. [15] W. van der Aalst, B. van Dongen, J. Herbst, L. Maruster, G. Schimm, A. Weijters, Workflow mining: a survey of issues and approaches, Data and Knowledge Engineering 47 (2) (2003) 237–267. [16] W. van der Aalst, H.A. Reijers, M. Song, Discovering social networks from event logs, Computer Supported Cooperative Work 14 (2005) 549–593. [17] N. Andrienko, G. Andrienko, P. Gatalsky, Exploratory spatio-temporal visualization: an analytical review, Journal of Visual Languages and Computing 14 (6) (2003) 503–541. [18] K.P. Hewagamage, M. Hirakawa, An interactive visual language for spatiotemporal patterns, Journal of Visual Languages & Computing 12 (3) (2001) 325–349. ¨ [19] T. Hagerstrand, What about people in regional science?, Papers of the Regional Science Association 24 (1970) 6–21. [20] M.P. Kwan, Time, information technologies and the geographies of everyday life, Urban Geography 23 (2002) 471–482. [21] M.P. Kwan, Feminist visualization: re-envisioning GIS as a method in feminist geographic research, Annals of the Association of American Geographers 92 (2002) 645–661. [22] H.J. Miller (Ed.), Springer Science, Dordrecht, The Netherlands, 2007. ¨ [23] H. Yu, S.-L. Shaw, Revisiting Hagerstrand’s time-geographic framework for individual activities in the age of instant access,

R. Aguirre, T. Nyerges / Journal of Visual Languages and Computing 22 (2011) 305–321

[24] [25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34] [35]

[36]

[37]

[38]

[39]

[40]

in: H.J. Miller (Ed.), Societies and Cities in the Age of Instant Access, Springer Science, Dordrecht, The Netherlands2007, pp. 103–118. A. Getis, A method for the study of sequences in geography, Papers of the Regional Science Association (1966) 87–92. M.S. Magnusson, Discovering hidden time patterns in behavior: T-patterns and their detection, Behavior Research Methods, Instruments, & Computers 32 (1) (2000) 93–110. G. Olson, M.J.D. Herbsleb, H.H. Rueter, Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques, Human–Computer Interaction 9 (3/4) (1994) 427–472. T. Nyerges, T.J. Moore, R. Montejano, M. Compton, Interaction coding systems for studying the use of groupware, Human– Computer Interaction 13 (2) (1998) 127–165. W.D. Gray, D.A. Boehm-Davis, Milliseconds matter: an introduction to microstrategies and to their use in describing and predicting interactive behavior, Journal of Experimental Psychology: Applied 6 (4) (2000) 322–335. P. Sanderson, Release notes for MacSHAPA 1.0.3, Crew System Ergonomics Information Analysis Center (CSERIAC), Wright-Patterson Air Force Base, OH, 1995. P. Jankowski, T. Nyerges, GIS-supported collaborative decision making: results of an experiment, Annals of the Association of American Geographers 91 (1) (2001) 48–70. A. MacEachren, Moving geovisualization toward support for group work, in: J. Dykes, J.A. MacEachren, M.J. Kraak (Eds.), Exploring Geovisualization, Elsevier2005, pp. 445–461. D. Haug, A.M. MacEachren, F. Hardisty, The challenge of analyzing geovisualization tool use: taking a visual approach, Pennsylvania State University, University Park, PA, 2001. S. Tanimoto, S. Hubbard, W. Winn, Automatic textual feedback for guided inquiry learning, in: Proceedings of the International Artificial Intelligence in Education (AIED) Society Conference, 2005. P. Keel, EWall: a visual analytics environment for collaborative sense-making, Information Visualization 6 (2007) 48–63. M. Zook, M. Dodge, Y. Aoyama, New digital geographies: information, communication, and place, in: S. Brunn, S.L. Cutter, J.W. Harrington Jr. (Eds.), Geography and Technology, Kluwer Academic Publishers, The Netherlands, 2004. R. Wallace, A fractal model of HIV transmission on complex sociogeographic networks: towards analysis of large data sets, Environment and Planning A 25 (1993) 137–148. N.R. Hedley, A. Lee, C.H. Drew, E. Arfin, Hagerstrand revisited: interactive space–time visualizations of complex spatial data, Informatica 23 (2) (1999) 155–168. A. Pred, Structuration, biography formation, and knowledge: observations on port growth during the late mercantile period, Environment and Planning D: Society and Space 2 (3) (1984) 251–275. J.M. Gudmundsson, M. van Kreveld, B. Speckmann, Efficient detection of motion patterns in spatio-temporal data sets, Technical report UU-CS-2005-044, Institute of Information and Computing Sciences, Utrecht University, 2005. M. Worboys, Event-oriented approaches to geographic phenomena, International Journal of Geographical Information Science 19 (1) (2005) 1–28.

321

[41] M. Worboys, K. Hornsby, From objects to events: GEM, the geospatial event model, in: Proceedings, Geographic Information Science 2004, Springer Verlag, Adelphi, MD, 2004, pp. 327–344. [42] D.J. Peuquet, N. Duan, An event-based spatiotemporal data model (ESTDM) for temporal analysis of geographical data, International Journal of Geographical Information Systems 9 (1) (1995) 7–24. [43] M. Yuan, K.S. Hornsby, Computation and Visualization for Understanding Dynamics in Geographic Domains: a Research Agenda, CRC Press, Boca Raton, FL, 2007. [44] K.S. Hornsby, M. Yuan, Understanding dynamics of geographic domains, CRC Press, Boca Raton, FL, 2008. [45] S.K. Card, J.D. Mackinlay, B. Shneiderman, Information visualization: using vision to think, Morgan-Kaufmann, San Francisco, CA, 1998. [46] R.E. Mayer (Ed.), Cambridge University Press, New York, 2005. [47] D. Billman, G. Convertino, J. Shrager, J.P. Massar, P. Pirolli, Collaborative intelligence analysis with CACHE and its effects on information gathering and cognitive bias, Palo Alto Research Center, Paper presented at the Human Computer Interaction Consortium Workshop, Snow Mountain, CO, 2006. [48] C. Steinitz, On scale and complexity and the needs for spatial analysis, Paper Presented at Spatial Concepts in GIS and Design, 2008, /http://ncgia.ucsb.edu/projects/scdgS. [49] N.S. Contractor, D.S. Siebold, Theoretical frameworks for the study of structuring processes in group decision support systems: adaptive structuration theory and self-organizing systems theory, Human Communication Research 19 (4) (1993) 528–563. [50] M.G. Kendall, B.B. Smith, The problem of m rankings, The Annals of Mathematical Statistics 10 (3) (1939) 275–287. [51] Connexor, 2008, /www.connexor.euS. [52] D.M. Mark, Spatial representation: a cognitive view, in: D.J. Maguire, M.F. Goodchild, D.W. Rhind, P. Longley (Eds.), Geographical Information Systems: Principles and Applications, 2nd Edition, Wiley, New York, 1999. [53] D.M. Mark, A. Skupin, B. Smith, Features, objects, and other things: ontological distinctions in the geographic domain, Lecture Notes in Computer Science 2205 (2001) 488–502. [54] J.J. Thomas, K.A. Cook, Illuminating the Path: the Research and Development Agenda for Visual Analytics, National Visualization and Analytics Center, Richland, WA, 2005. [55] G. Andrienko, N. Andrienko, P. Jankowski, D. Keim, M.-J. Kraak, A. MacEachren, S. Wrobel, Geovisual analytics for spatial decision support: setting the research agenda, International Journal of Geographical Information Science 21 (8) (2007) 839–857. [56] A. MacEachren, M. Gahegan, W. Pike, Visualization for constructing and sharing geo-scientific concepts, PNAS Early Edition, 2003. [57] C.D. Hundhausen, Using end-user visualization environments to mediate conversations: a ‘Communicative Dimensions’ framework, Journal of Visual Languages & Computing 16 (3) (2005) 153–185. [58] P. Compieta, S. Di Martino, M. Bertolotto, F. Ferrucci, T. Kechadi, Exploratory spatio-temporal data mining and visualization, Journal of Visual Languages and Computing 18 (3) (2007) 255–279. [59] R. Kingston, Public participation GIS and the internet, in: T. Nyerges, H. Couclelis, R. McMaster (Eds.), Handbook of GIS and Society Research, Sage Publications, London, forthcoming.