Authoring diagrams that adapt to their viewing context

Author’s Accepted Manuscript Authoring Diagrams That Adapt to Their Viewing Context Cameron McCormack, Kim Marriott, Bernd Meyer www.elsevier.com/loc...

Download PDF

792KB Sizes 0 Downloads 16 Views

Report

PDF Reader
Full Text

Author’s Accepted Manuscript Authoring Diagrams That Adapt to Their Viewing Context Cameron McCormack, Kim Marriott, Bernd Meyer

www.elsevier.com/locate/jvlc

PII: DOI: Reference:

S1045-926X(16)30202-6 http://dx.doi.org/10.1016/j.jvlc.2016.11.001 YJVLC765

To appear in: Journal of Visual Language and Computing Received date: 11 June 2015 Revised date: 28 June 2016 Accepted date: 1 November 2016 Cite this article as: Cameron McCormack, Kim Marriott and Bernd Meyer, Authoring Diagrams That Adapt to Their Viewing Context, Journal of Visual Language and Computing, http://dx.doi.org/10.1016/j.jvlc.2016.11.001 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting galley proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Authoring Diagrams That Adapt to Their Viewing Context1 Cameron McCormack, Kim Marriott2 , Bernd Meyer Monash University, Faculty of IT Melbourne, Victoria 3800, Australia

Abstract The Web and digital media require documents whose appearance and content adapt to the viewing context and to user interaction. While most previous research has focussed on adaptation for textual and multimedia content, this is also true for diagrammatic content. We (a) identify the reasons for adaptation and the diﬀerent kinds of adaptation that make sense for diagrams; (b) present an aspect-oriented model for diagram adaptation that separates adaptation into diﬀerent orthogonal components; (c) describe a diagram authoring tool based on this model; and (d) present the results of a user evaluation of the tool. Our model uses layout “conﬁgurations” to model signiﬁcantly diﬀerent layout alternatives and geometric constraints to perform minor layout adjustment. The author can also specify alternate representations for an object, alternate styles and alternate textual content. The resulting space of diﬀerent versions of the diagram is the cross product of these diﬀerent alternatives. At display time the version is selected from this cross product and constructed automatically, taking into account the author speciﬁed preference order on the alternatives, current viewing environment, and user interaction. Keywords: diagrams, adaptive layout, authoring, constraint-based graphics 1. Introduction There is widespread recognition that the web and digital media requires intelligent, adaptive documents whose appearance and content adapts to the viewing context and which support user interaction [2, 3]. Previous research has focussed on textual and multimedia content and largely ignored adaptation of diagrammatic content, treating this as a kind of image. This is unfortunate, since charts, maps, plans, networks and other diagrammatic notations are an important, commonly used vehicle for communication and can also beneﬁt from adaptive presentation. Figure 1 shows the kinds of adaptive presentation we wish to support. The preferred presentation is on the left and shows a cutaway of the Earth with labelled arrows pointing to the diﬀerent layers. At the bottom is a hint for interactive environments informing the user that they can move the cursor over the labels to reveal more detailed information. This is shown at the bottom of the view, replacing the hint. The two other presentations show how the diagram can be adapted to viewing environments with less display space. The presentation in the middle shows a rotated version. This orientation takes a little less space than the main view, due to the labels being laid out from left to right. The third view (on the right) is suitable for 1A

preliminary version of this paper appeared in [1] author: [email protected]

2 Corresponding

Preprint submitted to Elsevier

smaller displays such as mobile phones. In this view labels are not shown by default. Instead, small hot spot circles are positioned on the image and when the cursor hovers over them the corresponding label is shown nearby. As well as adapting to the display space, the diagram is designed to be presented in German as well as English and the styling can adapt in response to medium capabilities and accessibility concerns: a greyscale version can be shown instead of colour and the font size used for the text labels can be varied. Scalable Vector Graphics (SVG) [4], the web graphics format, provides some support for adaptive presentation of diagrams and there have been a number of proposals to extend SVG with more sophisticated layout capabilities. However, the sort of adaptive behaviour we wish to support requires complex scripting as well as advanced SVG. It is not reasonable to expect web developers to have to directly write SVG and script to encode such adaptive behaviour. Authoring tools are required that hide this encoding away from all but the most expert web developer. However, to the best of our knowledge there has been no research into the design of such tools. The above example illustrates a key problem when authoring adaptive diagrams: the combinatorial explosion of diﬀerent versions of the diagram when diﬀerent kinds of adaptation are considered. Explicitly authoring independent versions of a diagram for every combination of adaptation will obviously result in a large number of diagram versions and large amount of work for the author, June 3, 2016

Figure 1: Three diﬀerent versions of an adaptive diagram showing the structure of the earth.

especially if the diagram is subsequently changed. Our example diagram, for example, has 12 diﬀerent versions, just considering the three diﬀerent views, two presentation languages and two colour modes. Producing 12 independent versions of the diagram might be in the realm of feasibility for an author, but it is deﬁnitely not be acceptable if the author, after producing these 12 versions, has to return to make changes that aﬀect all of them. Here we present the ﬁrst tool explicitly designed for authoring diagrams that can adapt their presentation to diﬀerent viewing contexts. The main diﬃculties in designing such a tool were: (1) not overwhelming the author with the combinatorial explosion of diﬀerent versions of the diagram; and (2), ensuring that it was relatively easy to use and understand but still powerful enough to to specify the most useful kinds of adaptive behaviour for a wide variety of diﬀerent kinds of diagrams. Our speciﬁc technical contributions are the following. First, we determined the kinds of adaptive behaviour that the tool should support (Section 3). To do so we examined around 150 diagrams from a wide variety of application areas and detailed how each type of diagram could be sensibly adapted to diﬀerent viewing environments. We identiﬁed seven main kinds of adaptation–change of layout, style, form, text, and focus and the introduction/removal of indirection and dynamism. Figures 1 and 2 illustrate some of these diﬀerent kinds of adaptation. Second, we designed a novel model for adaptation of diagrams that supports these diﬀerent adaptive behaviours (Section 4.1). Our model uses layout “conﬁgurations” to model signiﬁcantly diﬀerent layout alternatives and geometric constraints to perform minor layout adjustment. This is similar to models previously used for authoring adaptive textual documents [5, 2] and adaptive multimedia documents [6]. The main innovation in the model is that we separate the speciﬁcation into diﬀerent aspects which we use in the sense of aspect-oriented programming [7]. This is based on the observation that object representa-

tion, style changes and text changes are largely orthogonal to layout changes. Thus, the model allows the author to specify alternate representations for an object, alternate layout conﬁgurations, alternate styles and alternate text. The resulting space of diﬀerent versions is the cross product of all these diﬀerent alternatives. At display time the version is constructed dynamically by solving the associated constraints, taking into account the author speciﬁed preference order on the alternatives, the current viewing environment, and previous user interaction. Our third contribution was the design and implementation of an authoring tool that embodies this model (Section 4.2). While the authoring tool is still a prototype, we believe it demonstrates that it is possible to build a diagram authoring tool that is reasonably easy to use yet allows speciﬁcation of powerful adaptive behaviour. A key question in the design was what kind of constraintsolving technology to use. After some experimentation we ended up using multi-way constraint solving as this allowed multi-directional constraint solving when authoring and the resulting diagrams and adaptive behaviour could be readily compiled into SVG with scripting. Our ﬁnal contribution was to conduct a in-depth open ended qualitative user study (Section 5). This was designed to evaluate the adaptive model and authoring tool and also the completeness of our categorisation of diagram adaptation kinds. While the study was limited to 8 IT students, the results provided empirical validation that our categorisation is complete and that someone with basic experience in the use of diagramming or illustration tools and in web design could understand our model and successfully use the tool to construct adaptive diagrams. Almost all participants said that they would prefer to use a (polished) version of the adaptive diagram authoring tool over more conventional diagramming tools for constructing adaptive diagrams.

2

2. Related Work

media queries to write rules that tailor the presentation to diﬀerent device capabilities, display sizes and orientations. Constraints, often in combination with templates, are a reasonably common approach to declaratively specify how layout of textual documents should adapt [13, 5, 14, 15, 16]. Constraints encode minor layout adjustment while templates more major layout adjustment. For example, in [16] the layout is speciﬁed using a set of grid-based templates. One-way constraints compute the width and height of template elements from the width and height of the page or the intrinsic size of an image. Templates have a special output variable representing a score for the layout and the choice of template is based on this and the template preconditions. A prototype authoring tool for templates was also developed, which allowed the author to place text boxes and ﬁgures with their edges attached to horizontal and vertical guides. To handle interactive authoring a multi-way propagation constraint solver was used. Another approach to textual document adaptation has been to take a document that was authored for a particular device, infer the layout and structure of the document and to adjust the layout automatically to ﬁt the document to the display size on a diﬀerent kind of device. For instance, [17] presented a tool that can infer the layout structure of a web page to adapt it to a small size viewing device such as a phone. Multimedia and hypermedia: Multimedia systems support presentations of two-dimensional arrangements of text, images and synchronised audio and video, and often support user interaction. Hypermedia systems are hypertext systems that have been extended with multimedia capabilities. Metadoc [18] is an adaptive hypertext reading system that allows adaptation of textual content using “stretchtext.” The Adaptive Hypermedia Architecture (AHA) [19] supports stretchtext, conditional inclusion of entire HTML fragments, and adaptive linking. In both Metadoc and AHA, the textual content is adapted to take into account the readers characteristics, interests and knowledge level. This reﬂects hypertext and hypermedias traditional use in educational scenarios. Brusilovsky, in his summary of the ﬁeld, goes into more detail about the kinds of adaptations available in hypermedia systems [20]. Work on the adaptation of multimedia presentations has focussed on layout adaptation and temporal adaptations which change the timing of objects within the document. There has been considerable research in this area including [21, 22, 6]. Most relevant is Madeus [6]. This is an authoring environment for interactive multimedia documents that uses constraints to model the spatial and temporal adaptive behaviour of a multimedia presentation. Initially one-way constraints were used then these were extended with linear constraints. Constraints are speciﬁed interactively by the author using high-level predeﬁned relations such as “the left edges of these two objects are

Document adaptation and models for authoring adaptive documents have been well-studied for text and hypertext, multimedia and hypermedia, and UI widgets. Much of this has utilised constraint-solving. Constraints are typically used to declaratively encode desired geometric relationships that the adapted layout should satisfy. Constraints can either be hard in which case they must be satisﬁed or soft in which case the layout should satisfy them if it is possible. Geometric constraint-solving: A wide variety of different constraint-solving techniques have been developed. One-way (or data-ﬂow) constraints are the simplest, most widely used kind of constraint [8]. A one-way constraint is of form x = f (y1 , ..., yn ) where the formula f details how to compute the value of variable x from the variables y1 , ..., yn . Whenever the value of any of the yi variables changes, the value of x is recomputed, ensuring that the constraint remains satisﬁed. One-way constraints are versatile, simple to implement and can be solved extremely eﬃciently. The downside is that they are directed and cannot be used to compute a value for, say y1 , given values for x and the other yi s. Multi-way propagation based constraint solving methods [9, 10] provide multi-directional constraint solving. All variables can potentially be output variables, so long as their value can be calculated from the values of the other variables. A multi-way constraint is speciﬁed by a set of one-way constraints which detail how the constraint can be solved for diﬀerent choices of output variable(s). Linear arithmetic constraint solving techniques [11] are becoming more common in graphical applications. Like multi-way propagation methods, linear constraint solving is multi-directional. Solving is, in essence, based on using variable elimination to dynamically produce a system of one-way constraints. Both multi-way and linear arithmetic solvers can handle hard and soft constraints. They diﬀer in the kind of constraints they can support. Both one-way and multiway constraints require a function to compute the value of the output variable: this means that they cannot readily handle linear inequalities or other kinds of relational constraints. However linear inequalities are handled by linear arithmetic constraint solvers. On the other hand linear arithmetic constraints cannot handle more complex functional constraints like making a text box tall enough to contain a paragraph of text for a given height. Text and hypertext: Virtually all document formatting software adapts presentation by automatically choosing line and page breaks [12]. Hypertext systems provide signiﬁcantly more layout adaptation because of the need to support variable size browser windows. For instance, Cascading Style Sheets (CSS) Level 3 provides automatic line and page breaking, ﬂoat placement, and column width resizing in tables. It also allows the document author to use 3

adaptive presentation of diagrams. It supports zooming and uniform rescaling of diagrams, media-speciﬁc styling of graphical content through CSS style sheets and allows alternate versions of document elements for diﬀerent media and languages. There have been a number of proposals to extend SVG with more sophisticated layout capabilities such as by adding linear constraints [36], one-way constraints [36, 37] or by using XSLT [38]. Somewhat related is research into techniques for ensuring layout stability in interactive information visualisation applications as the user explores information graphics such as network diagrams or tree maps [39, 40]. There has been no research into how to author adaptive diagrams. However, there has been considerable research into authoring of standard (non-adaptive) diagrams. Many owe some debt to Sutherland’s seminal interactive CAD tool Sketchpad [41] which allowed the user to specify geometric diagrams with a light-pen and constraints. As a consequence there is a long history of research into the use of constraints in CAD and generic diagramming tools, e.g. [42, 43, 44]. Geometric constraints are also provided in the graphic editors Microsoft Visio3 and ConceptDraw4 . In previous diagram authoring tools, however, constraints are used to preserve geometric relationships during editing and to support parametric objects. They are not intended to support adaptation of a diagram to diﬀerent viewing contexts. Thus, we see that, while there has been research into authoring adaptive text and hypertext, adaptive multimedia and hypermedia, and adaptive UIs, this is the ﬁrst research to address authoring of adaptive diagrams.

aligned.” Temporal relations are modelled using Allen’s interval algebra [23]. Madeus provides the author with multiple simultaneous views of the document. There are four types of views: presentation, which is the main view showing the entire document; object, which shows a single re-usable composite object; textual, which shows the document’s source text; and scenario or execution, which shows the document as presented in a particular adapted context [24]. The author can interactively modify any of the views of the document and have the changes automatically reﬂected in the other views. Laborie [25] describes a multimedia document adaptation technique that uses spatio-temporal proximities to determine the best modiﬁcations to make to a document. A similar approach is taken in [26, 27] where adaptations are found by choosing a path through a three dimensional space of adjacent spatial, temporal and interactive (STI) adaptations. UI Widgets: Most user interface widget toolkits provide the ability to deﬁne a layout that will reposition and resize widgets to account for diﬀerent window or screen sizes. Thus we can consider widget toolkits as having adaptive behaviour. An early example was the InterViews toolkit [28]. It used TEXs box/glue model and allowed widgets to be connected by springs in horizontal or vertical boxes. This layout model is now a standard feature of most popular widget toolkits, including Gtk+, Qt, Java AWT and Swing, Microsofts WPF and Mozillas XUL. High level layouts are also typically available in these toolkits. These support common layout tasks at a higher-level of abstraction than the corresponding box and glue based layout. For instance, BoxLayout, supports ﬂowing and resizing of objects arranged in a row or a column. Many research widget toolkits use layout constraints to control widget placement. One-way constraints underpin Garnet [29], Penguims [30] and subArctic [31]. An interactive authoring tool for Penguims called OPUS has also been developed [32]. Layout is speciﬁed graphically by the use of reference lines and one-way constraints. Multi-way and linear constraints have also proved popular. Amulet for instance extends Garnet to support both one-way and multi-way constraints [33]. The Auckland Layout Manager uses linear arithmetic constraints, among other types of constraints and layout features [34]. Recent versions of Apple’s Cocoa, the widget toolkit used in OS X and iOS, provide a constraint-based layout system that uses linear arithmetic constraint solving to support hard and soft linear equations and inequalities. Adaptation of user interfaces from desktop to mobile devices is considered explicitly in Dygimes [35]. It uses a markup language that allows the interface layout to be described at a high level by, for instance, stating that one widget should be to the right of another widget. Diagrams and graphics: The web graphics format Scalable Vector Graphics (SVG) [4] provides some support for

3. Kinds of Diagram Adaptation A necessary ﬁrst step in designing a tool for authoring adaptive diagrams is to determine what kinds of adaptation it should support. To do so we need to understand the possible reasons for adaptation and how they aﬀect the diagram’s presentation. To that end, we reviewed the reasons given for adaptation for other kinds of documents. Brusilovsky [20], gives two main reasons for adaptation in hypermedia systems: user characteristics, which encompasses the users goals, knowledge, background and interests, among others; and environment, including presentation platform (hardware and software), network bandwidth, and user location and orientation. Lewis [45] identiﬁed a number of “challenges” that authors face when designing web content for device independence. These fall into two categories: device and access mechanisms; and personalisation and accessibility. Finally, a review of automatic document layout [12] identiﬁes a number of reasons for automatic document layout: variable page size in 3 http://oﬃce.microsoft.com/visio 4 http://www.conceptdraw.com/

4

Indirection

Text

Dynanism

Form

Focus

Style

Layout Display size Medium Accessibility Internationalisation User characteristics Variable content

agrams) and covered relevant categories (maps, pictures, statistical charts, time charts, link diagrams and grouping diagrams) from von Engelhardt’s classiﬁcation of graphic representations [47]. While the sources we inspected had far more than 150 diagrams, we added to our collection only those diagrams that used a kind of representation that had not already been collected. For each diagram in the collection we identiﬁed possible reasons for adapting its presentation and checked that they were covered by one of the six reasons listed above, then identiﬁed ways in which the diagram could be usefully adapted while preserving its semantics. We grouped these into seven distinct kinds of adaptation which can be usefully applied to real-world diagrams. Layout: Layout involves repositioning and resizing diagram elements. The obvious application is to adapt the layout to the available viewing space by, for instance, collapsing whitespace, scaling the diagram, or changing its orientation. Changes in text may also lead to layout changes, and adaptation to a language with a diﬀerent reading order may lead to signiﬁcant layout changes. Style: Style captures non-geometric changes to the diagram elements, such as colour or choice of font. Style change is required to cater for user accessibility and preferences and to device capabilities. For instance, it is required when adapting a diagram utilising colour to one that is suited to monochrome printing or presentation to colour-blind users. Style change is also important to ensure that the choice of colours and symbols takes account of cultural associations. It can also be used to adapt to the style of the surrounding context. Focus: Focussing is used to draw attention to more important parts of a diagram. It takes two forms: zooming and importance reduction. Zooming displays only a part of the overall diagram such as the immediate neighbours of a node in node-link diagram or a detailed view of part of a map. Zooming is often interactive. On the other hand, importance reduction still shows the whole diagram but reduces the visibility of peripheral elements. Examples include scaling by using a ﬁsh-eye lens, or by restyling or repositioning elements to have less visual impact. Focussing also includes removal of redundant elements such as some of the labels on a chart axis or contour lines on a map. Similarly, diagrams that have repeated items in them often use some syntax, such as an ellipsis, to show that some of the items have been omitted (where the number of items omitted might depend on the amount of space available). Form: A more extreme kind of adaptation is to change the way in which the information is represented by the diagram. Changing form could change the spatial relationships used to encode a given relation; for example, changing the representation of a hierarchy from a layered tree to one based on containment. Charts are another example of a diagram type that could easily have its form changed – stacked bar charts that add up to 100%

Table 1: Summary of the main kinds of adaptation in diagrams and the primary reasons for using them

which to lay out the document; user requirements such as a preference for larger fonts; and dynamic content due to Variable Data Printing, VDP, where print material is customised to the individual reader. Based on these we identiﬁed six main reasons for modifying or adapting the presentation of a diagram: Display space: The dimension and resolution of presentation spaces varies, requiring the layout to be modiﬁed to make best use of the available space. Medium capabilities: A diagram should adapt to the presentation medium and viewing device capabilities, for instance whether it supports interaction or colour. Accessibility: The diagram may need to be modiﬁed in order to make its content more accessible to particular users. For instance, a tactile version might be produced for a blind user. Internationalisation: Adaptation to diﬀerent languages and cultural conventions such as the default reading order is another example of responding to user requirements. User characteristics: The diagram could be modiﬁed to take into account user history, interests, preferences or level of knowledge. Variable content: Some adaptation may be required if (parts of) the diagram are dynamically generated and so may be unknown at design time. Two less important reasons for adaption that we identiﬁed were: modifying the appearance to match the style of contextual material or to take account of additional information; and the need to cater for low bandwidth. In order to understand how diagrams might usefully adapt to diﬀerent viewing contexts we collected 150 representative diagrams from a wide range of application areas and from a variety of sources: academic journals, newspapers, text books and the web. The examples were chosen to cover common types of diagrams. The corpus included diagrams from all relevant categories (graphs, structure diagrams, process diagrams, time charts, maps, cartograms and network charts) identiﬁed in Lohse et al.’s [46] classiﬁcation of visual representations (the remaining three: tables, icons and pictures we did not consider to be di5

can be changed into pie charts, or vice versa. A more extreme form is to change a diagram to a completely textual description of the information therein. Changes in form can be used to reduce the spatial extent of the diagram and to improve accessibility to blind users. Indirection: Indirection refers to conventions that provide information about graphic elements in another part of the diagram and use some sort of correspondence to relate that information back to the appropriate graphic elements. Introducing indirection can reduce the size of a diagram. One example is a legend; rather than having labels next to all elements of a diagram, the elements can be coloured and these colours associated with the labels in the legend. Another example of indirection is moving text labels into a key, away from the elements that they label, and replacing them with numbers that are an index into the key. Introduction of indirection is useful, for instance, when preparing a tactile version of a diagram since Braille labels are usually much larger than the non-Braille labels and so may not ﬁt into the diagram [48]. Text: Clearly internationalisation may require the textual content of a diagram to be adapted. Alternate, equivalent text that is shorter than the original text can be used to conserve space. Dynamism: Animation and interaction can be introduced or removed from a diagram depending upon the capabilities of the presentation device. If a diagram is conveying change, for example, this can be shown as an animation on a laptop but in the printed paper version of the diagram as a sequence of snapshots. Interaction in an adaptive diagram has two broad roles. One is to control adaptation of the diagram, for example, the focus or level of detail, the font size or the size of the viewport. The other role is exploration of the diagram, in order to better understand what it represents. This includes, for example highlighting the corresponding entry in the legend when moving the mouse over a data point in a chart, or changing the inputs to a diagram that shows a process or computation. Table 1 summarises the main uses of these diﬀerent kinds of adaptation with respect to the reasons identiﬁed above. It is based on our analysis of the diagrams in our collection. Consider our motivating example from Figure 1. The three presentation views of the diagram illustrate both layout and form adaptation. In all three views, continuous layout adaptation responds to the amount of available space by positioning and sizing the elements of the diagram relative to other elements and to the page boundaries. More major layout adaptation occurs when switching between the main and rotated views, where the layout of the yellow labels move to the top of the diagram. The change in label appearance in the compact view is an example of form adaptation. The example also supports text adaptation to the desired language and styling adaptation to medium capabilities and accessibility. Finally, because

Figure 2: Example of diagram adaptation. The original diagram (a) shows the stages involved in cell division; (b) is an interactive adaptation to a small display in which the user steps through the stages one-at-a-time; (c) is a “landscape” version of the diagram; and (d) is a tactile graphic designed for a blind reader.

the choice of content can be controlled interactively in all three views, the diagram supports dynamism. As another example consider Figure 2(a). This comes from our collection. It depicts four steps in cell division with arrows indicating progression between the steps. We considered how to adapt the diagram’s presentation to different display spaces. Figure 2(c) shows a “landscape” presentation that is one way to adapt to a short but wide display space. This version preserves many of the original aesthetics. In the original arrows are equally long and the space taken up by all four steps are equal. In the adapted version, these properties are kept, but because the arrows now run horizontally, in the same direction as the text, the arrows must be long enough to accommodate the widest label. To ensure that the labels describing the parts of the cell in the ﬁrst step do not unnecessarily cause that step 6

tool.5 3. In particular, the speciﬁc types of adaptation the model and tool support should be understandable and predictable; we want the author to be able to control the adaptations and understand how to achieve particular adaptive behaviour. 4. Finally, the resulting adaptive diagrams should be able to be compiled into a reasonably compact representation supported by current standards such as SVG with scripting.

to take up more space, the layout of the labels is slightly diﬀerent from those in the original diagram. Figure 2(b), demonstrates adapting to even less available rendering space such is as on a mobile phone with a low resolution display. The smaller rendering space is insuﬃcient to show all four steps at once. Instead we can use interaction to allow a reader to navigate through the steps of the cell division process, showing one step at a time. Since navigation between the steps requires some form of on-screen control, each view has hyperlinks at the bottom of the diagram to move to the next and previous step, as appropriate. The steps could also be shown with animation. Figure 2(d) shows how to adapt the diagram for a blind user. One obvious transformation is that colours must be removed, and replaced by a texture or a raised area. In our cell division example the only information encoded with colour is the identiﬁcation of the plasma membrane and cell wall. Since these parts are labelled in the ﬁrst step and remain in the same relative position in subsequent steps, the colour can be removed. Note though that since the lines that represent the plasma membrane and cell wall have no separation, if these two parts are to be mapped on to the one colour they will need some gap so that the reader can distinguish them. A second transformation is to replace the text by Braille. It is not suﬃcient to simply print the text with a Braille font since Braille can use standard abbreviations. Another point is that small complex shapes are not as easily discerned in a tactile diagram, thus we have replaced the chromosome icon with a short textual identiﬁer (chro.) that is also speciﬁed in the Bacterial chromosome label. It is worth noting that when adapting the presentation of a diagram we required that the adaptation is informationally equivalent. However adaption can aﬀect how easy or hard it is for the reader to extract this information from the diagram. In particular changing form or adding or removing dynamism can have a signiﬁcant perceptual or cognitive impact.

4.1. Aspect-oriented Adaptation Model Our motivating example (Figure 1) illustrated a key problem when authoring adaptive diagrams: the combinatorial explosion of diﬀerent versions of the diagram when diﬀerent kinds of adaptation are considered. Our model for adaptive diagrams addresses this issue by separating the speciﬁcation into diﬀerent aspects which we use in the sense of aspect-oriented programming [7]. We separate object presentation, style, text, interaction and choice of layout view into diﬀerent largely independent aspects. Doing so dramatically eases the burden on the author by allowing these to be speciﬁed orthogonally. Diﬀerent versions of a diagram are deﬁned implicitly to be the cross product of these possible adaptations. This allows the (possibly quite large) number of diﬀerent versions of the diagram to be constructed automatically. If these aspects are not fully independent, the model allows the author to ﬁne tune adaptation by specifying compatible combinations. In more detail the model has seven major components: Graphic objects: the actual graphical elements which appear in the diagram; Layout constraints: which control the position and size of the graphic objects within a particular conﬁguration; Conﬁgurations: collections of graphic objects and layout constraints representing major presentation views of the diagram; Master objects: re-usable compound object deﬁnitions; Styling alternatives: styling palette and its use for style adaptation; Text alternatives: a mechanism for providing alternative text strings; and Actions: pre-deﬁned behaviours that can aﬀect the presentation of the diagram at display time in response to user interaction. Diﬀerent layout conﬁgurations cater for major changes in the layout such as orientation, form, use of indirection or

4. Adaptation Model and Authoring Tool While the kinds of adaptive behaviour identiﬁed above can be obtained by hand-coding scripted SVG, it is not reasonable to expect web developers to have to do this. In this section we describe a model for representing diagram with adaptive behaviour and an authoring tool that supports this model. The design requirements were that: 1. The model and tool should support the kinds of adaptation identiﬁed above; we want the author to be able to construct a wide variety of diagrams that can perform layout, style, focus, form, indirection, text and dynamism adaptations. 2. The tool should be usable; we want the author to be eﬀective, eﬃcient and satisﬁed in their use of the

5 ISO 9241:1998: Ergonomic requirements for oﬃce work with visual display terminals (VDTs) Part 11: Guidance on usability. International Organization for Standardization, Geneva, Switzerland.

7

animation. In our running example there are three conﬁgurations, corresponding to the three views shown in Figure 1. Layout constraints specify how to automatically adjust the layout of the graphic objects in the conﬁguration to cater for diﬀerent text, style, choice of diagram component representation, viewport dimensions and history of user interaction. The author speciﬁes how this layout adjustment should occur by placing persistent geometric constraints between the graphic elements that will be maintained during adaptation. As the choice of textual content is usually orthogonal to the layout of a diagram, text alternatives are speciﬁed separately from the conﬁgurations in a text master. For each piece of text in a conﬁguration the author can supply diﬀerent versions, one for each possible display language. Furthermore, the author can optionally supply for a particular display language alternate versions of the text, typically the full text and an abbreviated version. The system will choose, based on the available space for layout, the best alternate version of text to use for the current display language. This choice is global rather than made individually for each piece of text. Thus, if the abbreviated version is chosen, all text will be abbreviated. Similarly the author can specify styling alternatives which are independent of conﬁgurations or text. At presentation time, a particular style mode is applicable, e.g. colour or monochrome. It is chosen by the user agent based on the capabilities of the medium and accessibility requirements of the viewer. Viewer interaction can be used to control diagram adaptation. The author speciﬁes this by using global, numerical state variables whose values are changed by user interaction and whose values aﬀect the choice of conﬁguration, style class, and object representation. Any graphic object in the diagram can be annotated with a list of actions to perform when an event occurs on that object (such as moving the mouse cursor over it, or clicking it). These actions modify the state variables and in so doing, cause the diagram to adapt. To facilitate re-use, the author can specify an master object and use instances of it in diﬀerent conﬁgurations. This can be parametric in the layout, choice of text and style etc. A master object can specify alternate representations, for example, a compact and expanded representation. Each instance of a master object with alternate representations has an alternate selection policy, which is simply an expression over the state variables that, when evaluated, gives the index number of the alternate representation to use. When the diagram is to be displayed the system must choose which combination of conﬁguration, object alternatives, text and style is used to create the version of the diagram to be displayed. We distinguish between environment driven choices for which the viewing environment speciﬁes a unique choice, such as the display language; interaction driven choices for which the choice is also unique but which is the result of user interaction,

such as whether an object is expanded or collapsed; and search driven choices, such as which conﬁguration to use and which level of text abbreviation, for which the system must try diﬀerent possibilities in order to determine the best choice. Each conﬁguration has a set of text content and styles for which it is compatible. By default a conﬁguration is compatible with all text content and styles but the author can restrict the choice. This means that the author can create conﬁgurations that are language- or style modespeciﬁc. The layout constraints implicitly add compatibility restrictions on the conﬁguration as for instance they may not be satisﬁable if the viewport is too narrow. The author can add additional tests to the conﬁguration to check that the layout is reasonable. Tests are passive: the layout is computed and then the test is evaluated. One of the most diﬃcult decisions in the adaptation model was determining how to allow the author to specify the preference order on versions. One approach would have been to require the author to give an explicit objective function measuring “goodness” and for the system to choose the layout maximizing this function [16]. Although very general, we felt this would be diﬃcult for the author. Another approach is for the author to provide a preference ordering on the diﬀerent possible versions. We chose to take this approach. The diagram version is chosen dynamically at display time. The system examines those versions that are compatible with the environment and interaction driven choices. There is a total ordering on these versions from most preferred to least preferred. The primary ordering is given by the preference ordering on the conﬁgurations, with a secondary ordering given by the ordering on the alternate choices of text for the display language. The versions are created and examined in descending order of preference. The ﬁrst version created that is valid, i.e. for which the layout constraints and layout tests are satisﬁed, is chosen. One issue is what to do in the case that user interaction changes the object representation. This may mean that a more preferred version that was previously invalid becomes valid because of this new choice of representation. One approach would be to change versions. We felt that this might be disturbing to the user because of layout instability, and also has a high overhead since it means that the choice of diagram version has to be reconsidered after virtually all user interaction. Instead we have chosen to keep the current version until it becomes invalid. However, changes to the viewport dimensions or text size will lead to a total re-computation of the preferred version. 4.2. Prototype Authoring Tool We have built an authoring tool that supports the above aspect-oriented adaption model for diagrams. It is written in a combination of Java and ECMAScript and consists of approximately 57,000 lines of code. The authoring tool is a prototype and has a number of limitations. 8

Figure 3: Labelled screenshot showing the main UI features of the adaptive diagram authoring tool.

First, while the tool provides all seven major kinds of adaptation previously identiﬁed it currently does not support all examples of these kinds that were mentioned. Notably continuous animation is not supported, although other kinds of dynamism such as user controlled discrete animation and interaction are. Second, the tool supports only four of the six adaptation reasons we identiﬁed in Section 3: display space, medium capabilities (target device colour and interaction support), accessibility (desired font size) and internationalisation (preferred language). We have not added support for variable content, which would require dynamic generation of diagrams from database output, or adapting to user characteristics, which would require building a system that could learn from a viewers past behaviour and interests. Third, since there was a great diversity in kinds of layouts identiﬁed as part of the diagram survey, we decided to limit ourselves to diagrams whose layout is grid-like in the sense that objects in the diagram are primarily placed with respect to horizontal and vertical grid lines [49] This kind of layout is a commonly provided feature of commercial diagramming tools; for example, Adobe Illustrator, Microsoft Visio, OmniGraﬄe, Dia and ConceptDraw. Such layout encompasses a wide variety of diagrams but does not include, for example, radial layout of trees or organic layout of networks. Note that none of these limitations are due to any fundamental restrictions imposed by the design of authoring tool or its underlying model and with further implementation work they could be readily overcome. In order to make the tool easy to learn to use, we de-

cided that it should, as far as possible, extend the construction model commonly used in diagramming tools such as those listed above. An example editing session is shown in Figure 3. Here, three of the master objects and the three conﬁgurations required for the running example are shown. The author modiﬁes objects in the diagram by directly manipulating them in the editing canvas, which occupies most of the left hand side of the screen. The editing canvas shows a single conﬁguration or alternative from a master object a time. Conﬁgurations are the core of the authoring tool and behave similarly to the drawing canvas in a standard graphic editor. However, unlike in a traditional editor where multiple canvases represent diﬀerent diagrams, multiple conﬁgurations are alternate layouts for the same logical diagram. The right hand side of the window is taken up by the preview canvas, which allows the author to see how the adaptive diagram will behave and be presented in a particular context. Previews are analogous to scenarios in Madeus [24]. They can be created for particular languages, colour modes, default font sizes and page sizes, and these properties can all be modiﬁed in real time. As changes are made to a conﬁguration or master object, these will be reﬂected in the preview. The preview canvas is also used for testing the interactive behaviour of a diagram. Actions that have been assigned to objects can be invoked by performing the corresponding interaction in the preview canvas with the mouse. The tool currently supports three kinds of visual objects or graphic elements: Shapes (rectangles, ellipses, diamonds and parallelograms) which are deﬁned by top, left, 9

bottom and right edges of a rectangular area and which can have word wrapped text placed within them; aspectratio preserving raster or vector graphic images; and multisegment arrows. Layout constraints are speciﬁed using three kinds of layout objects which are visible in the conﬁguration but invisible when the diagram is displayed. They are: Guidelines: horizontal and vertical lines to which other objects can be aligned Rows and columns: horizontal and vertical containers for child objects to be placed at the same horizontal or vertical position and for their sizes in that same dimension to be made equal Spring distributions: horizontal and vertical containers for child objects to be positioned and sized along the page from left to right or top to bottom and connected by springs which have a desired, minimum and maximum length. Objects can be placed into a layout relationship in one of three ways: by attaching one object to another, creating a parent/child relationship or by anchoring objects. Most diagram objects have one or more attachment positions, which are x or y positions on the object that can be aligned with attachment positions on other objects. For example, shapes expose six attachment positions: four for the edges of the rectangular area, and midx and midy, which lie at the middle of the object. Arrow objects have an attachment point at each vertex which may be attached to any point on another object not only its attachment points. A vertical guideline exposes just a single attachment position: its x position. Figure 4 shows the layout objects required to specify part of the leftmost conﬁguration for the running example. The three labels are placed in a column (the pink rectangle), the image and column are in a horizontal spring distribution, the image is in vertical spring distribution and the arrows are attached to the labels and to the corresponding position on the image (crosses show. attachment points). Our use of layout objects to set up persistent placement constraints is not unusual. For instance, Madeus [24] provides guidelines and equal width/height constraints though distributions and other spring-based positioning are not supported in Madeus. Rows, columns and spring distributions provide similar layout behaviour as spring-based grid layouts that GUI design tools have, such as the one built in to the Netbeans Java IDE. A common adaptation is wanting to produce a portrait or vertical layout version of a diagram from a landscape or horizontal one, or vice versa. The reorientation tool allows the author to duplicate an existing conﬁguration and automatically reorient it 90 degrees. The tool converts rows into columns, changes text shape sizing modes if they do preferred width/height computations, changes attachments to the edges of shapes to their corresponding edge when rotated, as so on. This conversion is not quite a

Figure 4: Close-up of the layout constraints for part of the leftmost conﬁguration of Figure 1.

rotation, but more of a mirroring of the position and sizes of the diagram objects in the conﬁguration. The second conﬁguration of Earth structure diagram can be produced by reorienting the ﬁrst conﬁguration. Master objects behave similarly to Visio’s master shapes. As objects are added, modiﬁed and deleted from the master object these changes will propagate to all instances. Instances of master objects are parametric in their styling, positioning, text content and interactive behaviour, and any of these may be overridden on the corresponding objects of the instance. For simplicity master objects are not allowed to be nested. In the Earth structure example, each of the three labels is an instance of a master object. The master object deﬁnition has two objects, the yellow rectangle and an arrow object, with the start point of the arrow attached to the (x, midy) attachment point of the rectangle. Each instance of the master object overrides the text content of the rectangle with its own text. The three instances also have their arrow objects attached to diﬀerent points on the Earth image. The tool allows the alternatives for the master objects. Like the conﬁgurations of a diagram, each alternative representation of a master object may be annotated with particular colour modes or languages it is suitable for, and at display time the ﬁrst alternative that is suitable will be chosen for display. Alternatives may also be chosen as the result of user interaction. Unlike re-usable objects in other diagramming tools, the interface between an instance of a master object and the surrounding objects in the diagram conﬁguration is not simply a rectangular region. Instead, all objects that exist in the master objects may be attached to and placed within layout objects in the conﬁguration, and any layout objects from the master object may be used to control the layout of objects from the conﬁguration. An initial design required “exporting” speciﬁc guidelines and other layout objects in the master object by giving them matching names across all alternatives, in order to have the master 10

object provide a clear interface for how it may be hooked up to the surrounding objects in the conﬁguration. We found however that this was too restrictive, and instead simply allow each alternative of a master object to have its component objects attached and placed in parent/child relationships with objects from the surrounding conﬁguration separately. For simplicity the tool does not allow master objects to be nested. The authoring tool has a number of features that facilitate working with master objects: Instances of master objects are shown in conﬁgurations with a highlight colour around them and labelled with the name of the master object, so that they are easily identiﬁable. Overriding properties of an instances component objects is done in place by making the relevant changes within the conﬁguration. The currently shown alternative of a master object instance in a conﬁguration can be changed by the author. This allows the author to quickly see, in place, how the diﬀerent alternatives look and behave in context. It also allows the author to specify how the master object is placed relative to other objects in the conﬁguration, by directly creating layout relationships (such as guideline attachments) between the objects from the instance and other objects from the conﬁguration. Objects in a conﬁguration may be “promoted” to master objects without having to delete and recreate them as master objects explicitly. This means the author does not need to plan ahead and allows a master object to be designed in place rather than in a separate canvas without the context of the surrounding conﬁguration objects it will be connected to. Graphic objects can be styled with solid colour and hatch pattern paints for their ﬁll, stroke and text, and the font family, size, weight and style of the text within a shape can also be controlled. Rather than independently specifying the ﬁll and stroke paint of each object, these styles are set by referencing entries in a palette which is global to the diagram. Editing a palette entry aﬀects all objects that reference the entry. By default, the paint used in a palette entry will be transformed to a greyscale and monochrome version automatically, using a simple ﬁlter (average of the three colour components for the mapping to greyscale, and average brightness determine the mapping to black or white for monochrome). This can be overridden by the author to give a more speciﬁc styling adaptation to handle these colour modes. Fonts are not controlled by the palette and are speciﬁed independently on objects. Font size however can be speciﬁed as being relative to a default font size that is controllable at display time. We considered allowing nonpaint values such as stroke widths and font sizes to be speciﬁed in palette entries so that the author could control them in one place and specialise them for the diﬀerent

colour modes, but we felt that the primary use of styling specialisation would be for paints and thus preferred the simpler model. We also considered allowing more complex constraints on font style and size but decided that users might ﬁnd these diﬃcult to understand. The tool allows the author to associate a list of text strings with a graphic element. At display time, the text alternatives for the readers preferred language are selected. If there is more than one choice for that language the text string from the set of alternatives is chosen so that the constraints in the currently displayed conﬁguration are satisﬁed. The choice of text string in the set of alternatives is done globally across the conﬁguration. The intention is for the author to specify shorter text strings for the later alternatives, as they will have a greater chance of causing the shapes to have sizes that satisfy the diagrams constraints. The authoring tool supports interactive diagrams by allowing the author to assign actions to events that occur on graphic objects within the diagram. Instead of exposing a general purpose scripting environment, a limited set of actions are provided through an interaction editor which is revealed when an graphic object is selected. Our prototype provides two actions. The ﬁrst is to reveal an object on hover. This causes a target object to be hidden by default and to be made visible only when the pointer is over the object that the action is assigned to (the source object). Hiding and showing of the object does not aﬀect layout in any way; it only aﬀects whether the source object is rendered or not. The second action is to select an alternative. This action will change which alternative a master object instance is currently showing. 4.3. Constraint solving and compilation to SVG Minor layout adjustment, in which the layout changes smoothly in response to changes in text or minor changes in the viewport dimensions, is handled in our model and tool by geometric constraint solving. As we have seen geometric constraint solving has been widely used to provide adaptive layout for a wide variety of content and this is its primary role in our tool. The use of geometric constraintsolving is not unusual in diagram editors but previously this has been to facilitate subsequent editing rather than to specify how the layout should adapt. While important, this is a secondary role of constraint solving in our authoring tool. We ﬁrst considered using one-way constraints in the tool since they support standard graphical constraints and they can be readily compiled into script. However, they have a signiﬁcant limitation: constraint solving is directional and cyclic dependencies are not allowed. If one-way constraints are only used to specify how the layout should adapt, then the direction of constraint-solving is ﬁxed by the application with viewport size and text box sizes driving the rest of the layout. However, this directionality may not be that expected by the author when editing the diagram. In addition user-studies have shown that users 11

expect alignment and distribution constraints to behave symmetrically [50]. We next investigated linear arithmetic constraints. However, these are quite restricted and are not powerful enough to directly encode, for instance, that a text box should be large enough to contain its textual content. Furthermore, it is not that easy to compile linear arithmetic constraints into script if inequalities are allowed. We also experimented with a combination of one-way and linear arithmetic constraints, but we found that certain combinations of layout and object size constraints that we wanted to support still resulted in cyclic dependencies. Finally, we considered the use of multi-way propagation based constraint solving methods. We chose to use them in the tool because they support the standard graphical constraints and because, at least in theory, it is straightforward to compile them into script. The idea is that we generate a plan of how to adapt the layout given the viewport dimensions, text sizes and changes to interaction as input variables. Since this plan is a system of one-way constraints it is readily compiled into script. This was also the approach taken in [16]. One limitation of this approach was that it was not obvious how to support complex combinations of spring constraints so we required that they were combined either series or in parallel (similar restrictions apply in many widget toolkits). It was also not possible to handle inequalities in the solver: these were handled using a post-solving test for satisﬁability. One of the goals of our authoring tool was that diagrams constructed with it should be able to be compiled into a widely-used interactive graphics format. This naturally led us to target scripted SVG. In order to demonstrate the feasibility of compiling an adaptive diagram to SVG, we took the earth structure example and compiled it by hand, making sure that the structure of the resulting code mirrored our adaptive diagram model. The compilation was purely mechanical; no special optimisations were used in the hand-compiled version. Our conversion of the earth structure diagram into an SVG document resulted in an 80 KiB ﬁle, 74 KiB of which is JavaScript. The complete document can be seen at http://mcc.id.au/2014/earth.svg and has been tested in recent versions of Firefox. As discussed previously, a fundamental observation is that while multi-way constraints are used during editing, at presentation time the number of variables whose values can change due to user interaction is limited and so can be readily compiled into a system of one-way constraints. This is what we did in the hand compilation, also including a one-way constraint solver implemented in JavaScript. 5. User Evaluation We conducted an open ended qualitative user study in order to evaluate the adaptive model and authoring tool and also the completeness of our categorisation of diagram 12

adaptation kinds. The study was designed to answer three questions: What adaptations do such individuals come up with when asked to produce multiple, adapted versions of a diagram presented to them, and are these kinds of adaptations supported by the adaptive diagram model? Can someone with basic experience in the use of diagramming or illustration tools and in web design understand the model of adaptive diagrams underlying the tool? Can that person then use the tool to construct an adaptive diagram according to some given requirements? 5.1. Method Participants: 8 participants were recruited for the study from a university newsletter. All were third year or above IT students and were aged between 18 and 30. They received $20 for participating. Materials and design: The study had 6 parts: Introduction. Semi-structured interview questions were used to determine the participant’s relevant experience in web page and diagram authoring, and whether they had ever had to create multiple versions of the same diagram. Problem motivation. To provide participants with an understanding of the motivation for the tool they were shown (on paper) four variants of the Earth structure diagram demonstrating adaptation to the preferred language of the reader (English and German) and the amount of space available for laying out the diagram. They were asked how they would create and subsequently modify these diﬀerent versions using the tools they were familiar with and how they could create a web version that adapted to screen space and language preference. Adaptation sketching. Participants were presented with the two diagrams shown in Figure 5 and scenarios describing the additional contexts in which the diagrams needed to be presented. They were asked to produce, on paper, adapted versions of these diagrams. The intention of this part was to explore participants’ understanding of adaptation and also the completeness of our classiﬁcation of the various kinds of adaptation. Participants were asked to take 10 minutes to sketch ideas for each of the two diagrams. Training. In this part of the study participants were taught how to use the authoring tool and its underlying model by using the tool to construct an extended version of the Earth structure diagram. Training was hands on and participants were encouraged to ask the investigator questions. Two features of the tool required in the next part of the study but not present in this example were also explained with a small example (alignment of arrow

Figure 5: The two diagrams used in the adaptation sketching task in the user study.

bend points and interaction between guidelines and spring distributions.) Unguided diagram construction. Participants were then asked to construct an adaptive diagram with the tool without guidance from the investigator. The diagram was a ﬂow chart showing how to compute the average of an array of numbers. They were asked to produce horizontal and vertical conﬁgurations for display in colour, greyscale and monochrome. The main loop could be collapsed and expanded interactively. Participants were shown illustrations of the various layouts on paper and told “layout of the diagram must be as shown here and must reasonably accommodate diﬀerent page sizes by resizing and repositioning the objects in the diagram.” One of the example illustrations is shown in Figure 6. A concurrent think aloud protocol [51] was used so as to allow the investigators to better understand what the participants were attempting to do. If participants had trouble or were confused about the tool or the diagram they were permitted to ask questions. This allowed all participants to complete the construction task and the chance to use all aspects of the authoring tool required for this. It also helped emphasise to the investigators which parts of the adaptive diagram model or features of the tool the participants did not understand. Reﬂection. The ﬁnal part of the study was a semi -structured interview in which participants were asked to reﬂect on their use of the tool and adaptive diagrams in general. They were asked about whether the model for adaptation made sense and questions about the usability of the main components in the authoring tool. The complete materials provided to the participants in the user study are available at

http://mcc.id.au/2014/authoring-tool/. Equipment and procedure: The experiments were performed in a small room with a computer at Monash University, with a single participant at a time. The total study took about 3 hours per participant with a short break halfway through. We were unsure whether exposure to the tool might bias the responses to the adaptation sketching task or whether it would help them to think about what was possible. We therefore counter-balanced the presentation order of the adaptation sketching task and the unguided construction task. Participants were explicitly told that the adaptations need not be expressible with the tool if they had already seen it. 5.2. Results The semi-structured interview conducted in the initial part of the study veriﬁed that all of the participants did indeed have some experience using computer-based diagramming tools. Most participants stated that the diagrams they do create tend to be “one oﬀs,” or at most simply scaling of content. Two participants mentioned creating software engineering diagrams with similar content but with diﬀerent focus. When they were asked how they would go about creating the four diﬀerent views of the Earth structure diagram, all indicated one of two approaches: create the large English version ﬁrst, scale it down to produce the small English version, then make a copy of those two versions and translate them individually into German; or, create the large English version, translate it into the large German version, then scale the two large versions down into the two small versions. Most gave some indication that having to subsequently change the diagram would be an annoyance. Adaptation task: In the adaptation sketching task par13

There were two examples of a change to indirection. In the original version of the diagram colours were used to associate bars in the main chart area with their region names in the legend. One participant suggested removing the legend and having the country name appear on the bars themselves. Another participant changed the method of indirection; since the target device did not support colour, he suggested numbering the bars and using a key based on this number rather than colour. Most participants, however, adapted to the lack of colour output on the target device by choosing to render the bars with shades of grey or to use common hatching patterns. The second of the two diagrams was an illustration of a biological process of a type the participants were unlikely to have authored before. The scenario given was that the original diagram was as presented on the page and had to be adapted for presentation on a web page in a desktop web browser. They were told that the size of the browser window could vary signiﬁcantly. Participants came up with two ways of adapting to a small display space. The ﬁrst was a focus adaptation utilising interaction (dynamism) to allow the reader to select one of the steps in the depicted process to view. The second option was, recognising that the illustration depicted a sequence of steps, to make a more schematic version of the diagram by adapting its layout, where the six steps were placed vertically with arrows connecting them, forgoing the detail of the surrounding cytoplasm area from the original version. Three participants proposed the use of animation (also a kind of dynamism adaptation). It was suggested that it be used to move through the steps of the diagram, to help illustrate the movement of the potassium and sodium ions that, in the original version of the diagram, was indicated using arrows, or both. Participants did not suggest any style, form or indirection adaptations for this second diagram. Based on this task we can conclude that all of the adaptations suggested by the participants are covered by our proposed classiﬁcation but that some (scrolling and animation) were not supported by the tool though they did ﬁt into the adaptation model and it would be conceptually straightforward to extend the tool to support them. We also found that participants’ adaptation ideas covered, with varying frequency, all seven adaptation types from our classiﬁcation. Construction task: All participants completed the training satisfactorily. During the unguided diagram construction task participants were conﬁdent and had no trouble with general editing operations that they would have been familiar with from other tools. They also liked the diagram previews and the reorientation tool. All participants were able to complete the construction task (though Participant P1 ran out of time and skipped the colour adaptation) with varying degrees of help or hinting from the investigator. We can therefore respond with a qualiﬁed

Figure 6: Illustration for fully expanded vertical layout of the ﬂowchart used in the construction task of the user study.

ticipants were shown the diagrams in Figure 5. Half of the participants did this task before their use of the adaptive diagram authoring tool, while the others did it afterwards. The task order did not appear to aﬀect the response. The ﬁrst diagram was a bar chart showing six data series plotted at nine diﬀerent points along the x-axis. It was stated that the original version of the diagram was, as presented, reasonably large, taking up most of an A4 page. It also used colours to distinguish the diﬀerent data series. The adaptation scenario was that the diagram had to be presented on an e-book reader device which has a greyscale screen which was only as big as a quarter of a A4 page, and which could be held in either a portrait or landscape orientation. The hypothetical device also supported touch interaction. To adapt to the smaller display space and the two different orientations, six of the eight participants suggested using horizontal scrolling in the main chart area, allowing the user to select the range of years to show in the chart. We considered this to be a combination of dynamism (for the user interaction) and focus (for the data range). In addition to user-controlled scrolling, a number of participants took advantage of the touch screen to add further dynamism to the diagram by allowing the user to control the data series being shown. As for actual layout changes, we saw some instances of the legend items being rearranged to better ﬁt available space and or being moved to diﬀerent places on the page. Some participants sketched a simple reorientation of the diagram to handle the portrait mode, while others noted that doing this might result in text being too small to read. 14

“yes” to the question of whether participants can use the model and tool to construct an adaptive diagram according to requirements given to them. Feedback on features directly corresponding to the underlying model is summarised in Table 2. This also includes responses from the reﬂection task when this was about individual features. Text alternatives: All participants successfully assigned text strings to objects within the diagram. All but one expressed solely positive comments about using the feature to adapt the diagram to the viewers preferred language. For instance,

designed so that it is not possible to try to make such illegal combinations. Styling palette: The styling system drew consistent praise and participants generally had little trouble styling objects and using the palette to adapt the diagram to different colour modes. P1 That’s another wonderful thing. It can help to adapt into diﬀerent devices. So far as I can see, its very good for now. I havent thought of any setbacks yet. P4 It worked really well.

P2 Really easy, self explanatory.

However, there was one issue that did cause confusion. In the ﬁnal part of the task, participants had to ensure that the diagram had particular mappings for colours in the greyscale and monochrome versions. All participants P7 That was easy. That’s the most logical way to do it. who did this part (Participant P1 did not do the adaptation to colour mode due to time) ran up against the There were two main suggestions from participants for imissue of both black and white needing to map to diﬀerprovements to the text feature. The ﬁrst was to incorpoent shades in the other colour modes depending on where rate a dictionary or translation feature that would help in in the diagram those colours were being used. For examentering text strings for alternative languages. The secple, the white used as the background of the YES/NO obond was to provide a centralised interface showing a table jects must remain white in the greyscale and monochrome of all text strings used in the diagram so that translation modes, but the white used for the text of the purple and can occur in the one place, either in addition to or instead red shapes must become black. Understandably, particiof the current in-canvas interface. Initial design plans for pants had used the same white palette entry for both of the authoring tool did use such a view of the text strings, these things, and when they adjusted the mapping of this but it was thought later that being able to edit the text entry, more changed than they wanted to. Only one parin place would help authors see the resulting layout eﬀects ticipant immediately understood why this was and was of diﬀerent text string sizes more readily. In hindsight a able to restyle the shapes to use two diﬀerent palette endual view might be useful. tries. All others needed help to understand that you could Layout objects: Nearly all participants were positive have two whites in the palette with diﬀerent mappings in about the main layout objects and constraints available in other colour modes. One of the participants also persisted the tool and were able to achieve layout adaptation that in selecting particular objects before making changes in more or less followed the diagram requirements. While the palette, even after it having been explained that the some participants had troubles with aspects of the layout palette is independent of any one object. tools, nevertheless they were enthusiastic about them: Possibly as a result of this confusion two participants suggested that the palette should allow naming palette P3 Columns and rows were intuitive and clear. The springs, entries, so that it is clear that, even though two of them they work well, but I admit looking at them to start may have the same colour, they are actually distinct enoﬀ with I was thinking what are all these pink squigtries, and that changing one will not aﬀect objects styled gly lines doing? [...] As I played around with them a with the other. This is a reasonable suggestion; allowing bit more, especially towards the end of the [unguided authors to name entries would help them remember how diagram construction task], I found that they were each entry is used in the diagram by using names such as quite easy to use. Spring distributions within spring “ﬂow chart choice shape” and “heading text.” distributions can get a bit hairy; I found that when Conﬁgurations: Participants generally understood that doing the [...] expanding/collapsing bit for the loop. conﬁgurations were used for diﬀerent major layouts of the Two relatively common mistakes were not understanddiagram, and the mechanism by which the tool at display ing exactly how springs compute their sizes, or why one time would choose between the conﬁgurations. That the might choose to use a column instead of a guideline (a coltool would choose the best display of the diagram based umn also constrains the widths of the children as well as on the page size seemed to be the aspect that was liked the aligning them). A few participants also had some problems most (even though it really is up to the author to deﬁne with the restriction that springs could not be combined arwhat is best by controlling when conﬁguration switching bitrarily and instead must be used either in sequence or occurs). in parallel. This suggests that the layout model is too Interaction editor: Most participants were able to use restrictive or that the interface would beneﬁt from being the interaction editor without any assistance from the investigator. The task required them to add alternative P3 I thought it worked really well. [...] Having it one click change language is really, really good.

15

Participant Text alternatives Layout tools Styling palette Conﬁgurations Interaction editor Masters & alternates

P1

P2

P3

P4

P5

P6

P7

P8

Table 2: Features that participants were able to use and provided positive feedback.

switching actions in response to mouse clicks on the “expand/collapse” button in the diagram. Positive comments centred around the ease of use of the editor. Master objects and alternates: The use of masters and alternates was the aspect that diﬀers most from traditional diagram authoring tools, and so, as we expected, it was the feature that proved most diﬃcult for participants to use. To implement the expanding and collapsing of the ﬂow chart loop, we expected (and intended for) a master object with two alternatives to be used. The use of a master object whose currently displayed alternative was chosen as a result of user interaction paralleled closely the use of the hint master object in the earth structure example from the training. Five of the eight participants were able to use master objects to create the expandable loop section of the ﬂow chart without much intervention and provided favourable comments on it.

objects. The intention was that the entire expanded, vertically oriented ﬂow chart would be designed ﬁrst, and then the relevant objects from the loop selected and promoted to a master object so that they could then be replaced with an alternative, the collapsed view of the loop. The main issue was in selecting the right objects to promote. Mostly this was due to participant selecting only the visual objects, the shapes and arrows, and not the associated layout objects. A second issue was over-eager promotion. Two participants wanted to promote many sub-groups of the conﬁguration to master objects, for example one for diamonds, one for the green START ellipses, one for the red END ellipses, and so on. This would not have worked, since the model does not allow instantiation of master objects within other master objects. It would also not have lead to signiﬁcant re-use. Reﬂection: In the ﬁnal part of the study, we asked participants a number of questions around their use of and opinions about the diﬀerent features of the adaptive diagram authoring tool as well as their thoughts on diagram adaptation generally. Comments about speciﬁc authoring tool features provided in this part of the study were included in the above discussion, so we concentrate here on responses to the more general questions. Participants were ﬁrst asked whether thinking about a single, logical diagram, which adapts its presentation to diﬀerent conditions made sense to them. Answers were uniformly positive, most indicating a recognition of the need to present diagrams diﬀerently under diﬀerent viewing conditions:

P2 I’m an OO programmer, so I kind of understand that. You have an abstract thing and you just work on that. So that kind of makes sense to me straight away. I dont know if the term master object makes sense for a casual user, though. [...] template, is probably a better word [...] But actually working with the diﬀerent alternatives I thought worked ﬁne. The user interface made sense straight away. I didnt need too much explanation for that. P3 Deﬁnitely [made sense]. For me that feels a lot like Flash, thats how Flash kind of does stuﬀ. Inside a movie clip you have frames that you can go to. So the frames were similar [to the alternatives].

P2 Yeah, especially for web pages because these days you have netbooks and all that. You don’t have a standard resolution, especially for mobiles.

P6 Yep, that one was pretty much straightforward if youre doing CS, I guess inheritance and stuﬀ.

P3 Yeah, deﬁnitely. ... It is all about trying to reuse stuﬀ without doing it all from scratch. ... What was really good was using the master objects. If you change the text in one, no matter what the representation its going to replicate through all of them, which is really, really handy. Really simple. Its a simple idea, but never been executed very well. This was really good.

Two participants however initially wanted to use conﬁgurations to represent the two diﬀerent states. When reminded of how the interaction in the earth structure example was achieved, these participants readily changed tack to follow the same approach. We note that in the current model user interaction could not be directly used to change conﬁgurations. However it would be a simple extension to allow conﬁguration switching as a result of interaction. The diﬃculty that most participants encountered and which required the most intervention from the investigator was the promotion of conﬁguration-level objects to master

The second question asked participants whether, if they had to create diagrams like the earth structure or ﬂowchart diagrams they had seen in the previous tasks, whether they would prefer to use a polished version of the adaptive diagram authoring tool or to use the tools and techniques 16

they were already familiar with. All but one participant said that yes, they would prefer to use the adaptive diagram authoring tool, though most qualiﬁed this by saying that it would depend on the kind of diagram they were creating and how they expected to publish it. The one participant who did not say they would use the tool said that they did not have experience with tools that would let them create diagrams with any kind of adaptivity and so declined to say they would prefer to use the tool or not. Participants were then asked whether they thought existing diagramming tools should support functionality to author adaptive diagrams. All except one agreed, although one other was doubtful it would be possible. In their parting comments all participants were very positive about the tool and the underlying model.

The evaluation suggest that the tool could be improved by allowing nested object masters since a number of participants assumed that they could do this. Another issue that the study revealed is that some participants found it difﬁcult to understand the restrictions on combining springs and other layout constraints. This is due to limitations of the underlying constraint solver: more powerful constraint solving technology is required. The authoring tool supported a subset of the diﬀerent kinds of adaptation identiﬁed in our classiﬁcation. Further work would be to extend it to handle other kinds of adaptation and support for other kinds of dynamism including continuous animation, scrolling and ﬁltering. Currently removing or adding indirection requires the author to create new conﬁgurations: is there a way of supporting this as one logical adaptation? We would also like to provide automatic support, such as layout wizards, for constructing standard diagram types such as maps, bar charts or organization charts from external data. This would relieve the burden on authors and also allow the tool to support adaptation based on dynamic content.

6. Conclusion Previous research into document adaptation has focussed upon text, hypertext, multimedia and hypermedia content. Here we address adaptation of diagrams. As far as we are aware, this is the ﬁrst research into models and tools for authoring diagrams that can adapt to their viewing context including user requirements. Based on an examination of 150 representative diagrams from a variety of sources we have identiﬁed the reasons for adapting a diagram to its context and also the main kinds of adaptive behaviour that support these (change of layout, style, form, text, and focus and the introduction/removal of indirection and dynamism). While most are common to other content, indirection and some kinds of focus are peculiar to graphical content. One of the potential diﬃculties when authoring adaptive diagrams is the combinatorial explosion of diﬀerent versions of the diagram when diﬀerent kinds of adaptation are considered. Explicitly authoring independent versions of a diagram for every combination of adaptation is not practical. To address this we have introduced an aspectoriented model for adaptive layout that separates style, text, object, interaction and choice of layout view into different aspects. Doing so dramatically eases the burden on the author by allowing adaptive behaviour to be speciﬁed orthogonally. Diﬀerent versions of a diagram are deﬁned implicitly to be the cross product of these possible adaptations though the model allows the author to ﬁne tune adaptation by specifying compatible combinations. We conducted an empirical evaluation of an authoring tool based on this model. While the study was limited to 8 IT students it provides evidence that the model and tool can be understood and used to author complex adaptive behaviour. Furthermore, almost all participants in the study said that they would prefer to use a (polished) version of the adaptive diagram authoring tool over more conventional diagramming tools for constructing adaptive diagrams. Future work would be to trial the tool with graphic designers without a background in IT to see if the results carried over.

Acknowledgements We are grateful to the following people for resources, discussions and suggestions: Tim Dwyer, Nathan Hurst and Michael Wybrow. We acknowledge the support of the ARC through DP0449546 and DP0987168 [1] C. McCormack, K. Marriott, B. Meyer, Authoring adaptive diagrams, in: Proceedings of the 8th ACM Symposium on Document Engineering, ACM, 2008, pp. 154–163. [2] C. Jacobs, W. Li, E. Schrier, D. Bargeron, D. Salesin, Adaptive document layout, Communications of the ACM 47 (8) (2004) 60–66. [3] K. Moore, Every page is diﬀerent: a new document type for commercial printing, in: Proceedings of the 2006 ACM Symposium on Document Engineering, ACM, 2006, pp. 2–2. [4] J. Ferraiolo, J. Fujisawa, D. Jackson, Scalable Vector Graphics (SVG) 1.1 speciﬁcation, http://www.w3.org/TR/SVG11/ (January 2003). [5] A. Borning, R. K.-H. Lin, K. Marriott, Constraint-based document layout for the web, Multimedia Systems 8 (3) (2000) 177–189. doi:http://dx.doi.org/10.1007/s005300000043. [6] M. Jourdan, N. Laya¨ıda, C. Roisin, L. Sabry-Isma¨ıl, L. Tardif, Madeus, an authoring environment for interactive multimedia documents, in: Proceedings of the 6th ACM International Conference on Multimedia, ACM, 1998, pp. 267–272. [7] G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.M. Loingtier, J. Irwin, Aspect-oriented programming, Springer, 1997. [8] B. T. Vander Zanden, R. Halterman, B. A. Myers, R. McDaniel, R. Miller, P. Szekely, D. A. Giuse, D. Kosbie, Lessons learned about one-way, dataﬂow constraints in the garnet and amulet graphical toolkits, ACM Transactions on Programming Languages and Systems (TOPLAS) 23 (6) (2001) 776–796. [9] B. N. Freeman-Benson, J. Maloney, A. Borning, An incremental constraint solver, Communications of the ACM 33 (1) (1990) 54–63. [10] B. Vander Zanden, An incremental algorithm for satisfying hierarchies of multiway dataﬂow constraints, ACM Transactions on Programming Languages and Systems (TOPLAS) 18 (1) (1996) 30–72.

17

[11] A. Borning, K. Marriott, P. Stuckey, Y. Xiao, Solving linear arithmetic constraints for user interface applications, in: Proceedings of the 10th annual ACM Symposium on User Interface Software and Technology, ACM, 1997, pp. 87–96. [12] N. Hurst, W. Li, K. Marriott, Review of automatic document formatting, in: Proceedings of the 9th ACM Symposium on Document Engineering, ACM, 2009, pp. 99–108. [13] G. J. Badros, A. Borning, K. Marriott, P. Stuckey, Constraint cascading style sheets for the web, in: Proceedings of the 12th annual ACM symposium on User interface software and technology, ACM, 1999, pp. 73–82. [14] J. Lumley, R. Gimson, O. Rees, A framework for structure, layout & function in documents, in: Proceedings of the 2005 ACM Symposium on Document Engineering, ACM, New York, NY, USA, 2005, pp. 32–41. doi:http://doi.acm.org/10.1145/1096601.1096615. [15] X. Lin, Active layout engine: Algorithms and applications in variable data printing, Computer-Aided Design 38 (5) (2006) 444–456. [16] C. Jacobs, W. Li, E. Schrier, D. Bargeron, D. Salesin, Adaptive grid-based document layout, ACM Transactions on Graphics 22 (3) (2003) 838–847. doi:http://doi.acm.org/10.1145/882262.882353. [17] Y. Chen, X. Xie, W.-Y. Ma, H.-J. Zhang, Adapting web pages for small-screen devices, IEEE Internet Computing 9 (1) (2005) 50–56. [18] C. Boyle, A. O. Encarnacion, Metadoc: an adaptive hypertext reading system, in: Adaptive Hypertext and Hypermedia, Springer, 1998, pp. 71–89. [19] P. De Bra, A. Aerts, B. Berden, B. De Lange, B. Rousseau, T. Santic, D. Smits, N. Stash, Aha! the adaptive hypermedia architecture, in: Proceedings of the 14th ACM Conference on Hypertext and Hypermedia, ACM, 2003, pp. 81–84. [20] P. Brusilovsky, Adaptive hypermedia, User modeling and useradapted interaction 11 (1-2) (2001) 87–110. [21] L. Weitzman, K. Wittenburg, Automatic presentation of multimedia documents using relational grammars, in: Proceedings of the 2nd ACM International Conference on Multimedia, ACM, 1994, pp. 443–451. [22] L. Rutledge, J. Van Ossenbruggen, L. Hardman, D. C. Bulterman, A framework for generating adaptable hypermedia documents, in: Proceedings of the 5th ACM International Conference on Multimedia, ACM, 1997, pp. 121–130. [23] J. F. Allen, Maintaining knowledge about temporal intervals, Communications of the ACM 26 (11) (1983) 832–843. [24] M. Jourdan, C. Roisin, L. Tardif, Constraint techniques for authoring multimedia documents, Constraints 6 (1) (2001) 115– 132. [25] S. Laborie, Spatio-temporal proximities for multimedia document adaptation, in: Artiﬁcial intelligence: Methodology, systems, and applications, Springer, 2006, pp. 128–137. [26] B. Pellan, C. Concolato, Adaptation of scalable multimedia documents, in: Proceedings of the 8th ACM Symposium on Document Engineering, ACM, 2008, pp. 32–41. [27] B. Pellan, C. Concolato, Authoring of scalable multimedia documents, Multimedia Tools and Applications 43 (3) (2009) 225– 252. [28] M. A. Linton, P. R. Calder, J. M. Vlissides, InterViews: A C++ graphical interface toolkit, Computer Systems Laboratory, Stanford University, 1988. [29] B. A. Myers, D. A. Giuse, R. B. Dannenberg, B. V. Zanden, D. S. Kosbie, E. Pervin, A. Mickish, P. Marchal, Garnet: Comprehensive support for graphical, highly interactive user interfaces, Computer 23 (11) (1990) 71–85. [30] S. E. Hudson, User interface speciﬁcation using an enhanced spreadsheet model, ACM Transactions on Graphics (TOG) 13 (3) (1994) 209–239. [31] S. E. Hudson, I. Smith, Ultra-lightweight constraints, in: Proceedings of the 9th Annual ACM symposium on User Interface Software and Technology, ACM, 1996, pp. 147–155. [32] S. E. Hudson, S. P. Mohamed, Interactive speciﬁcation of ﬂex-

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45] [46]

[47] [48] [49] [50]

[51]

18

ible user interface displays, ACM Transactions on Information Systems (TOIS) 8 (3) (1990) 269–288. B. A. Myers, R. G. McDaniel, R. C. Miller, A. S. Ferrency, A. Faulring, B. D. Kyle, A. Mickish, A. Klimovitski, P. Doane, The amulet environment: New models for eﬀective user interface software development, IEEE Transactions on Software Engineering 23 (6) (1997) 347–365. C. Lutteroth, R. Strandh, G. Weber, Domain speciﬁc high-level constraints for user interface layout, Constraints 13 (3) (2008) 307–342. K. Coninx, K. Luyten, C. Vandervelpen, J. Van den Bergh, B. Creemers, Dygimes: Dynamically generating interfaces for mobile computing devices and embedded systems, in: Human-Computer Interaction with Mobile Devices and Services, Springer, 2003, pp. 256–270. G. J. Badros, J. J. Tirtowidjojo, K. Marriott, B. Meyer, W. Portnoy, A. Borning, A constraint extension to scalable vector graphics, in: Proceedings of the 10th International Conference on World Wide Web, ACM, 2001, pp. 489–498. C. McCormack, K. Marriott, B. Meyer, Adaptive layout using one-way constraints in SVG, in: SVG Open 2004: Proceedings of the 3rd Annual Conference on Scalable Vector graphics, 2004. J. W. Lumley, Functional, extensible, svg-based variable documents, in: Proceedings of the 2013 ACM Symposium on Document Engineering, ACM, 2013, pp. 131–140. K.-F. B¨ ohringer, F. N. Paulisch, Using constraints to achieve stability in automatic graph layout algorithms, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1990, pp. 43–51. T. Dwyer, K. Marriott, F. Schreiber, P. Stuckey, M. Woodward, M. Wybrow, Exploration of networks using overview+ detail with constraint-based cooperative layout, Visualization and Computer Graphics, IEEE Transactions on 14 (6) (2008) 1293–1300. I. E. Sutherland, Sketch pad a man-machine graphical communication system, in: Proceedings of the SHARE design automation workshop, ACM, 1964. G. Nelson, Juno, a constraint-based graphics system, in: ACM SIGGRAPH Computer Graphics, Vol. 19, ACM, 1985, pp. 235– 243. D. Kurlander, S. Feiner, Inferring constraints from multiple snapshots, ACM Transactions on Graphics (TOG) 12 (4) (1993) 277–304. T. Dwyer, K. Marriott, M. Wybrow, Dunnart: A constraintbased network diagram authoring tool, in: Graph Drawing, Springer, 2009, pp. 420–431. R. Lewis, Authoring challenges for device independence, http: //www.w3.org/TR/acdi/ (September 2003). G. L. Lohse, K. Biolsi, N. Walker, H. H. Rueter, A classiﬁcation of visual representations, Communications of the ACM 37 (12) (1994) 36–50. J. von Engelhardt, The language of graphics, Ph.D. thesis, Universiteit van Amsterdam (2002). P. Edman, Tactile graphics, American Foundation for the Blind, 1992. J. M¨ uller-Brockmann, Grid systems in graphic design, Hastings House Publishers, 1981. M. Wybrow, K. Marriott, L. Mciver, P. J. Stuckey, Comparing usability of one-way and multi-way constraints for diagram editing, ACM Transactions on Computer-Human Interaction (TOCHI) 14 (4) (2008) 19. K. A. Ericsson, H. A. Simon, Protocol analysis, MIT-press, 1984.

Authoring diagrams that adapt to their viewing context

Authoring diagrams that adapt to their viewing context

Recommend Documents