- Email: [email protected]
CHAIN: Developing model-driven contextual help for adaptive user interfaces
Accepted Manuscript
CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces Pierre A. Akiki PII: DOI: Reference:
S0164-1212(17)30250-9 10.1016/j.jss.2017.10.017 JSS 10056
To appear in:
The Journal of Systems & Software
Received date: Revised date: Accepted date:
7 February 2017 26 September 2017 13 October 2017
Please cite this article as: Pierre A. Akiki , CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces, The Journal of Systems & Software (2017), doi: 10.1016/j.jss.2017.10.017
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT The Journal of Systems and Software
1
Highlights
CE
PT
ED
M
AN US
CR IP T
CHAIN is an approach for developing model-driven contextual help for adaptive UIs. CHAIN help models are defined using a language (CHAINXML) and a visual notation. The definition of help models is supported by Cedar Studio. CHAIN help models work with both new and legacy application UIs. Two evaluation studies provided positive indications and insights for future work.
AC
ACCEPTED MANUSCRIPT 2
CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces Pierre A. Akiki Department of Computer Science, Notre Dame University – Louaize, Lebanon
AN US
CR IP T
Abstract— Adaptive user interfaces (UIs) change their presentation at runtime to remain usable in different contexts-of-use. Such changes can cause discrepancies between the UI and static help materials, e.g., videos and screenshots, thereby negatively impacting the latter’s usefulness (usability and utility). This paper presents Contextual Help for Adaptive INterfaces (CHAIN), which is an approach for developing model-driven contextual help that maintains its usefulness across UI adaptations. This trait is achieved by interpreting the help models at runtime and overlaying instructions on the running adapted UI. A language called Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) and a visual notation were developed for expressing and depicting help models. A technique was also devised for presenting CHAIN help models over legacy applications, whether or not their source-code is available. A supporting tool was developed as an extension to Cedar Studio. This work was empirically evaluated in two studies. The first study performed a preliminary evaluation of CHAIN’s visual notation. The second study evaluated CHAIN’s strengths and shortcomings after using it to develop help for real-life adaptive UIs. The results gave a positive indication about CHAIN’s technical qualities and provided insights that could inform future work. Keywords: Contextual Help, Adaptive User Interfaces, Model-Driven Help, Software Support, Design Tools and Techniques
1 INTRODUCTION
A
AC
CE
PT
ED
M
daptive user interfaces (UIs) change their presentation at runtime to preserve their usability in different contexts-of-use (user, platform, and environment) [1]. Such changes negatively affect the usefulness (usability and utility [2], p.25) of static end-user help such as videos and screenshots. This undesirable effect is due to discrepancies between the initial UI, on which the help is based, and the adapted UI. For example, a widget would become difficult to locate using static UI help if its location changes at runtime. Therefore, adaptive UIs require a new type of help system, which also needs to be adaptive [3]. This paper presents an approach for developing model-driven contextual UI help as an alternative to static help. Contextual help is a form of assistance that is presented to end-users on the running UI, rather than in a separate viewer such as a browser or a video player [4]. As end-users interact with a running UI, contextual help can provide assistance by displaying popup messages, entering example data, highlighting elements, and so on. Representing contextual help as interpreted runtime models, allows instructions and markings to be overlaid dynamically on the running UI, thereby maintaining the help‟s usefulness for different UI adaptations.
1.1 Advantages of Model-Driven Contextual Help Non-contextual help is presented apart from the UI, e.g., using a video player or a secondary display [5]. Moving back and forth between the UI and the help viewer could distract end-users and increase their memory load. Since contextual help is presented on top of the running UI, it can more effectively support the end-users‟ learning pro————————————————
Pierre A. Akiki is with the Department of Computer Science, Notre Dame University – Louaize, Zouk Mosbeh, P.O. Box: 72, Zouk Mikael, Lebanon. E-mail: [email protected]
cess [4]. A direct advantage of representing contextual help as models, is allowing its content to be updated if the UI is modified due to rapidly changing requirements, e.g., in web applications. On the other hand, static help artifacts like videos are more costly to update, because they would have to be redeveloped from scratch. Cheng et al. [6] consider that “software systems must become more versatile, flexible, … , customizable, configurable, and self-optimizing by adapting to changing operational contexts, environments or system characteristics”. Adaptive UIs have been researched as a way for solving usability problems when contexts-of-use vary. Several approaches have been devised for developing adaptive UIs [7]. For example, Comet(s) are a set of widgets that support UI plasticity by either self-adapting or being adapted [8]. Supple uses preference elicitation and cost functions to generate UIs, which are adapted to an end-user‟s abilities and preferences [9]. RBUIS adopts an interpreted runtime model-driven approach for developing adaptive UIs, which offer a minimal feature-set and an optimal layout based on the context-of-use [10]. Regardless of the adopted development approach, adaptive UIs can ultimately change their visual characteristics, thereby becoming inconsistent with the content of static help materials. The work presented in this paper, complements UI adaptation techniques by allowing software applications to benefit from adaptive UI capabilities, while providing end-users with contextual help that maintains its usefulness with different adaptations of the same UI. For example, enterprise applications like enterprise resource planning (ERP) systems can benefit from adaptive behavior to tailor their thousands of complex UIs for different contexts-of-use. These applications, e.g., Lawson Smart Office [11], can also benefit from offering help to their end-users.
2017
ACCEPTED MANUSCRIPT 3
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 1. An Example of a Single Contextual Help Scenario Running on Two Different Adaptations of the Same ―Customer Information‖ UI -
AN US
(a) UI V1 Groups its Widgets Using Group Boxes, whereas (b) UI V2 Groups its Widgets Using Tab Pages
AC
CE
PT
ED
M
1.2 Proposed Contextual Help Development Approach This paper presents a help development approach called Contextual Help for Adaptive Interfaces (CHAIN). Help models can be defined using an XML-compliant language called CHAINXML and a visual notation. A supporting visual tool was added as an extension to Cedar Studio, an integrated development environment (IDE) for adaptive model-driven UIs [12]. Sophisticated contextual help can be created using this tool without any programming. Software applications can request help models by calling a web-service that returns a CHAINXML document. This document is interpreted at runtime and presented over the running UI as contextual help. The example illustrated in Fig. 1 demonstrates part of a contextual help scenario running over two different adaptations of the same UI. This simple example shows the importance of the proposed approach for developing adaptive contextual help. If a static help development technique was used, then a different artifact (e.g., video) would be needed for each adaptation (V1 and V2) of the UI. Having many runtime variations of the context-ofuse makes developing different versions of the help very costly or even unfeasible. CHAIN help scenarios can be played on a timeline. The end-users control a help scenario by going forward and backward, as is done in videos, using a slider and a set of buttons like the ones shown in Fig. 1 – c. Endusers can also request part of a help scenario that explains a specific section of the UI. CHAIN is also useful for developing help for nonadaptive UIs. Yet, it primarily targets the development of contextual help for adaptive UIs. Hence, the latter will be the focus of this paper.
1.3 Structure of the Article Section 2 categorizes the related work according to the adopted development approach, and discusses its strengths and shortcomings. Section 3 presents CHAIN‟s metamodel, while Section 4 presents its XML-compliant lan-
guage (CHAINXML) and visual notation. Section 5 explains how help models can be developed and validated based on a proposed architecture. Section 6 explains how CHAIN help maintain its usefulness as the UI adapts. Section 7 explains how CHAIN help models can be used with legacy systems, regardless of whether the source-code of these systems is available. A tool for supporting the development of CHAIN help is presented in Section 8. The work presented in this paper is evaluated in two studies, which are presented in Sections 9 and 10. The first study is a preliminary evaluation to obtain feedback on CHAIN‟s visual notation. The second study evaluates the strengths and limitations of CHAIN through actual use. Finally, the conclusions are given and the future work is discussed in Section 11.
2 RELATED WORK Some software help systems, e.g., the work done as part of project Genie [13], date back several decades. To the best of my knowledge, contextual help that maintains its usefulness across different UI adaptations, has not been targeted by the state-of-the-art help development systems. Researchers have used different approaches for developing various forms of end-user help. This section classifies the existing works based on the adopted development approach and points out their strengths and shortcomings. A summary of the state-of-the-art help systems discussed in this section is presented in Table 1. This table briefly states each system‟s expected input and produced output. The UI level of abstraction that each system operates on is also shown in Table 1, with respect to the Cameleon Reference Framework (CRF) [1].
2.1 Static Non-Contextual Help Static non-contextual help can have different forms such as: external text-based documentation, pictures, and videos, and it accompanies software applications to guide end-users on how to perform certain tasks. Plaisant et al. provide guidelines for developing prerecorded demonstra-
ACCEPTED MANUSCRIPT 4
Pierre A. Akiki / The Journal of Systems and Software
TABLE 1 Summary of the State of the Art Help Systems CRF Level
Help and Manual [16]
Text and figures
Help document (PDF, Word, etc.)
-
ScreenSteps [19]
Text and screenshot
Help document (PDF, Word, etc.)
-
Steps Recorder [18]
Key log, screenshots, and annotations
Help steps as text and screenshots
-
Snagit [17]
Screenshots and audio recording
Help video
-
CogentHelp [21]
Help Parts
Help Pages
AUI
Balloon Tooltips [22]
Programmed tooltips
Tooltips showing brief messages
CUI
SideViews [23]
Programmed side view (MVC)
Interactive tooltip
CUI
Toolclips [24]
Text and video
Tooltips with text and video
CUI
Crystal framework [25]
Built-in answer generator
Dialogs that answer questions
CUI, FUI
Writer’s Aid [26]
User preferences, bibliography information, and AI planning
Help for identifying and inserting citations in a document
CUI, FUI
Dixon et al. [27]
UI content and hierarchy
Stencil-based tutorials
FUI
Sikulli [28]
Script and screenshots of UI elements
Slide show presented over the UI
FUI
Microsoft [29], Apple [30], JitBit [31], MiniMouse [32], Selenium [33], CoScripter [34]
End-user activities that are recorded or manually specified (e.g., using a script)
Activity playback over the UIs of desktop or web applications
FUI
Task-Interface-Log [35] OWL [36]
End-user UI interaction log for a task Logged user data (e.g., commands)
Greco et al. [37]
System logs
AIMHelp [38]
End-user interaction information and the monitored state of the GUI
Playback of previous interactions Intelligent tips and skillometer Helps administrators in identifying frequent workflow executions Automatically generated help information Previously asked questions and their answers for selected UI part Help wiki page created by the community
Xplain [41], [42]
AN US
M
Automatic context identification
FUI FUI FUI FUI FUI FUI
Task model association with the UI
Self-explanatory UI
Annotations on task models
Help for Adaptive UIs
Task
Pangoli et al. [43]
Tasks associated with interaction objects
Generated task oriented help
Task
Moriyon et al. [55]
Model (presentation, behavior, and help extensions) and help rules
Generated help text
CUI
CE
Kern et al. [3]
Task, CUI
Collagen [44]
Task model
DiamondHelp [56], [57]
Task model (based on Collagen), and user actions and utterances
Agent that helps end-users in performing certain tasks Collaborative dialogues between application and end-users
Dekoven [45]
Task model (based on Collagen)
Help for physical appliances
Task
HATS [46]
ATOMS task model
Task-oriented context sensitive help
Task
Gribova [47]
Ontology models linking tasks to UI
Context-sensitive help
AC
Model-Driven Engineering
ED
CHiC [40]
UI part on which end-user requires help
PT
Frameworks and Toolkit Extensions Usage Logging Crowdsourcing
Lemon-Aid [39]
CR IP T
-
Static NonContextual Help
Help document (PDF, Word, etc.)
Pixel Analysis
Output
Text and figures
Macro Recorders
Input Helpndoc [15]
tions on how to use the UIs of software systems [14]. A variety of authoring tools are available for supporting the development of static non-contextual help. For example, some tools support the preparation of HTML-based help files, while others support screen casting for recording demonstration videos. Helpndoc [15] and Help and Manual [16] are two examples of such tools. Snagit can be used to prepare screenshot tutorials [17], while Steps Recorder
Task Task
Task, CUI
[18] can create screenshots with descriptions based on a set of actions that an end-user performs on Windows. Although some tools such as ScreenSteps [19] refer to the type of help they support as “contextual”, but the help materials are mainly screenshots and not overlays on the running UI. For example, ScreenSteps was used with Sales Force [20] to develop help lessons that only improve on fully text-based help documents by including screenshots with annotations.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
PT
ED
M
AN US
2.2 Frameworks and Toolkit Extensions Frameworks and toolkit extensions can be built on top of presentation technologies such as Java Swing and Windows Forms, in order to support the development of enduser help systems. For example, CogentHelp was developed with Java‟s Abstract Windowing Toolkit (AWT) as a prototype tool for authoring, maintaining, and customizing help [21]. With CogentHelp, technical stakeholders, e.g., developers, author the help system parts that describe how individual windows or widgets function. Then, CogentHelp generates help pages based on the individual parts that these stakeholders produced. Balloon tooltips [22] is a help system that presents basic explanation inside popup balloons. Other examples such as Side Views [23] and ToolClips [24] can present richer help content. The ability to ask questions about the UI allows endusers to stay informed on what is happening [48]. One example targeting this ability is the Crystal framework, which goes beyond simple help catalogues into answering questions on: “why do things occur or do not occur”, and “how to use certain application features” [25]. AI planning is used by some systems such as SmartAidè [49] and Writer‟s Aid [26], in order to offer advanced help. Despite their importance, these techniques do not offer contextual help that works with adaptive UIs. Although some systems, e.g., CogentHelp, can generate contextual help content that targets a particular context, this help does not adapt to varying context-of-use. Also, frameworks and toolkit extensions can make the help content technology dependent. Other techniques such as pixel analysis and model-driven engineering can provide technology independence.
2.4 Macro Recorders Macro recorders are used to record user-activity on a UI in order to replay it later. These tools are not primarily intended for developing end-user help, but are aimed at automating routine activities. Microsoft Office applications support the creation of macros for automating tasks [29]. JitBit [31] and Mini Mouse [32] are two examples of macro recorders that can support various Windows applications. Apple Automator automates activities on a variety of Mac OS applications [30]. Selenium is a browser plugin that supports recording and playback of activities on web applications [33]. One example from the academic literature is CoScripter [34], which supports collaboration on task recording and automation in web-based systems. Macro recorders do not support annotations such as markings and callouts, which are necessary for end-user help to provide explanation. Some macro recorders are negatively affected by resolution change. This limitation also hinders the ability to operate on different adaptations of a UI. Furthermore, macro recorders cannot validate if a macro still conforms to a UI after it gets updated, whereas this trait is possible with help models through model validation. A macro-like tool for recording user-activities can complement a visual help model design tool and facilitate the creation of help content. Developers or expert users could perform a task using a UI and their actions would be recorded in order to serve as a basis for defining help models.
CR IP T
As mentioned in Section 1.1, the main shortcoming of non-contextual help materials such as prerecorded videos and screenshots lies in their static nature, which prevents them from preserving their usefulness as the UI adapts.
5
AC
CE
2.3 Pixel Analysis Pixel analysis is used by some techniques for providing contextual help on UIs without accessing the software application‟s source code. These techniques can be considered as attachments, which are small interactive programs that augment the functionality of other applications [50]. Dixon et al. used pixel analysis for extending UIs with stencil-based tutorials [27]. Sikuli allows endusers to query a help system using screenshots from the GUI [28]. Yeh et al. presented a technique that analyzes the pixels of GUI screenshots and provides contextual help using a scripted slideshow [4]. An advantage of pixel analysis is its ability to support the development of technology independent help, which can be added to existing applications without code modifications. Yet, pixel analysis is inhibited by scale changes and theme variations in the UI [4]. Hence, if it is used with adaptive UIs, these limitations could be crippling. Adaptive UIs adapt at runtime characteristics such as: accessibility of functions, colors, icons, information density, image-to-text ratio, and widget types [51]. Adapting such characteristics can make it impossible to locate a UI element through its initial pixel representation.
2.5 Usage Logging Some approaches log the usage history data of end-users over a long period of time and play it back to explain how certain software activities are used. Babaian et al. provide a model called the Task-Interface-Log (TIL), which logs the tasks performed by enterprise system end-users for later playback [35]. Organization-Wide Learning (OWL) is a recommender system that logs end-user data and uses it to provide learning tips [36]. Greco et al. use system logs to help administrators in identifying the frequent past choices that lead to a desired configuration [37]. AIMHelp generates help content by eliciting interaction information at runtime [38]. Like macro recorders, this approach does not provide the ability to add annotations on the UI. Existing techniques such as the TIL do not work with a legacy system unless there is access to its source-code. Logging data in large-scale systems such as enterprise applications can consume a lot of storage space, and it could be hard to identify the part of the data that is needed for explaining an activity. On the other hand, CHAIN help models are composed based on knowledge from human experts. Striking a balance between automation and human input is important for realizing systems that efficiently produce a high quality output. Another disadvantage that usage logging shares with macro recorders is the difficulty of supporting multilingual help. Querying usage logs for entries in the target language could be difficult. On the other hand, help models can target multiple human languages.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
Fig. 2. A Class Diagram Representing the Meta-Model of CHAIN’s Abstract Concepts
CR IP T
6
AC
CE
PT
ED
M
2.6 Crowdsourcing Some tools rely on crowdsourcing to involve end-users in the process of developing help for software applications. LemonAid is a tool that leverages the knowledge of the crowd to provide contextual help for web applications [39]. End-users can benefit from previously answered questions about specific parts of the UI. While using a web application, end-users can ask for help on a specific part of the UI, and LemonAid provides them with a searchable list of questions and answers that were generated by the crowd. CHiC is another example tool in this category. It provides community help for the Eclipse development environment [40]. Although this approach does not directly target adaptive UIs, it has a promising potential for involving endusers in the development of contextual help for such UIs. The data collected through crowdsourcing can potentially be used for producing CHAIN models. For example, if end-users were given an automation tool (like a macro recorder) for recording their on-screen activities, it would be possible to generate a CHAIN model from this recording. Crowdsourcing already has its applications in the discipline of software engineering [52], and it shows a potential for involving end-users in UI adaptation [53]. Hence, it could allow expert end-users to create help for end-users who are less knowledgeable about how to use a certain software system.
2.7 Model-Driven Engineering Model-driven engineering (MDE) can be leveraged to develop end-user help. With MDE, it is possible to offer help by anticipating problems that end-users may face at runtime [54]. García Frey et al. used MDE to develop selfexplanatory UIs [41] with the help of a tool called Xplain [42]. Kern et al. used annotations on task models to provide adaptive help for adaptive UIs [3]. Several MDEbased approaches target automatic generation of end-user help. Pangoli et al. automatically generate task-oriented help from abstract models [43]. Moriyon et al. automati-
cally generate help text from models [55]. DiamondHelp [56], [57] uses task models and the Collagen [58] middleware for supporting collaborative dialogues between an application and its end-users. Collagen takes a task model as an input and provides an agent that helps end-users in performing certain tasks [44]. Dekoven uses the same approach to produce textual help points that primarily target physical appliances [45]. HATS uses task models to generate context sensitive help, which informs end-users about how the interaction could be done in order to achieve certain tasks [46]. Gribova presented an approach for generating context-sensitive help by using ontology models that link tasks to their respective UI elements [47]. The existing work demonstrates the potential of using MDE for developing help systems, which are technology-independent and dynamic. However, it does not target help that remains useful when a UI adapts, thereby leaving many challenges to be addressed. One exception is the work from Kern et al. [3], but this work is at a preliminary stage and does not demonstrate a way to replace complex static help artifacts like videos. The automatic help generation offered by some MDEbased approaches could reduce development costs and anticipate help scenarios at runtime. However, in many cases, full automation might not be practical enough for developing complex help scenarios, which require the input of a domain expert. Hence, the approach presented in this paper offers human experts the tool support they need for developing model-driven contextual help. Modelling languages like the Unified Modelling Language (UML) [59], do not provide dedicated models for representing end-user help. The Cameleon Reference Framework [1] suggests multiple levels of abstraction for developing model-driven UIs: task model (e.g., ConcurTaskTrees [60]), abstract UI (AUI), concrete UI (CUI), and final UI (FUI). Yet, these levels of abstraction only represent a UI‟s content. This paper presents a novel approach for representing model-driven contextual help.
ACCEPTED MANUSCRIPT 7
AC
CE
PT
ED
M
AN US
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 3. A Class Diagram Representing the Meta-Model of CHAIN’s Concrete Concepts
2.8 Challenges for Help Systems Some challenges can be deduced after exploring the related work and the different approaches that were adopted for developing end-user UI help. Help systems can become more effective by overcoming these challenges. As
explained previously, some challenges were overcome by certain approaches but not others. The first challenge lies in providing technology independence, which can be overcome by adopting a modeldriven approach. The second challenge is the efficient development of a high quality help, by providing a bal-
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
CR IP T
8
M
Fig. 4. A Class Diagram Representing CHAIN Resources
CE
PT
ED
ance between human input on the help content and automation through tool support. The third challenge is involving end-users in the definition of help. Olsen suggests that UI systems could empower new design participants such as end-users by introducing them into the design process [61]. Crowdsourcing can be one way to achieve this suggestion. Finally, a major challenge is developing end-user help that can adapt and maintain its usefulness when its corresponding UI adapts. The latter is not addressed by any of the existing systems; hence, it will be the main focus of this paper without disregarding the other key challenges.
3 CHAIN CONCEPTS
AC
Contextual Help for Adaptive INterfaces (CHAIN) supports the definition of end-user contextual help, which maintains its usefulness across different adaptations of the same UI. Some systems such as Helpndoc [15] and Help and Manual [16] are used to develop tutorials and manuals for end-users. Other systems are not dedicated for developing help, but provide a form of guidance for endusers. For example, the UI development systems MacIDA and TRIDENT offer guidance related to task progression, so end-users can identify the tasks that were done and those that remain [62]. On the other hand, the work presented in this paper is a UI help system that is dedicated for the development of contextual help. In model-driven UIs that are based on the Cameleon Reference Framework, abstract UI (AUI) models are independent of any modality, e.g., graphical, while con-
crete UI (CUI) models are modality-dependent. Since help models are related to UIs, CHAIN encompasses abstract (Fig. 2) and concrete (Fig. 3) concepts. CHAIN currently supports the graphical UIs at the concrete level. Yet, if other modalities are added in the future the abstract help model will be mapped to the concrete help models representing the different modalities.
3.1 Abstract Concepts The class diagram shown in Fig. 2 represents CHAIN‟s abstract concepts. Abstract help models are composed of AbstractHelpModelElements, which can be one of four types: AbstractHelpContainer, AbstractHelpOperation, AbstractUserInput, and AbstractResponseToUserInput. These elements have a sortorder attribute that indicates the order in which an element is presented to the end-user. An AbstractHelpContainer groups related help model elements that will be presented together. For example, elements that explain a related part of the UI can be grouped under one container. AbstractHelpOperations embody the help that will be presented to the end-users. Each operation can be one of three types: explanation, demonstration, and emphasis. Explanation operations instruct end-users on how to perform a task. For example, on the concrete level, these operations can be mapped to a voice narration or a callout that displays text. On the other hand, demonstration operations show the end-users how to perform a task. For example, these operations can involve moving the cursor, typing text, selecting values, and so on. Finally, emphasis operations are used to make parts of the UI
ACCEPTED MANUSCRIPT 9
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 5. A Class Diagram Representing Some Possible CHAIN Concrete User Inputs and Responses
them in the execution of a certain task. For example, in a line-of-business application, a Scenario can demonstrate how a sales invoice is created. A speed can be specified to indicate how fast a Scenario is presented. A Scene is a single step in a Scenario. For example, “select customer” can be a Scene in the “create sales invoice” Scenario. A ScenarioFlow indicates how a Scenario moves from one Scene to another. The flow can be automatic like a video, or it can be manual thereby end-user input would be required to proceed. A Scenario can also be used as a static annotation, where all Scenes are presented at once like a single snapshot. A SceneFlow indicates whether a Scene should run in parallel with other Scenes or sequentially after the previous one. Each Scene is composed of a number of SceneElements, each having one of the following four types: Action, Creation, Speech, and Animation. Action scene-elements invoke a change such as an input. TextInput simulates human-like typing at a specified speed, one letter at a time, in an input ConcreteUIElement (e.g., text box). This simulation gives data-entry a natural feel. A ValueChange can be used for changing the value of a selection ConcreteUIElement (e.g., combo box). The MouseMovement simulates cursor movement to guide end-users from one ConcreteUIElement to another. Mouse and Keyboard Commands can also be sent to the UI. Creation scene-elements create help visualizations. A Callout is created next to a UIElement to describe its purpose. A Marking can be used to attract the end-user‟s attention to part of the UI, by highlighting it with a different color or drawing a shape around it. A UIWindow can be used to show supporting UIs, e.g., those used for looking up information. Spoken narration is recommended by existing guidelines on producing help demonstrations [14]. Both Synthesized and Human Speech are supported by CHAIN. SynthesizedSpeech configures and invokes a text-to-speech engine to read a text, while HumanSpeech plays a prerecorded human voice.
PT
ED
M
AN US
more apparent to the end-users. For example, these operations can be realized by highlighting or magnifying a widget. AbstractHelpOperations are applied to AbstractUIElements, which are in turn mapped to ConcreteUIElements. Help models can become interactive by eliciting enduser input and providing responses. On the abstract level, inputs and responses are specified using AbstractUserInputs and AbstractResponsesToUserInputs. Examples of user input on the concrete level include: pressing a button, selecting an option, typing a question, and so on. As a response to an end-user‟s input, the system can, for example, present a relevant part of the help model. Each of the classes representing abstract concepts are assigned one of the following stereotypes: <>, <>, <>, and <>. These stereotypes are also assigned to classes that represent help models at the concrete level, in order to show how the concrete concepts match their abstract counterparts.
AC
CE
3.2 Concrete Concepts The class diagram shown in Fig. 3 represents CHAIN‟s concrete concepts for graphical UIs. Other modalities, e.g., voice and tactile, can be explored further in the future. For example, the current support for speech could form a basis for exploring fully vocal help in the future. Speech is currently supported in CHAIN as part of the help for graphical UIs. Yet, in the future, speech could be explored with purely vocal UIs. The concrete concepts shown in Fig. 3 map to their abstract counterparts in Section 3.1. For example, Callout, TextInput, and Marking are concrete implementations of the explanation, demonstration, and emphasis AbstractOperations, respectively. Classes representing the related abstract concepts (Fig. 2) are also presented in Fig. 3 (colored in gray) to show the relationship between CHAIN‟s abstract and concrete concepts. Each CHAIN Document comprises a help Scenario composed of several Scenes. A Scenario represents contextual help that is presented to end-users for assisting
ACCEPTED MANUSCRIPT 10
Pierre A. Akiki / The Journal of Systems and Software
Listing 1. An Example Excerpt of a Help Scenario Represented as a CHAINXML Document 1. Enter the maximum purchases that this customer can make on credit. Entrez les achats maximales que ce client peut faire à crédit.
17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40.
PT
ED
M
AN US
CR IP T
2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.
AC
CE
41.
Fig. 6. An Example Excerpt of a Help Scenario Represented using CHAIN’s Visual Notation
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
11
(<>). Hence, UserInput operations such as UserKeyPress are defined within a help Scenario and are triggered before or after certain Scenes.
4 LANGUAGE AND VISUAL NOTATION
This section presents an XML-compliant language and a visual notation that can be used for defining CHAIN help models based on the concepts presented in Section 3.
M
AN US
Animations have been used with UIs in a variety of ways. Some researchers integrated animations as part of help systems [63], while others used animated transitions for conveying UI adaptations to end-users [64]. Animated Characters can in some applications enhance the end-users‟ interaction with the help system by serving as a façade to SynthehizedSpeech and Callouts. For example, the Companions system uses a human-like avatar to make conversations between end-users and applications seem more natural [65]. CHAIN supports the inclusion of animated Characters, which can be moved around the UI or simply remain fixed in a corner. Magnification can be used to make part of the UI more apparent, by zooming in on a particular point or a path. Each CHAIN document has a ResourceDictionary (Fig. 3). The possible entries for this dictionary are depicted by the class diagram presented in
CR IP T
Fig. 7. Implementation Architecture for Supporting the Development of End-User Help Using CHAIN (It is Also Possible to Export CHAINXML Documents from the Database to be used as Local Files)
AC
CE
PT
ED
Fig. 4. These resources include: Styles, Images, and MultilingualTexts, which are defined at design-time and may get updated at runtime. Additionally, these resources include collections of DynamicElements that are added at runtime. A Style can be either defined individually per SceneElement or globally in the ResourceDictionary to be shared among multiple elements. Multilingual Texts are used for on-screen display and for SpeechSynthesis. Having multilingual capabilities in a single help scenario is an advantage over approaches that create different help artifacts for each human language. Images can help in pointing out the use of some help model elements like indicating the type of a Callout. DynamicElementCollections are needed for applications that create DynamicElements. For example, a help Scenario might demonstrate how a modelling tool is used to draw shapes. The abstract concepts that were explained in Section 3.1 included elements that represent user input and responses to that input. Two concrete implementations of these abstract concepts are shown in Fig. 5. Classes representing the related abstract concepts (from Fig. 2) are also presented in Fig. 5 (colored in gray) to show the relationship between CHAIN‟s abstract user inputs and responses to user inputs and their concrete counterparts. A UserKeyPress indicates that a response is triggered when an end-user presses a certain key. The UserKeyPress is linked to one ConcreteUIElement or an entire UI. As a response to a UserKeyPress input, a Scene can be presented to provide an explanation. The UserKeyPress class inherits from UserInput, which is part of a Scenario
4.1 Language The Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) is an XML-compliant language for expressing help models. An excerpt of a document defined using this language and pertaining to the scenario illustrated in Fig. 1 is presented in Listing 1. A number of XML-style tags are supported for defining end-user help based on the concepts shown by the class diagrams in Fig. 2 to Fig. 5. The supported tags are summarized in the Appendix. 4.2 Visual Notation An example of a help model expressed using CHAIN‟s visual notation is presented in Fig. 6. This notation can be used to specify help models without having to explicitly write CHAINXML tags. The “cognitive dimensions” framework [66] was taken into consideration when designing CHAIN‟s visual notation as summarized in the following paragraphs. CHAIN‟s visual notation uses rectangles to consistently represent help model elements. In terms of diffuseness, each meaning (e.g., SceneElement) is expressed using one symbol and a text description. Hence, the notation is terse enough to represent large help models on the screen thereby improving visibility. The example shown in Fig. 6 is detailed and expanded, but zooming out reduces the height of the rectangles. As shown in Fig. 3, a SceneElement is applied to a ConcreteUIElement and thereby depends on it. In order to avoid hidden dependencies, the related ConcreteUIElment is highlighted in the supporting tool when a SceneElement is selected. The relation between SceneElements and ConcreteUIElements does not force a premature commitment since the SceneElements can be specified first and then linked to the UI. This way an early stage help model can be independent from the UI design. Progressive evaluation can be done by executing a help model even if it is not fully complete. Hence, novice designers and developers can evaluate their progress at frequent intervals. The viscosity of CHAIN‟s visual notation is low, since very little
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
Listing 2. CHAINXML Presenter - Speech Synthesis Code (Excerpt) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
//Get properties from CHAINXML document function SESynthesizedSpeech(XmlData) { this.ReadPropertiesFromXML(XmlData); } //Synthesize text using properties read from CHAINXML doc. SESynthesizedSpeech.prototype.Present = function () { var utterance = new SpeechSynthesisUtterance(); SetUtterProperties(utterance, this.Voice, this.Volume, this.Rate, this.Pitch, this.Text, this.Language); utterance.onend = function (event) { //Notify the scene that the text has been synthesized Scene.ElementPresentedSuccessfully(IndexInScene); }; window.speechSynthesis.speak(utterance); } //Cancel speech synthesis while in operation SESynthesizedSpeech.prototype.End = function () { window.speechSynthesis.cancel(); }
5 IMPLEMENTING HELP USING CHAIN
M
This section explains how CHAIN help models can be implemented based on the architecture shown in Fig. 7 and how these models are validated and fixed.
CE
PT
ED
5.1 Developing Help Models As depicted by the architecture in Fig. 7, CHAIN help models are stored in a relational database and can be accessed by software applications through a web-service. The CHAINXML Converter converts help models from relational data into CHAINXML documents. These documents contain XML-style tags that are based on the concepts discussed in Section 3. An excerpt of a CHAINXML document pertaining to the scenario illustrated in Fig. 1 is presented in Listing 1. A CHAINXML Presenter dynamically interprets and presents CHAINXML documents on the client-side as interactive contextual-help. Help models are technologyindependent. A presenter is dependent on a particular presentation technology such as: HTML, WPF, Swing, and so on. Nonetheless, it can be used across different software applications that adopt the same presentation technology. A Caching Engine caches CHAINXML documents on the client-side in order to improve performance, in case an end-user requested a help scenario more than once. To test CHAIN, different CHAINXML Presenters were developed using JavaScript for the web, C# with the .Net Framework for the desktop, and C# with Xamarin [67] for the mobile. The code excerpt shown in Listing 2 is taken from the JavaScript version of the presenter and is responsible for SpeechSynthesis. The code in Listing 3 shows how a CHAIN scenario is loaded and presented simply by calling a web-service using the Service Access API. The code excerpt shown in Listing 2 provides an
AC
1. function PresentScenario(ScenarioID) { 2. //Load a CHAINXML document from web-service using API 3. var doc = LoadChainXMLScenario(ScenarioID); 4. //Call CHAINXML Presenter to present the document 5. PresentChainXMLScenario(doc); 6. }
Listing 4. Load a Single CHAIN Scene to Explain Part of the UI When the End-User Requests Help by Pressing a Key (Excerpt) 1. 2. 3. 4. 5. 6. 7.
$("#UIElementID").keydown(function (e) { if (KeyCode == Code) //e.g., 112 indicates F1 Pressed { PresentScene(ScenarioID, SceneID); return false; //do not propagate the key press } });
example of how the presentation is done. The code shown in Listing 4 adds an event handler for the “keydown” event of a ConcreteUIElement, in case a UserKeyPress input is specified with a PresentScene response. Such event handlers are automatically added by the CHAINXML Presenter. The CHAINXML Presenter targeting HTML leverages the speech synthesis capabilities of HTML5 (Listing 2), which are supported by Google Chrome. The other versions of the presenter synthesize speech using the libraries provided by C#. Microsoft Agent (Double Agent) [68] is used to present Animated Characters, but CHAIN is not restricted to this technology.
AN US
effort is required to change a SceneElement. Developers can simply add a new element from the toolbox and delete the old one. The supporting tool also allows developers to right-click on an element and switch its type, thereby preserving common properties between the old type and the new one. Help models are stored in a database, thus it is possible to perform global changes across all models by writing queries. Packaging these queries into reusable transactions makes them executable without having to write code each time a change is needed.
Listing 3. Load and Present Complete CHAIN Scenario (Excerpt)
CR IP T
12
5.2 Transforming Help Models at Different Levels of Abstraction and Mapping these Models to the UI When multiple levels of abstraction are used in modeldriven development, transformations are applied to map a model on one level of abstraction to its counterpart on another level. For example, in model-driven UI development, transformations are used to perform mappings between models on the different levels of abstraction (task model, AUI, CUI, and FUI) defined by the Cameleon Reference Framework. The approach presented in this paper also applies transformations between the help models represented at different levels of abstraction. Abstract help models (corresponds to AUI) are transformed to concrete help models (corresponds to CUI), and the latter are transformed to working contextual help (corresponds to FUI) that is presented at runtime over the UI. Transformations between abstract and concrete help models are done using two techniques that are already supported by Cedar Studio for UI models. The first technique involves the definition of transformation rules through a visual mapping between the abstract and concrete help model elements. Cedar Studio shows a list of all the elements in a model and their possible counterparts on the other level of abstraction. This list can be used to map each element to the one that mostly fits a particular help scenario. For example, an Abstract Help Operation can be transformed to different types of Scene Elements such as: TextInput, Callout, SynthesizedSpeech, and so on. The second technique involves transforming one level of abstraction to another by defining scripts. Cedar Studio currently supports Iron Python, but it is
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Declare @SceneElemID int, @SceneElemType int Declare HMSECursor Cursor For --Get cursor movement scene elements GetSceneElements(@@HelpModelID, 'MouseMovement') Open HMSECursor --Iterate around the scene elements Fetch Next From HMSECursor Into @SceneElemID,@SceneElemType While @@FETCH_STATUS = 0 Begin --If previous element is a marking add both elements to result Select @PrevElemID = GetPreviousElement(@SceneElemID,'Marking') if @PrevElemID is not null Insert Into #ErroneousSceneElements Values(@@HelpModelID, @SceneElemID, @PrevElemID) Fetch Next From HMSECursor Into @SceneElemID,@SceneElemType End Close HMSECursor Deallocate HMSECursor Select * From #ErroneousSceneElements
Declare @SortOrder1 int, @SortOrder2 int Set @SortOrder1 = GetSortOrder(@@SceneElement1ID) Set @SortOrder2 = GetSortOrder(@@SceneElement2ID) /*Swap the sort orders to move the ‘Hightlight’ scene element after the ‘Cursor Movement’ one*/ Update SceneElements Set SortOrder = @SortOrder2 Where SceneElementID = @@SceneElement1ID and HelpModelID = @@HelpModelID Update SceneElements Set SortOrder = @SortOrder1 Where SceneElementID = @@SceneElement2ID and HelpModelID = @@HelpModelID
are with constraint languages such as OCL. In a previous work, SQL was used to perform similar validation on UI models [10]. The CHAINXML Validator, shown in Fig. 7, can be invoked from Cedar Studio to validate and fix help models. Syntactic errors can be avoided by using the provided visual development tool. However, different types of semantic errors of varying complexity can be validated. The validation can detect simple errors such as forgetting to assign a property, e.g., text, on a help model element. It can also detect more complex errors pertaining to the illogical ordering or use of certain elements within a scene. The following are two examples of validation operations that can be applied to help models: 1. SceneElements associated with UI elements that were deleted due to changes in requirements, e.g., a marking that is associated with a deleted text box 2. Illogical ordering of SceneElements, e.g., highlighting a text box before moving the cursor to it An excerpt of the SQL code that is used for realizing example 2 is presented in Listing 5. Validation SQL queries enable Cedar Studio to display errors and warnings based on the retrieved erroneous SceneElements. In case the validation needs to query the UI representation, which is the case of example 1, a model-driven UI representation is needed. UIs that are not model-driven can be reverse-engineered as shown in a previous work [70]. The reverse-engineering process is capable of generating complete UI models that can be viewed and modified using Cedar Studio. Help models can be developed for these UI models, which are dynamically presented to the end-users as working UIs. SQL queries can also be used for fixing errors by modifying help models. The SQL query presented in Listing 6 fixes the erroneous SceneElements retrieved by the validation query presented in Listing 5. In case the validation query detected errors, the developers can decide whether or not to automatically fix these errors by executing the related model fixing query. Although the validation and fixing technique is not directly related to runtime adaptation, it can circumvent human errors that might occur when help models are being developed and maintained.
AC
CE
PT
ED
M
AN US
possible to add support for other scripting languages or dedicated transformation languages in the future. The CHAINXML Presenters discussed in Section 5.1 are used to interpret the concrete help models and transform them to running contextual help. The mapping between the help models and the UI, depends on the nature of the latter. In case the UI was defined using Cedar Studio, then it follows the modeldriven approach that was presented in previous work [69]. In this case, the mapping is done between each help model element and its corresponding UI element at the appropriate level of abstraction. The abstract help model elements are mapped to their corresponding AUI element. As shown in Fig. 2, there is a one to many relationship connecting the AbstractUIElement to the AbstractHelpOperation. The mapping model in this case maps the instances of the participating classes using their ElementIDs. The concrete help model elements are mapped to their corresponding counterparts on the CUI. As shown in Fig. 3, there is a one to many relationship connecting the ConcreteUIElement to the SceneElement. The mapping model in this case also maps the elements based on their identifiers. Following the Cedar Architecture [69], UI models are interpreted, adapted, and presented at runtime. The help models presented in this paper follow a similar interpreted runtime model-driven approach and are presented by the CHAINXML Presenters over the FUI in order for the help to adapt with the UI. In case the UI was not model-driven, then the CHAINXML Presenters directly map the concrete help models to the FUI. There several advantages for adopting a model-driven approach for both the UI and its help. These advantages include: technology independence, supporting multiple modalities, and being able to validate whether a help model still conforms to a UI as the latter is modified due to changes in requirements.
Listing 6. Example of SQL-based Help Model Fix (Excerpt) Fixing Illogical Ordering by Swapping the Sort Order of a Marking and a Mouse Movement Scene-Element Pair
CR IP T
Listing 5. Example of SQL-based Help Model Validation (Excerpt) Illogical Ordering – Highlighting an Element before Moving the Cursor to it
13
5.3 Validating and Fixing Help Models Performing validation is necessary for uncovering errors, which might exist in help models due to human mistakes or changes that have occurred to the UI over time. As shown by the architecture in Fig. 7, the help models are stored in a relational database. This design choice makes it is feasible to perform model validation using SQL queries, which embody logic for validating if a help model has errors and retrieving these errors as relational data. The main reason for using SQL in this case, is that developers are generally more familiar with it than they
6 MAINTAINING HELP USEFULNESS AS UIS ADAPT This section provides an explanation with a few examples on how CHAIN help remains useful for end-users as UIs adapt. This explanation covers how the CHAINXML Pre-
ACCEPTED MANUSCRIPT 14
Pierre A. Akiki / The Journal of Systems and Software
(b) Customer UI Adapted Version (V2)
AN US
CR IP T
(a) Customer UI Initial Version (V1)
(c) Examples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
M
Date of Birth
The UI element is hidden. Hence, the help is not shown.
ED
Billing Cycle
Adapted Version (V2)
PT
Notes Formatting Toolbar
CE
The UI element is hidden. Hence, the help is not shown.
Fig. 8. Examples of UI Adaptation Operations that can be Accommodated by CHAIN Help Models while Remaining Useful
AC
senter presents the help over initial and adapted UIs and the abilities of CHAIN help to work with adaptive UI. UIs can adapt in different ways. Feature-set minimization and layout optimization are two types of UI adaptation [10]. The first type removes or disables UI elements that are not required in a particular context-of-use, while the second type adapts UI element properties such as size and grouping. The UI models are kept alive at runtime in order to perform the adaptation. The UI adaptation approach is comprehensively presented in another paper [69]. As shown by the class diagram in Fig. 3, SceneElements are associated with ConcreteUIElements. The CHAINXML Presenter maintains this association when the UI adapts. It plays a key role in enabling CHAIN help to work with a wide variety of UI adaptations. Presentation technologies support identifiers for UI elements. Two examples are “id” in HTML and “Name” in WPF. The CHAINXML Presenter obtains a ConcreteUIElement by its identifier (UIElementID) and
presents its related SceneElement. Hence, help scenarios are presented in a way that is independent from UI element properties that might change. A number of UI adaptation operations were explored to test the ability of CHAIN help models to adapt as the UI adapts. This exploration was done before conducting evaluation studies with participants. The adaptations were applied to prototype data entry UIs for managing: bank accounts, customers, inventory items, and sales invoices. The UIs were adapted in ways that could potentially occur in real-life enterprise applications. Applying adaptations to prototype UIs allowed the assessment of the help‟s adaptation alongside the UI. This assessment showed that CHAIN help remained useful when various UI adaptation operations were applied. What follows are examples of these adaptation operations and an explanation of how CHAIN help adapts accordingly. Nonetheless, other adaptation operations are also possible. Each operation is related to one or
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
tual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
ED
M
Date of Birth
PT
Billing Cycle
Notes Formatting Toolbar
CE
The UI element is hidden. Hence, the help is not shown.
AC
Fig. 8. The “Notes” text-box in the adapted version of the UI has additional buttons above it showing functionality that could have been accessed in the initial version through a context-menu. The “Date of Birth” example presented in (c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – c, shows a callout displayed next to the date of birth field in the initial UI. This callout is not displayed in the adapted UI, because the date of birth field is hidden. The inverse is shown in the “Notes Formatting Toolbar” example ((c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
AN US
6.1 Hiding or Disabling Widgets Widgets that are not required in a particular context-ofuse can be hidden or disabled in order to reduce UI bloat. For example, in enterprise applications, complex UIs can be simplified by hiding or disabling the fields that are not required by users with a certain role [10]. Another example is hiding unnecessary items from a bloated menu like the one in older versions of Microsoft Word [72]. Furthermore, adapting the accessibility-of-functions and information-density of a UI can also lead to widget hiding and disabling [73]. An example of adapting the accessibility-of-functions is shown in (c) Examples of Contex-
Notes Formatting Toolbar
CR IP T
more of the three context-of-use dimensions: user, platform, and environment. For example, the icons and colors can be adapted based on each end-user‟s culture [51]. Features can be hidden or disabled, for example, based on each end-user‟s job within an enterprise or computer literacy level [10]. The layout grouping and the widgets‟ types, sizes, and positions can be adapted based on the selected platform and each end-user‟s motor abilities [9]. The end-user‟s distance from a screen or mobility within a physical environment can also affect the layout grouping and widget sizes and positions [71].
15
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown. Adapted Version (V2)
Fig. 8. – c), where a callout is displayed over the formatThe UIUI. element is hidden. ting toolbar in the adapted This toolbar is not shown Hence,the thenotes help is notbox shown. in the initial UI, because text has a lower accessibility of functions. Hence, the callout does not appear in the initial UI. However, the formatting toolbar is part of the initial UI, but it is merely hidden. This example shows that hiding elements does not prevent a CHAIN scenario from being properly presented. The CHAINXML Presenter uses UIElementIDs to check whether a ConcreteUIElement was disabled, removed, or made invisible. For example, the JavaScript method “document.getElementById ('id')” and the HTML/CSS property “visibility” can be used to perform checking whether or not a UI element is visible. The CHAINXML Presenter disregards scene elements that are related to hidden or removed UI elements. For example, assume that a TextInput Action SceneElement is specified on a text box that gets hidden or disabled by a UI adaptation. In this case, the CHAINXML Presenter simply moves to the next SceneElement. Similarly, Animation, Creation and Speech, SceneElements are skipped if they are applied to a ConcreteUIElement that gets hidden by a UI adaptation. The Adapted Version (V2) ability of the CHAINXML Presenter to accommodate UI adaptations that hide or disable UI elements, allows it to Thethe UI contextual element is hidden. present end-users with help part that is Hence, theEnd-users help is notwill shown. relevant to the adapted UI. not have to see, for example, an explanation of a UI element that is no longer visible in their version of the UI. Here, the elements are removed at runtime for a particular context-of-use. In case some elements were removed at design-time for all contexts-of-use, validation and fixing can be applied to completely remove the associated help model SceneElements as discussed in Section 5.3.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
6.2 Changing the Layout Grouping Changing the layout grouping is performed for several reasons such as the accommodation of different screen sizes [9]. The UIs shown in Fig. 1 provide an example of a layout optimization adaptation that switches group-boxes with tab-pages. In Fig. 1 – a, the SceneElements are presented directly because all the related ConcreteUIElements are visible. On the other hand, in Fig. 1 – b, the CHAINXML Presenter first brings into view the relevant ConcreteUIElements by selecting their parent tab-page. Then, it presents the associated SceneElements. In some cases, a UI element such as a menu item cannot be located directly on the UI. In such cases, the path property of the ConcreteUIElement is used to indicate the UI elements that have to be loaded before accessing the target UI element. In all cases, when the layout grouping is changed the CHAINXML Presenter only has to locate the widget using its identifier, bring it into view, and present the relevant SceneElement whether its type is Action, Creation, Speech, or Animation.
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI
M
Initial Version (V1)
ED
Date of Birth
The UI element is hidden. Hence, the help is not shown.
CE
Notes Formatting Toolbar
PT
Billing Cycle
AC
Fig. 8, the widget representing the “Billing Cycle” field was adapted from a combo box ((c) Examples of Con-
textual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – b). The “Billing Cycle” example presented in (c)
AN US
6.3 Changing Widget Types One example of this adaptation is changing the type of the data selection widgets to accommodate different enduser motor abilities [9]. In the example shown in (c) Ex-
Fig. 8. – a) to a radio-button group ((c) Examples of
CR IP T
16
The UI element is hidden. Hence, the help is not shown.
Examples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Adapted Version (V2)
The UI element is hidden. Hence, the help is not shown.
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – c, shows that in the initial UI the billing cycle combo box has both a callout displayed next to it and a marking (border) placed around it. We can see that adapting the combo box to a radio-button group, does not prevent the callout and the marking from being displayed. This is possible because both the callout and the marking are displayed with respect to the boundary of the target UI element whether it is a combo box, radio-button group or another type of widget. Adapted Version (V2) When changing a widget‟s type, the assumption is that the widget is being substituted with an alternative The UI element is hidden. that has an equivalent role. For example, a selection Hence,box the can helpbe is not shown. with a widget such as a combo substituted radio-button group or a list box. The post-adaptation type of a widget is identified by the CHAINXML Presenter (e.g., using the “GetType()” method on objects in C#) in order to decide how to present the SceneElements. Assume that ValueChange, Callout, and Marking SceneElements were specified on a combo box that was adapted to a radio-button group. The CHAINXML Presenter is able to present the SceneElements on the adapted widget if its identifier (UIElementID) does not change. First, the
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Callout and Marking are positioned next and around the radio-button group respectively. Then, the radio button representing the value specified by the ValueChange action is selected. A similar example is shown in (c) Ex-
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
17
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown.
Fig. 8 – a, was relocated next to the “Personal Details” group-box as shown in Adapted (c) Examples Version of (V2)Contextual
Help as Shown on the Initial (V1) and Adapted (V2) Customer UI
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Notes Formatting Toolbar
AN US
The UI element is hidden. Hence, the help is not shown.
Billing Cycle
CR IP T
The UI element is hidden. Initial Version (V1) Hence, the help is not shown.
Date of Birth
ED
M
Fig. 8 – c (billing cycle). Assume that a click MouseCommand was set on a checkbox that got adapted to a switch button (two-state checkbox alternative). The CHAINXML Presenter sends the MouseCommand to the new widget in order to switch its state just like the checkbox would have done. Speech SceneElements are presented in the same way for the different widget type. The only constraint is that the speech should, as a best practice, refer to the concept represented by a widget rather than the widget‟s type, e.g., “select a monthly billing cycle” rather than “click the monthly billing cycle radio button”.
PT
6.4 Resizing or Relocating Widgets With layout optimization, widgets can be resized or relocated for many reasons including filling the space that was left after hiding unnecessary widgets. For example, the “Financial Information” group-box, shown in (c) Ex-
CE
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
AC
Date of Birth
Billing Cycle
The UI element is hidden. Hence, the help is not shown.
Fig. 8 – b. Hence, it occupied the space that was created by hiding the “Contact Information” group-box. Also, as shown in (c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
Notes Formatting Toolbar
The UI element Adapted Version (V2)is hidden. Hence, the help is not shown.
The UI element is hidden. Hence, the helpwas is not shown. Fig. 8 – b, the “Notes” group-box resized to occupy the space that was created after relocating the “Financial Information” group-box.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
CR IP T
18
Fig. 9. CHAIN Components Work with Legacy Applications whether the Source Code is Available or Unavailable
Fig. 10. CHAIN Help Models Presented on Existing Desktop and Web Applications (a) OFBiz: the source-code is available, (b) Excel: the source code is not available
AC
CE
PT
ED
M
CHAIN help model Scene Elements are not affected by this type of adaptation, because the presentation is done with respect to the on-screen size and position of the ConcreteUIElements. The CHAINXML Presenter is able to locate a ConcreteUIElement using its UIElementID and thereby determine its post-adaptation location and size. For example, Markings are presented around the new borders of a widget, and Callouts are positioned with respect to the new position (left, right, top, and bottom) of the target widget. After determining the new location and size of a ConcreteUIElement, the CHAINXML Presenter is able to present Actions such as: TextInput and ValueChange (if the UI element type supports these operations), MouseMovement, MouseCommand, and KeyboardCommand. Animation SceneElements, whether CharacterAnimation or Magnification, depend on locations and paths that involve points on the UI. These points could be fixed, e.g., show a character in the corner at point (0, 0). Yet, these points could also depend on the location of certain ConcreteUIElements, e.g., to magnify a specific widget. In the latter case, the relevant UI element(s) is accessed using its identifier by the CHAINXML Presenter in order to obtain the post-adaptation location. Speech SceneElements are not directly affected by the resizing and relocation of UI widgets. Speech can be introduced anywhere in a Scene and is presented in the same way in both initial and adapted UIs. The CHAINXML Presenter just brings the relevant UI part into view, before presenting a speech SceneElement.
6.5 Changing Colors, Icons, and Fonts Adapting a UI‟s colors, icons, and fonts does not always have a direct effect on the way CHAIN Scenarios are pre-
sented. Changing the font-size of ConcreteUIElement might cause its size to change as well. Nonetheless, this change can be accommodated as explained in Section 6.4. Since CHAIN help is contextual, it remains useful when colors and icons are adapted without needing additional measures from the CHAINXML Presenter. On the other hand, static help causes discrepancies in such cases.
6.6 Direct and Indirect UI Adaptations Adaptive UIs can perform context-dependent adaptations upon loading (indirect adaptation) or while the end-user is working (direct adaptation). Performing an indirect adaptation and offering the adapted UI as an alternative can reduce user confusion [72]. However, in some cases, e.g., ubiquitous applications, it may be necessary to perform a direct adaptation while the user is working. For example, MASP detects changes in the environment and adapts the UIs of smart home systems accordingly [71]. Since CHAIN help models are interpreted and presented at runtime, UI adaptation operations such as the ones described in the previous sections can be detected in both direct and indirect adaptations. Hence, the help is presented in accordance with the adapted UI. CHAIN help models do not contain elements that are related to specific UI adaptation operations. This makes CHAIN useful in a wider variety of scenarios.
7 USING CHAIN WITH LEGACY SYSTEMS Olsen notes in his paper on evaluating UI systems research that legacy code can be a barrier to progress; hence rewriting applications is sometimes the price of progress
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
AC
CE
PT
ED
M
AN US
7.1 When there is Access to the Source-Code When developers have access to the source-code of a legacy software system, CHAIN help models can be leveraged by simply adding a few lines of code to integrate CHAIN‟s client components. These components become part of the application as shown in Fig. 9 – a. The ServiceAccess component calls a web-service to load help models, which are then presented by the CHAINXML Presenter. JavaScript examples of the lines-of-code to be added are shown in Listing 3 and Listing 4. The example in Fig. 10 – a shows a web-based and open-source ERP system called Apache Open for Business (OFBiz) [74]. OFBiz uses HTML to represent its UIs. In a previous work [70], the UI adaptation mechanism RBUIS [10] was integrated into OFBiz. RBUIS was used to adapt the “Product Store” UI of OFBiz by reducing its features and changing its layout grouping from collapsible group-boxes to tab-pages. The same CHAIN Scenario was presented over both the initial and adapted versions of the UI. The running Scenario shown in Fig. 10 – a, has Callouts that explain the purpose of the “visual theme” field. This field is presented as a combo box in the initial UI and as a set of radio buttons in the adapted UI. Nonetheless, the CHAINXML Presenter is able to show the field‟s respective Callout underneath it regardless of the adaptation.
7.2 When there is No Access to the Source-Code In some cases, there is a need to create help scenarios for a software system, without having access to its source-code. For example, IT personnel may need to create help Scenarios for commercial software such as enterprise resource planning (ERP) systems adopted by their companies. Another example can be general purpose software tools such as spreadsheet applications and email clients, which are used in many modern offices. An approach was devised for allowing the CHAINXML Presenter to function without requiring access to the target application‟s source-code. This approach uses existing UI automation libraries to support the presentation of CHAIN help models on top of the UIs of both desktop and web application. With UI automation, it is possible for a software application to access and manipulate the UI elements of another application using an Application Programming Interface (API). Hence, this technique can be used to present help models over the UIs of legacy applications. Some UI automation solutions are the outcome of academic research, while others are commercial or open-source products. One example from the literature is Sikuli, which was included in Section 2.3 under pixel analysis [28]. PUMA is another UI automation system that was used for dynamically analyzing mobile applications [75]. As shown in Fig. 9 – b, the Mediator hosts Client-Components and third-party UI automation components. This mediator has a Request Listener that relays help requests from the target applications to the CHAINXML Presenter, which in turn sends commands to the UI automation components. The Mediator and Presenter were developed using C#. The following paragraphs offer more details on how UI automation is used with CHAIN. Microsoft‟s UI Automation framework [76] was used to present help models over the UIs of desktop applications. This framework was chosen due to its compatibility with C# and its advanced capabilities. For example, some UI testing libraries such as TestStack [77] are based on it. The UI Automation framework can retrieve UI information. One example is retrieving a widget‟s position, which is useful for highlighting a widget or displaying callouts next to it. This framework also supports controlling UIs using commands such as mouse-clicks.
CR IP T
[61]. Yet, he also states that it is desirable to provide a new advance while maintaining legacy code. CHAIN aims to achieve the latter, by supporting help models for the UIs of legacy software systems. CHAIN help models can be used with both newly developed and legacy software systems. For the latter, two cases are considered. The first case is when the sourcecode of the legacy software system is available, and the second case is when this source-code is unavailable. The architecture previously shown in Fig. 7 is further detailed in Fig. 9 to show the interaction between the CHAINXML Presenter and the software applications that leverage UI help models. The approach addresses the two previously mentioned cases of source-code availability as explained in Sections 7.1 and 7.2. An example on each case is illustrated in Fig. 10.
19
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
M
AN US
CR IP T
20
ED
Fig. 11. CHAIN Help Model Visual-Design Tool in Cedar Studio
AC
CE
PT
The technique devised for web applications, uses the Selenium WebDriver [33] to access and control HTML pages through the web-browser. It provides access to an HTML page‟s elements and allows the modification of their properties, in addition to the injection and execution of Java Script code on the page. For example, Keyboard and Mouse commands can be sent to particular elements on the page, and Java Script code can be executed to create elements such as Callouts or modify properties such as colors. The need for cross-origin HTTP requests that are blocked due to security constraints is avoided by using the Selenium WebDriver. The SceneElements are presented with respect to ConcreteUIElements that are located using UIElementIDs. When the help models are developed, the UIElementIDs are specified using the same values found in the UIs. These IDs are the UI automation IDs in desktop applications and HTML IDs in web applications. The automation IDs and HTML element IDs are the primary choices since they uniquely identify a UIElement. Yet, other properties, e.g., text and widget type, can serve as plausible Alternative Identifiers (UIElement class in Fig. 3) in case IDs were not specified. The example in Fig. 10 – b shows CHAIN help presented on Excel‟s ribbon UI, which collapses and adapts its shape when the window is resized. The help is demonstrating how Excel‟s “Scenario Manager” tool is accessed. The same help model is used for both adapta-
tions, when the ribbon is expanded or collapsed.
8 TOOL SUPPORT A tool was developed to support technical stakeholders, e.g., developers and IT personnel, in defining and maintaining CHAIN help models. This tool provides a visual canvas for designing help models using CHAIN‟s visual notation. It can also generate CHAIN help models from macro-like activity recordings, which are done on the running UI using the Selenium IDE [33]. These two features complement each other and facilitate the process of developing help models.
8.1 CHAIN’s Visual Design Tool Cedar Studio is an IDE for developing adaptive modeldriven UIs [12]. It provides visual-design tools for defining UIs on different levels of abstraction including: task models, abstract UI (AUI), and concrete UI (CUI). It also supports visual-design and code editing tools for defining adaptive behavior, which can be executed at runtime to adapt UIs according to the context-of-use. In this paper, Cedar Studio was extended by adding the help model visual-design tool shown in Fig. 11. This tool supports the visual design of help models, which embody the CHAIN concepts presented in Section 3. By using the visual representation, help models can be managed more easily without writing CHAINXML tags
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
21
(b) Selenium IDE Recording Tool
(c) Selenium IDE Recorded File (HTML Format)
(d) CHAIN Help Model (Can be Modified)
11. 12. select 13. id=cboGender 14. label=Female 15. Import to 16. Cedar Studio 17. click 18. id=btnSubmit 19.
AN US
1. 2. type 3. id=txtFirstName 4. Jane 5. 6. 7. type 8. id=txtLastName 9. Smith 10.
CR IP T
(a) Personal Information UI (Basic Example)
Fig. 12. End-Users can Record their Activities on a UI Using the Selenium IDE (a, b) and then Save the Result (c), which is Automatically Converted to a CHAIN Help Model (d) Using Cedar Studio. The CHAIN Help Model can be modified by Adding More Elements.
AC
CE
PT
ED
M
(e.g., Listing 1). As explained in Section 3, CHAIN supports the representation of help models at two levels of abstraction: abstract (Fig. 2) and concrete (Fig. 3). When designing UIs, it would be better if a designer could manipulate a concrete object rather than its abstraction [78]. Cedar Studio already supports both a bottom up and a top down approach for UI design. Other UI design tools, e.g., Damask [79], support a bottom-up UI design approach. Hence, developers can design the concrete UI, and the abstract UI is generated in the background. This notion also applies to help models and is thereby supported by Cedar Studio. The help model design tool offered by Cedar Studio uses a canvas (Fig. 11 - 5), which visualizes CHAIN Scenes as columns (Fig. 11 - 2) containing rectangles representing SceneElements such as: Callout, Marking, Speech, and so on. Each SceneElement has an icon indicating its type. Icons represent concrete objects (SceneElement) and make diagrams more understandable [80]. Scenes and SceneElements can be added as needed from the toolbox (Fig. 11 - 1). Both Scenes and SceneElements can be dragged and dropped in order to change their order, and their properties can be modified using a property box (Fig. 11 - 6). Cedar Studio‟s visual paradigm is familiar to developers since it is similar to the one offered by commercial IDEs, e.g., Visual Studio and Eclipse. As shown in Fig. 11, Cedar Studio offers a split interface that shows both the concrete UI (Fig. 11 - 3) and help
model design tool (Fig. 11 - 5). This split interface helps in associating UI elements with SceneElements. One example is associating a Callout with a text box. It is possible to define a validation rule to check whether or not a SceneElement is associated with an adequate type of ConcreteUIElement, by using the approach discussed in Section 5.3. For example, it would not be appropriate to associate a TextInput SceneElement with a ConcreteUIElement that does not support typing. In the future, this validation rule could be integrated directly into the visual design tool to prevent non-adequate associations before they occur. This could be done by giving a visual indicator over the ConcreteUIElements that are adequate to choose from, when a SceneElement is selected. An error message could be shown if a wrong choice is made. Help models can be executed and tested against their corresponding UI (Fig. 11 - 4). Cedar Studio interprets and presents both the UI and the help model at runtime.
8.2 Recording CHAIN Scenarios A preliminary evaluation of CHAIN, which is discussed later in the paper, showed that developers would like to use Cedar Studio‟s help model design tool in combination with a recording tool. The latter produces a usage scenario by recording a task as it is being performed on the UI. They thought that the recording tool helps them in quickly creating an initial help model, which they can later adjust using the design tool.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
CR IP T
22
Fig. 13. Participants’ Experience in the Software Industry
Fig. 14. Time taken by the Participants to Complete Two Tasks Each of which Required them to Define a CHAIN Help Model
*13 Participants, 8 males and 5 females, aged 21 to 30
*There is a noticeable improvement between the Task 1 and Task 2
Fig. 16. Participants’ Preference between Using Cedar Studio to Create Help Models from Scratch or Using it in Combination with a Recording Tool that is Mainly Intended for End-Users
CE
PT
ED
M
AN US
* (n) is the number of participants in each level of experience
Fig. 15. Hand-Drawn Sketch of CHAIN Model (Excerpt) *The icons are annotated to indicate the concepts that they represent in CHAIN’s metal-model.
AC
An existing tool called Selenium IDE [33] can record and playback activities on web-application UIs. This tool can be used to record activities and save them to a file in HTML format as illustrated in Fig. 12 (a, b, and c). An extension was added to Cedar Studio to support importing a Selenium IDE recording and automatically converting it to a CHAIN help model such as the one shown in Fig. 12 - d. This help model can then be modified in Cedar Studio by adding more elements, which provide further explanation on how to use the UI. This possibility overcomes some of the limitations of macro recorders that are mentioned in Section 2.4. It is also possible to specify general settings to configure the conversion process. For example, the general settings can be used to specify if UI elements should be highlighted when clicked, the default color for highlighting UI elements, and so on. In the example shown in Fig. 12 – d, Cedar Studio can automatically generate MouseCommand (click)
Fig. 17. Participants’ Preference between Using Basic Shapes (e.g., circle, triangle, and so on) to Represent Help Model Elements When Conducting Manual Sketches (e.g., on paper or whiteboard), or Using the Same Symbols (Icons) Used in Cedar Studio
and Marking (highlight) elements before the TextInput elements, based on the configuration specified in the general settings. This type of configuration reduces even further the number of elements that are required to make a help model complete. The example shown in Fig. 12 was kept simple for space considerations, but the tools are not limited to simple examples.
9 PRELIMINARY EVALUATION A focus group was held to perform a preliminary evaluation of CHAIN‟s visual notation. Assessing the visual notation is not directly related to CHAIN‟s primary objective, which is adaptive help. Nonetheless, conducting this focus group was essential to obtain input that can be used for tuning CHAIN if necessary, before assessing its technical capabilities in another study.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
CR IP T
Fig. 13 show the general experience of the participants, in addition to their experience in developing UIs and help materials. Their collective experience in the software industry included developing web, desktop, and mobile line-of-business applications related to: banking, retail, supply chain management, and so on. The participants had also developed UIs using different presentation technologies such as: HTML/CSS, Windows Forms, Oracle Forms, and so on. As for developing help materials, the participants who had done work in this area mentioned that they have used different types of tools. Some had used non-specialized help development tools such as Microsoft Word and PowerPoint for developing guidelines and tutorials. Others mentioned using specialized tools such as Snagit and Microsoft Steps Recorder for developing screenshot-based help materials.
9.3 Tasks Given to the Participants The participants were asked to create two help models using the visual notation shown in Fig. 6. They were asked to base these models on snapshots that depict what the outcome would look like if the help was presented over the UIs. The help models were sketched on papers as shown in Fig. 15. Although Cedar Studio was presented during the focus group session, sketching the help models on paper was enough for this preliminary evaluation. The first help model that the participants were asked to create represents the use of a few features from the “Item Maintenance” UI in the Great Plains ERP system [85]. The second model represents launching the scenario manager tool in Microsoft Excel (Fig. 10 – b). Each help model contains three different Scenes, and each of these Scenes contains 2 to 5 elements.
AC
CE
PT
ED
M
AN US
9.1 Study Design Focus groups are carefully planned discussions with a relatively small number of participants. Purposive sampling is usually used to select participants based on their individual characteristics as related to the session topic [81]. The main characteristic that was considered when recruiting participants for this focus group is their expertise in the software industry. The focus group setting enables participants to provide rich information and build on the ideas of others [82]. Focus groups are usually used to collect qualitative data. In this focus group, the discussion was guided to yield qualitative data, and the participants were also asked to answer a few questions that provide quantitative data. The combination of both types of data offers a richer outcome. The session was video-recorded and analyzed later. The focus group session took approximately one hour. In the week prior to this session, the students were introduced to model-driven UI development in a three hour session. Before running the focus group, the participants received an explanation on the different types of CHAIN help model elements. The participants were asked how they would like to approach the construction of a CHAIN help model. One option is to use Cedar Studio‟s visual-design tool, shown in Fig. 11, for building the model from scratch. Another option is to use a recording tool (e.g., Section 8.2) to create an initial help model, which can be extended with more elements using Cedar Studio. The participants were also asked if it would be easier for them to use basic shapes such as squares and circles to represent help model elements when drawing manual sketches. This question assessed the understandability of CHAIN‟s help model element symbols and the ease of drawing them manually if sketching was needed for having reflective conversations (e.g., [83], [84]). The participants were asked to sketch two CHAIN help models, and the completion time was recorded. This task familiarized participants with CHAIN before answering the two previously mentioned questions. It also provided an initial indication about the simplicity and learnability of CHAIN‟s visual notation. The participants had not previously attempted defining help models using CHAIN before being asked to sketch the two help models during this focus group.
23
9.2 Participants The participants in this focus group were 13 Master‟s Computer Science students enrolled in a graduate HCI course at Notre Dame University – Louaize. Out of the 13 participants, 12 are working professionally in the software industry. Hence, these participants are practitioners who hold a Bachelor of Science in Computer Science and are working in the software industry while pursuing their Master degree. The data in
9.4 Primary Results The primary results of this focus group cover the three points discussed in Section 9.1. Task Completion: As shown in Fig. 14, the participants completed tasks 1 and 2 in a mean time of 10.77 (SE=0.95, SD=3.44) and 5.46 (SE=0.47, SD=1.71) minutes respectively. The improvement in the task completion time, between the first task and the second one, can be attributed to the learning curve that was higher in the first task. Eleven out of the thirteen participants (85%) were able to complete the tasks without any mistakes. Two of the participants forgot to add a marking in one case and added a UI Window element in another case when it was not required. Some participants were slightly confused in the beginning between explicitly specifying that a UI element should be highlighted, and simply specifying that the mouse should move to a UI element. This confusion happened with a menu item that can natively highlight itself when the mouse hovers over it. Considering that none of the participants had prior knowledge of CHAIN, these results provide a positive preliminary indication about the simplicity and learnability of CHAIN‟s visual notation.
ACCEPTED MANUSCRIPT 24
Pierre A. Akiki / The Journal of Systems and Software
M
AN US
CR IP T
Desktop Version of Diet Plan User Interface
Fig. 18. Example from the Second Study Showing a Desktop User Interface with Contextual Help
AC
CE
PT
ED
Design Approach: Concerning the approach for designing CHAIN help models, the pie-chart in Fig. 16 shows that 70% (9 out of 13) of the participants answered that they would generally prefer to combine a recording tool with Cedar Studio, because building an initial help model by carrying out a task that is recorded automatically could be faster than manual construction. The other 30% (4 out of 13) of the participants who said that they would like to start by defining the models using Cedar Studio, thought that it is easier to think from the beginning about dividing the help models into Scenes and SceneElements. They also noted that the simplicity of CHAIN‟s visual notation and Cedar Studio‟s visual-design tool would facilitate their work. Visualization: The participants thought that in some cases it could be useful to draw manual sketches for having reflective conversations on what the help model should contain. The pie-chart in Fig. 17 shows that 70% (9 out of 13) of the participants prefer to use the proposed icons rather than basic shapes. They thought that the icons were not hard to identify and draw by hand. They also thought that even though it is simpler to draw the basic shapes, it is easier to mentally associate an icon with a help model element. For example, it is easier to mentally associate a loudspeaker rather than a circle with SynthesizedSpeech. Hence, it is possible to say that the symbols representing CHAIN‟s help model elements are semantically immediate indicating that a novice reader can infer their meaning simply from their appearance [86]. Furthermore, some of the participants (part of the 30%) who initially said that they would pre-
fer to use basic shapes when manually sketching a help model, re-agreed with their colleagues after hearing their rationale about how the icons are easier to remember. In case the icons were not semantically immediate, a text tooltip would have to be added to explain each one. One such case from the literature is the temporal operators of the ConcurTaskTrees [87].
9.5 Suggestions Given by the Participants Some suggestions were given by the participants regarding adding features to Cedar Studio. One participant suggested that it could be clearer if the visual-design tool eliminates the space between the SceneElements that are going to be presented over the UI in parallel. This way the developer can mentally associate these elements with each other. Another participant suggested supporting filtering (search) the SceneElements of a particular Scenario. A third participant suggested supporting the ability to define templates for a Scene. This way, developers can work faster by dragging a Scene template containing predefined Scene Elements. Then, they can modify it as necessary. For example, a template for a Scene that explains the purpose of a UI field can contain: MouseMove, MouseCommand (click), TextInput, Marking (highlight), Callout, and SpeechSynthesis. These suggestions are useful for improving CHAIN‟s tool support and were thereby added to Cedar Studio. The suggestion on adding templates proved to be particularly useful in the study that evaluated CHAIN in actual use and is explained in Section 10.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
25
PT
ED
M
AN US
CR IP T
Mobile Version of Diet Plan User Interface
CE
Fig. 19. Example from the Second Study Showing a Mobile User Interface with Contextual Help (adaptation of the desktop UI)
AC
9.6 Threats to Validity and Limitations The participant sample size (n=13) taken in this study might not be sufficient for claiming the generalizability of the results. Nonetheless, this sample size is considered typical for a focus group study that is intended to provide an indication about a certain product. The tasks that were given to the participants were sketched on paper. Yet, this was enough for getting insights from this preliminary study. A second study was conducted with developers who used Cedar Studio to carry out more detailed tasks. The results of this study included some quantitative data, which are presented in Fig. 14, Fig. 16, and Fig. 17. However, most of the results were qualitative. This is considered normal for a focus group. Nonetheless, other studies can be conducted in the future in order to elicit more quantitative data. Future studies could involve more participants in order to improve the generalizability of the results.
10 EVALUATION THROUGH ACTUAL USE Another study was conducted to evaluate CHAIN by acquiring feedback from software developers who were asked to develop CHAIN help models for real-life adaptive UIs.
10.1 Study Design This study primarily focused on asking developers to evaluate CHAIN‟s strengths and shortcomings after using it to develop help for two sample UIs from a nutrition application that automates dietitians‟ clinics. Each of the sample UIs has two different adaptations, one for the desktop and another for the mobile. The desktop UIs run on Windows using C# and the .Net Framework, while the mobile UIs run on Android using C# and Xamarin. CHAIN was used to create help for the UIs. One help model was developed for each of the two sample UIs, and was used across a UI‟s various adaptations. For example,
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
10.4 Lessons Learned The following are the main strengths of CHAIN that the participants observed and mentioned. The participants thought that the overall approach for creating UI help with CHAIN is simple enough to learn and use. They valued the possibility of composing help by dragging predefined concepts and templates from the toolbox. In some other tools, e.g., video editors, concepts like callouts and markings are created by putting together primitives such as shapes and labels. The ability to develop a single help model that can be presented over multiple adaptations of the same UI was perceived as a main strong point in CHAIN. The participants observed that this feature significantly reduces development time. For example, if video recording was used, multiple videos would have to be recorded. Then, these videos would be imported into a video editing tool like “Adobe After Effects”, in order to be edited and extended with overlays, e.g., callouts. Additionally, a video requires rendering in order to be previewed. This activity takes more time than running the UI from Cedar Studio, which can instantly present it with the respective help model. The participants also perceived that having one model provides better maintainability since changes are only done in one place. The participants appreciated the ability to record UI actions that can be used to automatically generate help models, because this feature also significantly reduces the development time. The appreciation of this feature complies with the result that was obtained in the preliminary study presented in Section 9. In that study, the participants stated that they prefer to create help models by using Cedar Studio in combination with a tool that records activities performed on a UI (refer to Fig. 16). In addition to showing some of CHAIN‟s strong points, this study produced insights, based on the participants‟ feedback, on how to improve CHAIN and Cedar Studio. The following paragraphs explain these insights and offer suggestions on how each point could be potentially addressed. Accessing part of a widget that does not have an ID: CHAIN works on the basis of locating widgets using their identifiers (IDs). However, in some cases there is a need to locate one or more widget parts that do not have IDs. One example is having to explain data presented by widgets such as charts and progress bars. In this case, the part of the widget that shows a particular datum of interest has to be highlighted and explained. Another example is that of multi-record data input using a data grid widget. In this case, the data-grid rows and cells are examples of parts of the data-grid that do not have IDs. A potential solution for this limitation is enabling more communication between the CHAINXML Presenter and the toolkit being used to render the UI. Hence, the parent widget, e.g., data grid, is accessed using its ID and used to access its child UI parts that are associated with help model elements. This can be done by enabling the CHAINXML Presenter to invoke methods on the parent widget that would return the desired child UI parts. For example, a function on the data-grid widget can be called to retrieve a row or a cell. Since different versions of the CHAINXML Presenter are used for each presenta-
AN US
the diet plan UI has two adaptations, namely desktop and mobile, but only requires one help model. The desktop and mobile adaptations of one of the sample UIs (diet plan) that were used in this study are shown in Fig. 18 and Fig. 19 respectively. The diet plan UI allows dietitians to plan the meals that their patients would consume if they were following a particular diet. The help model is presented all at once on top of the UIs in Fig. 18 and Fig. 19 in order to conserve space in the paper. Nonetheless, the Scenes can also be presented sequentially in order to chronologically demonstrate the steps to the end-users. An indirect mode of adaptation (refer to Section 6.6) was used in these UIs. The participants used Cedar Studio to develop and run the help models. These models were developed for the UIs with the least constraints (e.g., a complete set of features). In this study, the desktop versions of the UIs had a full feature-set, while the mobile adaptations had less features. Hence, the help models were developed for the desktop UIs and were used with both the desktop and mobile UI versions. Since the same help models are used for both platforms, the presenter ignores the help model elements that do not apply on the platform with more constraints. For example, a Mouse Movement is ignored on the mobile. Also, certain elements are presented based on what each platform supports. For example, a left button Mouse Command is simply translated to a tap on the mobile.
CR IP T
26
AC
CE
PT
ED
M
10.2 Participants Six software developers participated in this study. The participants included two females and four males, with ages ranging between 23 and 28 years. Two of these participants are doing freelance development on the nutrition software application from which the sample UIs were selected. The other participants are employed at software companies. The work experience of the participants in the software industry has a mean of five years (SE=0.73, SD=1.78) and collectively included developing desktop, web, and mobile line-of-business applications, and mobile games. The participants had also developed UIs using different technologies such as: HTML/CSS, Windows Forms, Windows Presentation Foundation (WPF), and Unity. The participants had also previously developed video tutorials for software applications using video recording and editing tools, primarily “Debut Video Capture” [88] and “Adobe After Effects” [89]. 10.3 Tasks Given to the Participants Each participant completed this study in around one and a half hours. During this time CHAIN and Cedar Studio were introduced and each participant was asked to develop a help scenario for each of the two sample UIs that were selected from the nutrition application. After developing the help models, participants were asked to provide feedback on what they thought were the main strengths of CHAIN and the points that can be improved in this approach and its supporting tool.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
CR IP T
rently done by an external tool (Selenium IDE), in Cedar Studio for improving productivity. Currently, developers run and test UIs and their related help models from Cedar Studio. Developers run a UI in the browser and use Selenium to record activities, which they can save to a file and import it to Cedar Studio. If a tool like Selenium was incorporated in Cedar Studio, the UI actions would get immediately transformed to a help model without the intermediate steps of saving and importing. Although this is a minor feature, incorporating scenario recording in Cedar Studio could reduce the development time to some extent.
10.5 Threats to Validity and Limitations This study only involved six participants who mostly provided qualitative data. This is not sufficient for claiming a statistically significant result. However, the nature of the provided data was quite rich. Since the participants were software developers, they were able to give insightful feedback that showed the strengths of the approach and highlighted the points that can be improved. This study obtained the feedback of practitioners on the use of CHAIN for developing real-life UI help. The participants expressed their satisfaction with how the help got adapted and its post-adaptation usefulness, based on the two UIs that were used in the study. However, the number of participants and UIs is not sufficient to report a significant assessment. In the future, another study can be conducted to assess the usefulness of the help when it is presented on the adapted UI. This assessment could provide further confirmation of the help‟s ability to accommodate various types of UI adaptation. This point and others can be assessed quantitatively with a larger number of participants (practitioners and end-users) and UIs in order to add a different dimension to the results obtained in this study.
AC
CE
PT
ED
M
AN US
tion technology, adding this feature will not affect the ability to use CHAIN with different technologies. Some software systems, e.g., legacy systems, might not use unique identifiers in their UIs. This was mitigated in CHAIN by supporting alternative non-unique identifiers that might not be optimal, e.g., caption text and widget type, but are still plausible in many cases. Displaying help for external software: A help model is created in CHAIN for a single software application. Yet, in some cases it is useful to run a help model across several applications. For example, in web applications, the help model might need to cover UIs that are part of the browser, e.g., print dialog, as well as HTML UIs that are part of the website. The same case applies to desktop applications when a task requires switching to another application, e.g., an operating system dialog. A potential solution for this limitation is extending CHAIN‟s meta-model to support the development of help models that target multiple applications. Hence, every Scenario would have a target application assigned to it. Each Scenario could then be presented by a version of the CHAINXML Presenter, which works with the target application‟s presentation technology. Executing tasks with side-effects: With help models that are rendered at runtime over the UI, executing some tasks could have undesirable side-effects. For example, it might not be desirable to send an email while demonstrating to an end-user how to work with notifications. Another example can be not wishing to store data in a live database while demonstrating how to perform a type of activity, e.g., diet plan, appointment, patient record, and so on. One way to potentially address this limitation is by supporting annotations over help model Action elements to specify the ones that execute an operation with a sideeffect. Hence, the CHAINXML Presenter can highlight the associated UI element, e.g., button, and wait for the end-user‟s confirmation before proceeding. Displaying scenarios on a timeline: One suggested improvement for Cedar Studio is the addition of a view that represents Scenarios and Scenes on a timeline. Timelines are common in video editing tools like “Adobe After Effects”. Hence, the users of these tools might find timelines to be compact and capable of accommodating more elements on the screen. Timelines can also visually illustrate whether two or more Scenarios are going to run in parallel. This suggestion partially overlaps with a comment given by a participant in the preliminary study (refer to Section 9.5) on eliminating the space between parallel SceneElements. Cedar Studio supports displaying Scene Elements as narrow rectangles when the help model is zoomed out. These narrow rectangles resemble the sprites on a timeline view that is offered by video editors. Nonetheless, it could be useful to add a timeline as an alternative view that resembles video editing tools. Incorporating scenario recording (e.g., using Selenium) in Cedar Studio: Another suggestion was to incorporate the automated scenario recording, which is cur-
27
11 CONCLUSIONS AND FUTURE WORK This paper contributes a novel approach called Contextual Help for Adaptive INterfaces (CHAIN) for developing model-driven contextual UI help, which maintains its usefulness (usability and utility) when the UI adapts. A language called Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) is presented alongside a visual notation for representing help models. CHAIN has several advantages over existing help development techniques. The main advantage is that CHAIN help maintains its usefulness by adapting at runtime as the UI adapts. Furthermore, CHAIN help models can be modified to reflect new requirements, instead of having to create completely new help artifacts like videos. It is also possible to validate a CHAIN help model to see if it still conforms to the UI it is associated with and to check for the presence of human errors. End-users can navigate to different segments of a CHAIN help scenario and can interact with it by providing an input and awaiting a response.
ACCEPTED MANUSCRIPT 28
Pierre A. Akiki / The Journal of Systems and Software
TABLE 2 Summary of the CHAINXML Tags Tag Name
Icon
+ChainXML +ResourceDictionary -Style -Text -Image
Description Highest level tag in a CHAINXML document A collection of static and dynamic resources Specifies an element‟s formatting in terms of: color, font, and so on Any alphanumeric string that is displayed or read during a scene Any picture or icon that is used as part of a help model
-DynamicElementCollection
Elements are added at runtime that the help scenario needs to access Helps the end-users in understanding how to perform a certain task Part of a scenario that shows how to perform part of a task
CR IP T
+Scenario +Scene -TextInput
Simulates human-like text input in a UI widget such as a text box
-ValueChange
Changes the value in a UI widget such as a combo box
-MouseMovement
Moves the mouse around a UI at a certain speed
-MouseCommand
Sends specified mouse button clicks to a UI
, Mouse Double Click
, Mouse Up
, Mouse Down
AN US
Mouse Click
-KeyboardCommand
Sends specified keystrokes to a UI
-Marking
Emphasizes parts of a UI, by highlighting or drawing shapes
Highlight Marking
, Square Marking
, Ellipse Marking
Shows a text message in a pop-up balloon
-UIWindow
Loads a helper UI window
-SynthesizedSpeech
Uses a speech synthesis engine to read text
-HumanSpeech
Plays back a prerecorded human voice
-CharacterAnimation
Shows and manages an animated character that can explain the UI
-Magnification
Magnifies an important or recently updated area in the UI
PT
-UnregisterDynamicElement +UserInteractions -UserKeyPress (input) -PresentScene (response)
ED
-RegisterDynamicElement
M
-Callout
Adds dynamically created elements (e.g., drawings) to a collection Removes dynamically created elements from a collection User inputs asking for help and responses to those inputs Detects a user‟s key press (e.g., F1) on part of a UI to ask for help Triggers the presentation of a help scene as an answer to a user input
AC
CE
Legacy software applications were also taken into consideration. Hence, it is possible to present CHAIN help models on the UIs of existing applications, whether or not their source-code is available. If the source-code is available, CHAIN components like the CHAINXML Presenter are integrated as part of the target application. Otherwise, these components become part of a mediator that links CHAIN help models to target applications by using UI automation. Cedar Studio [12] was extended with a visual design tool that supports the development of CHAIN help models. This tool is complemented by the Selenium IDE [33], which is used for automatically recording activities that are performed on a UI. These activities can be saved to a file and imported to Cedar Studio, which automatically converts them to a help model. This work was evaluated in two separate studies. The first study was conducted as a focus group to acquire feedback primarily on CHAIN‟s visual notation. The results of this focus group provided a good indication about the technical qualities of this notation and offered some insights on the expectations of developers. The
second study evaluated CHAIN by asking developers to use it to develop help for real-life adaptive UIs. This study provided insights about CHAIN‟s strengths and highlighted points that inform future work aiming towards improving its existing features. In addition to the points that were highlighted by the second study, CHAIN can be used as a basis for future work targeting a variety of contextual help topics. The following paragraphs provide two examples of these topics. Future work targeting end-user development and usage of help systems, can be based on the outcome of this paper. One interesting direction is providing specialized end-user development tools. Support for crowdsourcing would also be an interesting trait, since it could serve as a means of collaboration and sharing among end-users and expert communities. The work presented in this paper can also form the basis for building help systems that follow an advanced cooperative model, which engages end-users in a dialog with the system. End-users would be able to ask advanced questions about certain points in which they
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
[7]
ACKNOWLEDGMENT
[12]
[8]
[9]
[10]
[11]
AN US
This work is supported in part by ERC Advanced Grant 291652. Many thanks go to all the participants for taking the time to participate in the studies and for their valuable feedback.
[13]
APPENDIX
PT
ED
M
A summary of the tags supported by CHAINXML is provided in Table 2. This summary includes the tag name supported by the XML-compliant syntax, the icon used as part of the visual notation that is supported by Cedar Studio, and a short description explaining the purpose of the tag. When manually sketching a help model using the visual notation, it possible to draw variations of the icons that are presented in Table 2. These variations aim at making the sketching process easier, and include for example drawing only the contour of an icon, tilting it differently, eliminating minor details, and so on.
[3]
[4]
[5]
[6]
G. Calvary, J. Coutaz, D. Thevenin, Q. Limbourg, L. Bouillon, and J. Vanderdonckt, “A Unifying Reference Framework for Multi-Target User Interfaces,” Interact. Comput., vol. 15, no. 3, pp. 289–308, 2003. J. Nielsen, “A model of attributes of system acceptability,” in Usability Engineering, 1st ed., Morgan Kaufmann, 1994, p. 25. M. Kern, F. Trollmann, M. Blumendorf, and S. Albayrak, “Adaptive User Interface Assistance in Smart Environments,” in Proceedings of the Workshop on Meaning and Matching (AISB 2010), 2010. T. Yeh et al., “Creating Contextual Help for GUIs Using Screenshots,” in Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology, 2011, pp. 145–154. J. Matejka, T. Grossman, and G. Fitzmaurice, “Ambient Help,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 2751–2760. B. H. C. Cheng et al., “Software Engineering for Self-Adaptive Systems: A Research Roadmap,” in Software Engineering for SelfAdaptive Systems, vol. 5525, B. H. C. Cheng, R. Lemos, H. Giese, P. Inverardi, and J. Magee, Eds. Springer, 2009, pp. 1–26.
AC
[2]
CE
REFERENCES [1]
P. A. Akiki, A. K. Bandara, and Y. Yu, “Adaptive Model-Driven User Interface Development Systems,” ACM Comput. Surv., vol. 47, no. 1, p. 64:1-64:33, 2014. A. Demeure, G. Calvary, and K. Coninx, “COMET(s), A Software Architecture Style and an Interactors Toolkit for Plastic User Interfaces,” in Proceedings of the 15th International Workshop on Interactive Systems Design Specification and Verification, Kingston, Canada, 2008, pp. 225 – 237. K. Z. Gajos, D. S. Weld, and J. O. Wobbrock, “Automatically Generating Personalized User Interfaces with Supple,” Artif. Intell., vol. 174, no. 12, pp. 910–950, Aug. 2010. P. A. Akiki, A. K. Bandara, and Y. Yu, “RBUIS: Simplifying Enterprise Application User Interfaces through Engineering Role-Based Adaptive Behavior,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 3–12. M. Allbee, “Lawson Smart Office brings WPF Goodness to the Enterprise,” 2008. [Online]. Available: http://bit.ly/1mdhYJ0. [Accessed: 01-Aug-2013]. P. A. Akiki, A. K. Bandara, and Y. Yu, “Cedar Studio: An IDE Supporting Adaptive Model-Driven User Interfaces for Enterprise Applications,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 139–144. H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human-Computer Interaction, 2nd ed. Wiley, 2007. C. Plaisant and B. Shneiderman, “Show Me! Guidelines for Producing Recorded Demonstrations,” in Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, 2005, pp. 171–178. “HelpNDoc.” [Online]. Available: http://www.helpndoc.com. [Accessed: 03-Nov-2014]. “Help & Manual.” [Online]. Available: http://www.helpandmanual.com. [Accessed: 03-Nov-2014]. Techsmith, “Snagit.” [Online]. Available: http://snagit.en.softonic.com. [Accessed: 22-Dec-2015]. Microsoft, “Steps Recorder.” [Online]. Available: http://windows.microsoft.com/en-us/windows7/how-do-iuse-problem-steps-recorder. [Accessed: 22-Dec-2015]. “ScreenSteps.” [Online]. Available: http://www.screensteps.com. [Accessed: 03-Nov-2014]. “Add Contextual Help to Salesforce.” [Online]. Available: https://www.youtube.com/watch?v=5aiyuReI7dU. [Accessed: 17-Jun-2015]. D. E. Caldwell and M. White, “CogentHelp: A Tool for Authoring Dynamically Generated Help for Java GUIs,” in Proceedings of the 15th Annual International Conference on Computer Documentation, 1997, pp. 17–22. D. K. Farkas, “The Role of Balloon Help,” ACM SIGDOC Asterisk J. Comput. Doc., vol. 17, no. 2, pp. 3–19, 1993. M. Terry and E. D. Mynatt, “Side Views: Persistent, OnDemand Previews for Open-Ended Tasks,” in Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology, Paris, France, 2002, pp. 71–80. T. Grossman and G. Fitzmaurice, “ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2010, pp. 1515–1524. B. A. Myers, D. A. Weitzman, A. J. Ko, and D. H. Chau, “Answering Why and Why Not Questions in User Interfaces,” in Proceedings of the SIGCHI conference on Human Factors in compu-
CR IP T
require assistance, and the system could respond for example by playing a relevant help Scenario. Hence, CHAIN can be extended to support different types of UserInputs and their related Responses such as questions and answers. Another possible direction is to monitor end-user behavior in order to determine if they require assistance. For example, the help system could detect if an end-user is confused and use the CHAIN help models, which are kept alive at runtime, to propose assistance and provide guidance. Plasticity denotes a UIs ability to adapt to the contextof-use, while preserving its usability [90]. The help developed using CHAIN preserves its usefulness as the UI adapts, since it is presented over the adapted version of the UI. In the future, it would interesting to research the plasticity of UI help when alternate interaction modalities, e.g., voice, tactile, and so on, are considered depending on changes in the context-of-use.
29
[14]
[15] [16] [17] [18]
[19] [20]
[21]
[22] [23]
[24]
[25]
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
[44]
[45]
[46]
[47]
[48]
[49]
oriented Help,” in Proceedings of the 8th Annual ACM Symposium on User interface and Software Technology, 1995, pp. 181–187. J. Eisenstein and C. Rich, “Agents and GUIs from Task Models,” in Proceedings of the 7th International Conference on Intelligent User Interfaces, 2002, pp. 47–54. E. A. DeKoven, Help Me Help You: Designing Support for PersonProduct Collaboration. TU Delft, Delft University of Technology, 2004. F. García, J. Contreras, P. Rodríguez, and R. Moriyón, “Help generation for task based applications with HATS,” in Engineering for Human-Computer Interaction, Springer, 1999, pp. 149–167. V. Gribova, “Automatic Generation of Context-Sensitive Help Using a User Interface Project,” in Proceedings of the 8th International Conference” Knowledge-Dialogue-Solution, 2007, vol. 2, pp. 417–422. J. Nielsen and R. Molich, “Heuristic Evaluation of User Interfaces,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1990, pp. 249–256. A. Ramachandran and R. M. Young, “Providing Intelligent Help Across Applications in Dynamic User and Environment Contexts,” in Proceedings of the 10th International Conference on Intelligent User Interfaces, 2005, pp. 269–271. D. R. Olsen Jr, S. E. Hudson, T. Verratti, J. M. Heiner, and M. Phelps, “Implementing Interface Attachments Based on Surface Representations,” in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1999, pp. 191–198. K. Reinecke and A. Bernstein, “Improving Performance, Perceived Usability, and Aesthetics with Culturally Adaptive User Interfaces,” ACM Trans. Comput.-Hum. Interact., vol. 18, no. 2, pp. 1–29, Jun. 2011. K. Mao, L. Capra, M. Harman, and Y. Jia, “A Survey of the Use of Crowdsourcing in Software Engineering,” J. Syst. Softw., vol. 126, pp. 57–84, 2017. P. A. Akiki, A. K. Bandara, and Y. Yu, “Crowdsourcing User Interface Adaptations for Minimizing the Bloat in Enterprise Applications,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 121–126. A. G. Frey, G. Calvary, and S. Dupuy-Chessa, “Users need your models!: Exploiting Design Models for Explanations,” in Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers, 2012, pp. 79–88. R. Moriyon, P. Szekely, and R. Neches, “Automatic Generation of Help from Interface Design Models,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1994, pp. 225–231. C. Rich and C. L. Sidner, “DiamondHelp: A Generic Collaborative Task Guidance System,” AI Mag., vol. 28, no. 2, p. 33, 2007. C. Rich, C. Sidner, N. Lesh, A. Garland, S. Booth, and M. Chimani, “DiamondHelp: A Graphical User Interface Framework for Human-Computer Collaboration,” in IEEE International Conference on Distributed Computing Systems Workshops, 2005, pp. 514–519. C. Rich, N. Lesh, A. Garland, and J. Rickel, “A Plug-in Architecture for Generating Collaborative Agent Responses,” in Proceedings of the 1st International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2, 2002, pp. 782–789. M. Fowler, UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd ed. Addison-Wesley Professional, 2004. F. Paternò, C. Mancini, and S. Meniconi, “ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models,” in Pro-
AC
CE
PT
ED
M
AN US
ting systems, Montreal, Quebec, Canada, 2006, pp. 397–406. [26] T. Babaian, B. J. Grosz, and S. M. Shieber, “A Writer‟s Collaborative Assistant,” in Proceedings of the 7th International Conference on Intelligent User Interfaces, 2002, pp. 7–14. [27] M. Dixon, D. Leventhal, and J. Fogarty, “Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 969–978. [28] T. Yeh, T.-H. Chang, and R. C. Miller, “Sikuli: Using GUI Screenshots for Search and Automation,” in Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, 2009, pp. 183–192. [29] R. Mansfield, Mastering VBA for Microsoft Office 2013. Sybex, 2013. [30] T. Myer, Apple Automator with AppleScript Bible. Wiley, 2009. [31] “JitBit Macro Recorder.” [Online]. Available: https://www.jitbit.com/macro-recorder. [Accessed: 21-Jun2015]. [32] “Mini Mouse Macro.” [Online]. Available: http://sourceforge.net/projects/minimousemacro. [Accessed: 21-Jun-2015]. [33] “SeleniumHQ.” [Online]. Available: http://www.seleniumhq.org. [Accessed: 10-Oct-2015]. [34] G. Leshed, E. M. Haber, T. Matthews, and T. Lau, “CoScripter: Automating & Sharing How-To Knowledge in the Enterprise,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2008, pp. 1719–1728. [35] T. Babaian and W. T. Lucas, “Leveraging Usage History to Support Enterprise System Users,” in Proceedings of the 14th International Conference on Enterprise Information Systems, Wroclaw, Poland, 2012, pp. 35–44. [36] F. Linton, D. Joy, H.-P. Schaefer, and A. Charron, “OWL: A Recommender System for Organization-Wide Learning,” Educ. Technol. Soc., vol. 3, no. 1, pp. 62–76, 2000. [37] G. Greco, A. Guzzo, G. Manco, and D. Sacca, “Mining and Reasoning on Workflows,” Knowl. Data Eng. IEEE Trans. On, vol. 17, no. 4, pp. 519–534, 2005. [38] Y. A. Chakravarthi, C. Lutteroth, and G. Weber, “AIMHelp: Generating Help for GUI Applications Automatically,” in Proceedings of the 10th International Conference NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction, 2009, pp. 21–28. [39] P. K. Chilana, A. J. Ko, and J. O. Wobbrock, “LemonAid: Selection-based Crowdsourced Contextual Help for Web Applications,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2012, pp. 1549– 1558. [40] G. Stevens and T. Wiedenhöfer, “CHIC- A pluggable solution for community help in context,” in Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, 2006, pp. 212–221. [41] A. García Frey, “Self-Explanatory User Interfaces by ModelDriven Engineering,” in Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Berlin, Germany, 2010, pp. 341–344. [42] A. García Frey, G. Calvary, and S. Dupuy-Chessa, “Xplain: An Editor for Building Self-Explanatory User Interfaces by ModelDriven Engineering,” in Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Berlin, Germany, 2010, pp. 41–46. [43] S. Pangoli and F. Paternò, “Automatic Generation of Task-
CR IP T
30
[50]
[51]
[52]
[53]
[54]
[55]
[56] [57]
[58]
[59] [60]
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
[67] [68] [69]
[70]
[71]
[72]
[73] [74] [75]
[76] [77]
CR IP T
[78] A. Demeure, J. Meskens, K. Luyten, and K. Coninx, “Design by Example of Graphical User Interfaces Adapting to Available Screen Size,” in Proceedings of the 7th International Conference on Computer-Aided Design of User Interfaces, Albacete, Spain, 2009, pp. 277–282. [79] J. Lin and J. A. Landay, “Employing Patterns and Layers for Early-Stage Design and Prototyping of Cross-Device User Interfaces,” in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy, 2008, pp. 1313– 1322. [80] M. Petre, “Why Looking Isn‟t Always Seeing: Readership Skills and Graphical Programming,” Commun. ACM, vol. 38, no. 6, pp. 33–44, 1995. [81] J. Kontio, L. Lehtola, and J. Bragge, “Using the Focus Group Method in Software Engineering: Obtaining Practitioner and User Experiences,” in Empirical Software Engineering, 2004. ISESE ’04. Proceedings. 2004 International Symposium on, 2004, pp. 271–280. [82] J. Langford and D. McDonagh, Focus Groups: Supporting Effective Product Development. CRC press, 2003. [83] N. Mangano, T. D. LaToza, M. Petre, and A. Van der Hoek, “How Software Designers Interact with Sketches at the Whiteboard,” Softw. Eng. IEEE Trans. On, vol. 41, no. 2, pp. 135–156, 2015. [84] Q. Chen, J. Grundy, and J. Hosking, “An E-whiteboard Application to Support Early Design-Stage Sketching of UML Diagrams,” in Human Centric Computing Languages and Environments, 2003. Proceedings. 2003 IEEE Symposium on, 2003, pp. 219– 226. [85] “Dynamics GP Inventory.” [Online]. Available: http://bit.ly/1N8bE5x. [Accessed: 23-Nov-2015]. [86] D. L. Moody, “The „Physics‟ of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering,” Softw. Eng. IEEE Trans. On, vol. 35, no. 6, pp. 756–779, 2009. [87] G. Mori, F. Paternò, and C. Santoro, “CTTE: Support for Developing and Analyzing Task Models for Interactive System Design,” IEEE Trans. Softw. Eng., vol. 28, no. 8, pp. 797–813, 2002. [88] “Debut Video Capture.” [Online]. Available: www.nchsoftware.com/capture. [Accessed: 15-Dec-2016]. [89] “Adobe After Effects.” [Online]. Available: www.adobe.com/products/aftereffects.html. [Accessed: 15Dec-2016]. [90] G. Calvary et al., “Plasticity of User Interfaces: A Revisited Reference Framework,” in In Task Models and Diagrams for User Interface Design, 2002, pp. 127–134.
AN US
[66]
M
[65]
ED
[64]
PT
[63]
CE
[62]
AC
[61]
ceedings of the 6th International Conference on Human-Computer Interaction, Sydney, Australia, 1997, vol. 96, pp. 362–369. D. R. Olsen,Jr., “Evaluating User Interface Systems Research,” in Proceedings of the 20th ACM SIGCHI Symposium on User Interface Software and Technology, Newport, Rhode Island, USA, 2007, pp. 251–258. F. Bodar, A.-M. Hennebert, J.-M. Leheureux, I. Provot, J. Vanderdonckt, and G. Zucchinetti, “Key Activities for a Development Methodology of Interactive Applications,” in Critical Issues in User Interface Systems Engineering, D. Benyon and P. Palanque, Eds. London: Springer London, 1996, pp. 109–134. P. Sukaviriya and J. D. Foley, “Coupling a UI Framework with Automatic Generation of Context-sensitive Animated Help,” in Proceedings of the 3rd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology, New York, NY, USA, 1990, pp. 152–166. C.-E. Dessart, V. Genaro Motti, and J. Vanderdonckt, “Showing User Interface Adaptivity by Animated Transitions,” in Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, New York, NY, USA, 2011, pp. 95–104. A. Dingli, Y. Wilks, R. Catizone, and W. Cheng, “The Companions: Hybrid-world Approach,” in Proceedings of the 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, 2009, p. 60. T. R. G. Green and M. Petre, “Usability Analysis of Visual Programming Environments: A „Cognitive Dimensions‟ Framework,” J. Vis. Lang. Comput., vol. 7, no. 2, pp. 131–174, 1996. “Xamarin.” [Online]. Available: https://www.xamarin.com/. [Accessed: 14-Apr-2017]. “Double Agent on SourceForge.net.” [Online]. Available: http://doubleagent.sourceforge.net. [Accessed: 05-Jun-2015]. P. A. Akiki, A. K. Bandara, and Y. Yu, “Engineering Adaptive Model-Driven User Interfaces,” IEEE Trans. Softw. Eng., vol. 42, no. 12, pp. 1118–1147, Apr. 2016. P. A. Akiki, A. K. Bandara, and Y. Yu, “Integrating Adaptive User Interface Capabilities in Enterprise Applications,” in Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India, 2014, pp. 712–723. S. Feuerstack, M. Blumendorf, and S. Albayrak, “Bridging the Gap between Model and Design of User Interfaces,” in GI Jahrestagung (2), Dresden, Germany, 2006, vol. P-94, pp. 131– 137. J. McGrenere, R. M. Baecker, and K. S. Booth, “An Evaluation of a Multiple Interface Design Solution for Bloated Software,” in Proceedings of the 20th SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, USA, 2002, pp. 164–170. K. Reinecke, “Culturally Adaptive User Interfaces,” PhD Thesis, University of Zurich, Zurich, 2010. “Apache Open For Business (OFBiz).” [Online]. Available: http://ofbiz.apache.org. [Accessed: 29-Jul-2013]. S. Hao, B. Liu, S. Nath, W. G. Halfond, and R. Govindan, “PUMA: Programmable UI-Automation for Large-Scale Dynamic Analysis of Mobile Apps,” in Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, 2014, pp. 204–217. Microsoft, “Microsoft UI Automation.” [Online]. Available: http://bit.ly/1Z1xvO5. [Accessed: 13-May-2015]. “TestStack/White.” [Online]. Available: https://github.com/TestStack/White. [Accessed: 13-May2015].
31
ACCEPTED MANUSCRIPT 32
Pierre A. Akiki / The Journal of Systems and Software
AC
CE
PT
ED
M
AN US
CR IP T
Pierre A. Akiki is an assistant professor of computer science at Notre Dame University – Louaize, Lebanon. He received a Ph.D. in Computing from The Open University, U.K. in 2014. His research interests include adaptive model-driven interactive software systems, enterprise applications, and the internet of things. His work on adaptive model-driven user interfaces was published in several venues including IEEE TSE, ACM Computing Surveys, and ICSE, and it received the best paper award at ACM SIGCHI EICS’13. Further information about Pierre's research work and publications can be found at: http://www.pierreakiki.com.
CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces Pierre A. Akiki PII: DOI: Reference:
S0164-1212(17)30250-9 10.1016/j.jss.2017.10.017 JSS 10056
To appear in:
The Journal of Systems & Software
Received date: Revised date: Accepted date:
7 February 2017 26 September 2017 13 October 2017
Please cite this article as: Pierre A. Akiki , CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces, The Journal of Systems & Software (2017), doi: 10.1016/j.jss.2017.10.017
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT The Journal of Systems and Software
1
Highlights
CE
PT
ED
M
AN US
CR IP T
CHAIN is an approach for developing model-driven contextual help for adaptive UIs. CHAIN help models are defined using a language (CHAINXML) and a visual notation. The definition of help models is supported by Cedar Studio. CHAIN help models work with both new and legacy application UIs. Two evaluation studies provided positive indications and insights for future work.
AC
ACCEPTED MANUSCRIPT 2
CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces Pierre A. Akiki Department of Computer Science, Notre Dame University – Louaize, Lebanon
AN US
CR IP T
Abstract— Adaptive user interfaces (UIs) change their presentation at runtime to remain usable in different contexts-of-use. Such changes can cause discrepancies between the UI and static help materials, e.g., videos and screenshots, thereby negatively impacting the latter’s usefulness (usability and utility). This paper presents Contextual Help for Adaptive INterfaces (CHAIN), which is an approach for developing model-driven contextual help that maintains its usefulness across UI adaptations. This trait is achieved by interpreting the help models at runtime and overlaying instructions on the running adapted UI. A language called Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) and a visual notation were developed for expressing and depicting help models. A technique was also devised for presenting CHAIN help models over legacy applications, whether or not their source-code is available. A supporting tool was developed as an extension to Cedar Studio. This work was empirically evaluated in two studies. The first study performed a preliminary evaluation of CHAIN’s visual notation. The second study evaluated CHAIN’s strengths and shortcomings after using it to develop help for real-life adaptive UIs. The results gave a positive indication about CHAIN’s technical qualities and provided insights that could inform future work. Keywords: Contextual Help, Adaptive User Interfaces, Model-Driven Help, Software Support, Design Tools and Techniques
1 INTRODUCTION
A
AC
CE
PT
ED
M
daptive user interfaces (UIs) change their presentation at runtime to preserve their usability in different contexts-of-use (user, platform, and environment) [1]. Such changes negatively affect the usefulness (usability and utility [2], p.25) of static end-user help such as videos and screenshots. This undesirable effect is due to discrepancies between the initial UI, on which the help is based, and the adapted UI. For example, a widget would become difficult to locate using static UI help if its location changes at runtime. Therefore, adaptive UIs require a new type of help system, which also needs to be adaptive [3]. This paper presents an approach for developing model-driven contextual UI help as an alternative to static help. Contextual help is a form of assistance that is presented to end-users on the running UI, rather than in a separate viewer such as a browser or a video player [4]. As end-users interact with a running UI, contextual help can provide assistance by displaying popup messages, entering example data, highlighting elements, and so on. Representing contextual help as interpreted runtime models, allows instructions and markings to be overlaid dynamically on the running UI, thereby maintaining the help‟s usefulness for different UI adaptations.
1.1 Advantages of Model-Driven Contextual Help Non-contextual help is presented apart from the UI, e.g., using a video player or a secondary display [5]. Moving back and forth between the UI and the help viewer could distract end-users and increase their memory load. Since contextual help is presented on top of the running UI, it can more effectively support the end-users‟ learning pro————————————————
Pierre A. Akiki is with the Department of Computer Science, Notre Dame University – Louaize, Zouk Mosbeh, P.O. Box: 72, Zouk Mikael, Lebanon. E-mail: [email protected]
cess [4]. A direct advantage of representing contextual help as models, is allowing its content to be updated if the UI is modified due to rapidly changing requirements, e.g., in web applications. On the other hand, static help artifacts like videos are more costly to update, because they would have to be redeveloped from scratch. Cheng et al. [6] consider that “software systems must become more versatile, flexible, … , customizable, configurable, and self-optimizing by adapting to changing operational contexts, environments or system characteristics”. Adaptive UIs have been researched as a way for solving usability problems when contexts-of-use vary. Several approaches have been devised for developing adaptive UIs [7]. For example, Comet(s) are a set of widgets that support UI plasticity by either self-adapting or being adapted [8]. Supple uses preference elicitation and cost functions to generate UIs, which are adapted to an end-user‟s abilities and preferences [9]. RBUIS adopts an interpreted runtime model-driven approach for developing adaptive UIs, which offer a minimal feature-set and an optimal layout based on the context-of-use [10]. Regardless of the adopted development approach, adaptive UIs can ultimately change their visual characteristics, thereby becoming inconsistent with the content of static help materials. The work presented in this paper, complements UI adaptation techniques by allowing software applications to benefit from adaptive UI capabilities, while providing end-users with contextual help that maintains its usefulness with different adaptations of the same UI. For example, enterprise applications like enterprise resource planning (ERP) systems can benefit from adaptive behavior to tailor their thousands of complex UIs for different contexts-of-use. These applications, e.g., Lawson Smart Office [11], can also benefit from offering help to their end-users.
2017
ACCEPTED MANUSCRIPT 3
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 1. An Example of a Single Contextual Help Scenario Running on Two Different Adaptations of the Same ―Customer Information‖ UI -
AN US
(a) UI V1 Groups its Widgets Using Group Boxes, whereas (b) UI V2 Groups its Widgets Using Tab Pages
AC
CE
PT
ED
M
1.2 Proposed Contextual Help Development Approach This paper presents a help development approach called Contextual Help for Adaptive Interfaces (CHAIN). Help models can be defined using an XML-compliant language called CHAINXML and a visual notation. A supporting visual tool was added as an extension to Cedar Studio, an integrated development environment (IDE) for adaptive model-driven UIs [12]. Sophisticated contextual help can be created using this tool without any programming. Software applications can request help models by calling a web-service that returns a CHAINXML document. This document is interpreted at runtime and presented over the running UI as contextual help. The example illustrated in Fig. 1 demonstrates part of a contextual help scenario running over two different adaptations of the same UI. This simple example shows the importance of the proposed approach for developing adaptive contextual help. If a static help development technique was used, then a different artifact (e.g., video) would be needed for each adaptation (V1 and V2) of the UI. Having many runtime variations of the context-ofuse makes developing different versions of the help very costly or even unfeasible. CHAIN help scenarios can be played on a timeline. The end-users control a help scenario by going forward and backward, as is done in videos, using a slider and a set of buttons like the ones shown in Fig. 1 – c. Endusers can also request part of a help scenario that explains a specific section of the UI. CHAIN is also useful for developing help for nonadaptive UIs. Yet, it primarily targets the development of contextual help for adaptive UIs. Hence, the latter will be the focus of this paper.
1.3 Structure of the Article Section 2 categorizes the related work according to the adopted development approach, and discusses its strengths and shortcomings. Section 3 presents CHAIN‟s metamodel, while Section 4 presents its XML-compliant lan-
guage (CHAINXML) and visual notation. Section 5 explains how help models can be developed and validated based on a proposed architecture. Section 6 explains how CHAIN help maintain its usefulness as the UI adapts. Section 7 explains how CHAIN help models can be used with legacy systems, regardless of whether the source-code of these systems is available. A tool for supporting the development of CHAIN help is presented in Section 8. The work presented in this paper is evaluated in two studies, which are presented in Sections 9 and 10. The first study is a preliminary evaluation to obtain feedback on CHAIN‟s visual notation. The second study evaluates the strengths and limitations of CHAIN through actual use. Finally, the conclusions are given and the future work is discussed in Section 11.
2 RELATED WORK Some software help systems, e.g., the work done as part of project Genie [13], date back several decades. To the best of my knowledge, contextual help that maintains its usefulness across different UI adaptations, has not been targeted by the state-of-the-art help development systems. Researchers have used different approaches for developing various forms of end-user help. This section classifies the existing works based on the adopted development approach and points out their strengths and shortcomings. A summary of the state-of-the-art help systems discussed in this section is presented in Table 1. This table briefly states each system‟s expected input and produced output. The UI level of abstraction that each system operates on is also shown in Table 1, with respect to the Cameleon Reference Framework (CRF) [1].
2.1 Static Non-Contextual Help Static non-contextual help can have different forms such as: external text-based documentation, pictures, and videos, and it accompanies software applications to guide end-users on how to perform certain tasks. Plaisant et al. provide guidelines for developing prerecorded demonstra-
ACCEPTED MANUSCRIPT 4
Pierre A. Akiki / The Journal of Systems and Software
TABLE 1 Summary of the State of the Art Help Systems CRF Level
Help and Manual [16]
Text and figures
Help document (PDF, Word, etc.)
-
ScreenSteps [19]
Text and screenshot
Help document (PDF, Word, etc.)
-
Steps Recorder [18]
Key log, screenshots, and annotations
Help steps as text and screenshots
-
Snagit [17]
Screenshots and audio recording
Help video
-
CogentHelp [21]
Help Parts
Help Pages
AUI
Balloon Tooltips [22]
Programmed tooltips
Tooltips showing brief messages
CUI
SideViews [23]
Programmed side view (MVC)
Interactive tooltip
CUI
Toolclips [24]
Text and video
Tooltips with text and video
CUI
Crystal framework [25]
Built-in answer generator
Dialogs that answer questions
CUI, FUI
Writer’s Aid [26]
User preferences, bibliography information, and AI planning
Help for identifying and inserting citations in a document
CUI, FUI
Dixon et al. [27]
UI content and hierarchy
Stencil-based tutorials
FUI
Sikulli [28]
Script and screenshots of UI elements
Slide show presented over the UI
FUI
Microsoft [29], Apple [30], JitBit [31], MiniMouse [32], Selenium [33], CoScripter [34]
End-user activities that are recorded or manually specified (e.g., using a script)
Activity playback over the UIs of desktop or web applications
FUI
Task-Interface-Log [35] OWL [36]
End-user UI interaction log for a task Logged user data (e.g., commands)
Greco et al. [37]
System logs
AIMHelp [38]
End-user interaction information and the monitored state of the GUI
Playback of previous interactions Intelligent tips and skillometer Helps administrators in identifying frequent workflow executions Automatically generated help information Previously asked questions and their answers for selected UI part Help wiki page created by the community
Xplain [41], [42]
AN US
M
Automatic context identification
FUI FUI FUI FUI FUI FUI
Task model association with the UI
Self-explanatory UI
Annotations on task models
Help for Adaptive UIs
Task
Pangoli et al. [43]
Tasks associated with interaction objects
Generated task oriented help
Task
Moriyon et al. [55]
Model (presentation, behavior, and help extensions) and help rules
Generated help text
CUI
CE
Kern et al. [3]
Task, CUI
Collagen [44]
Task model
DiamondHelp [56], [57]
Task model (based on Collagen), and user actions and utterances
Agent that helps end-users in performing certain tasks Collaborative dialogues between application and end-users
Dekoven [45]
Task model (based on Collagen)
Help for physical appliances
Task
HATS [46]
ATOMS task model
Task-oriented context sensitive help
Task
Gribova [47]
Ontology models linking tasks to UI
Context-sensitive help
AC
Model-Driven Engineering
ED
CHiC [40]
UI part on which end-user requires help
PT
Frameworks and Toolkit Extensions Usage Logging Crowdsourcing
Lemon-Aid [39]
CR IP T
-
Static NonContextual Help
Help document (PDF, Word, etc.)
Pixel Analysis
Output
Text and figures
Macro Recorders
Input Helpndoc [15]
tions on how to use the UIs of software systems [14]. A variety of authoring tools are available for supporting the development of static non-contextual help. For example, some tools support the preparation of HTML-based help files, while others support screen casting for recording demonstration videos. Helpndoc [15] and Help and Manual [16] are two examples of such tools. Snagit can be used to prepare screenshot tutorials [17], while Steps Recorder
Task Task
Task, CUI
[18] can create screenshots with descriptions based on a set of actions that an end-user performs on Windows. Although some tools such as ScreenSteps [19] refer to the type of help they support as “contextual”, but the help materials are mainly screenshots and not overlays on the running UI. For example, ScreenSteps was used with Sales Force [20] to develop help lessons that only improve on fully text-based help documents by including screenshots with annotations.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
PT
ED
M
AN US
2.2 Frameworks and Toolkit Extensions Frameworks and toolkit extensions can be built on top of presentation technologies such as Java Swing and Windows Forms, in order to support the development of enduser help systems. For example, CogentHelp was developed with Java‟s Abstract Windowing Toolkit (AWT) as a prototype tool for authoring, maintaining, and customizing help [21]. With CogentHelp, technical stakeholders, e.g., developers, author the help system parts that describe how individual windows or widgets function. Then, CogentHelp generates help pages based on the individual parts that these stakeholders produced. Balloon tooltips [22] is a help system that presents basic explanation inside popup balloons. Other examples such as Side Views [23] and ToolClips [24] can present richer help content. The ability to ask questions about the UI allows endusers to stay informed on what is happening [48]. One example targeting this ability is the Crystal framework, which goes beyond simple help catalogues into answering questions on: “why do things occur or do not occur”, and “how to use certain application features” [25]. AI planning is used by some systems such as SmartAidè [49] and Writer‟s Aid [26], in order to offer advanced help. Despite their importance, these techniques do not offer contextual help that works with adaptive UIs. Although some systems, e.g., CogentHelp, can generate contextual help content that targets a particular context, this help does not adapt to varying context-of-use. Also, frameworks and toolkit extensions can make the help content technology dependent. Other techniques such as pixel analysis and model-driven engineering can provide technology independence.
2.4 Macro Recorders Macro recorders are used to record user-activity on a UI in order to replay it later. These tools are not primarily intended for developing end-user help, but are aimed at automating routine activities. Microsoft Office applications support the creation of macros for automating tasks [29]. JitBit [31] and Mini Mouse [32] are two examples of macro recorders that can support various Windows applications. Apple Automator automates activities on a variety of Mac OS applications [30]. Selenium is a browser plugin that supports recording and playback of activities on web applications [33]. One example from the academic literature is CoScripter [34], which supports collaboration on task recording and automation in web-based systems. Macro recorders do not support annotations such as markings and callouts, which are necessary for end-user help to provide explanation. Some macro recorders are negatively affected by resolution change. This limitation also hinders the ability to operate on different adaptations of a UI. Furthermore, macro recorders cannot validate if a macro still conforms to a UI after it gets updated, whereas this trait is possible with help models through model validation. A macro-like tool for recording user-activities can complement a visual help model design tool and facilitate the creation of help content. Developers or expert users could perform a task using a UI and their actions would be recorded in order to serve as a basis for defining help models.
CR IP T
As mentioned in Section 1.1, the main shortcoming of non-contextual help materials such as prerecorded videos and screenshots lies in their static nature, which prevents them from preserving their usefulness as the UI adapts.
5
AC
CE
2.3 Pixel Analysis Pixel analysis is used by some techniques for providing contextual help on UIs without accessing the software application‟s source code. These techniques can be considered as attachments, which are small interactive programs that augment the functionality of other applications [50]. Dixon et al. used pixel analysis for extending UIs with stencil-based tutorials [27]. Sikuli allows endusers to query a help system using screenshots from the GUI [28]. Yeh et al. presented a technique that analyzes the pixels of GUI screenshots and provides contextual help using a scripted slideshow [4]. An advantage of pixel analysis is its ability to support the development of technology independent help, which can be added to existing applications without code modifications. Yet, pixel analysis is inhibited by scale changes and theme variations in the UI [4]. Hence, if it is used with adaptive UIs, these limitations could be crippling. Adaptive UIs adapt at runtime characteristics such as: accessibility of functions, colors, icons, information density, image-to-text ratio, and widget types [51]. Adapting such characteristics can make it impossible to locate a UI element through its initial pixel representation.
2.5 Usage Logging Some approaches log the usage history data of end-users over a long period of time and play it back to explain how certain software activities are used. Babaian et al. provide a model called the Task-Interface-Log (TIL), which logs the tasks performed by enterprise system end-users for later playback [35]. Organization-Wide Learning (OWL) is a recommender system that logs end-user data and uses it to provide learning tips [36]. Greco et al. use system logs to help administrators in identifying the frequent past choices that lead to a desired configuration [37]. AIMHelp generates help content by eliciting interaction information at runtime [38]. Like macro recorders, this approach does not provide the ability to add annotations on the UI. Existing techniques such as the TIL do not work with a legacy system unless there is access to its source-code. Logging data in large-scale systems such as enterprise applications can consume a lot of storage space, and it could be hard to identify the part of the data that is needed for explaining an activity. On the other hand, CHAIN help models are composed based on knowledge from human experts. Striking a balance between automation and human input is important for realizing systems that efficiently produce a high quality output. Another disadvantage that usage logging shares with macro recorders is the difficulty of supporting multilingual help. Querying usage logs for entries in the target language could be difficult. On the other hand, help models can target multiple human languages.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
Fig. 2. A Class Diagram Representing the Meta-Model of CHAIN’s Abstract Concepts
CR IP T
6
AC
CE
PT
ED
M
2.6 Crowdsourcing Some tools rely on crowdsourcing to involve end-users in the process of developing help for software applications. LemonAid is a tool that leverages the knowledge of the crowd to provide contextual help for web applications [39]. End-users can benefit from previously answered questions about specific parts of the UI. While using a web application, end-users can ask for help on a specific part of the UI, and LemonAid provides them with a searchable list of questions and answers that were generated by the crowd. CHiC is another example tool in this category. It provides community help for the Eclipse development environment [40]. Although this approach does not directly target adaptive UIs, it has a promising potential for involving endusers in the development of contextual help for such UIs. The data collected through crowdsourcing can potentially be used for producing CHAIN models. For example, if end-users were given an automation tool (like a macro recorder) for recording their on-screen activities, it would be possible to generate a CHAIN model from this recording. Crowdsourcing already has its applications in the discipline of software engineering [52], and it shows a potential for involving end-users in UI adaptation [53]. Hence, it could allow expert end-users to create help for end-users who are less knowledgeable about how to use a certain software system.
2.7 Model-Driven Engineering Model-driven engineering (MDE) can be leveraged to develop end-user help. With MDE, it is possible to offer help by anticipating problems that end-users may face at runtime [54]. García Frey et al. used MDE to develop selfexplanatory UIs [41] with the help of a tool called Xplain [42]. Kern et al. used annotations on task models to provide adaptive help for adaptive UIs [3]. Several MDEbased approaches target automatic generation of end-user help. Pangoli et al. automatically generate task-oriented help from abstract models [43]. Moriyon et al. automati-
cally generate help text from models [55]. DiamondHelp [56], [57] uses task models and the Collagen [58] middleware for supporting collaborative dialogues between an application and its end-users. Collagen takes a task model as an input and provides an agent that helps end-users in performing certain tasks [44]. Dekoven uses the same approach to produce textual help points that primarily target physical appliances [45]. HATS uses task models to generate context sensitive help, which informs end-users about how the interaction could be done in order to achieve certain tasks [46]. Gribova presented an approach for generating context-sensitive help by using ontology models that link tasks to their respective UI elements [47]. The existing work demonstrates the potential of using MDE for developing help systems, which are technology-independent and dynamic. However, it does not target help that remains useful when a UI adapts, thereby leaving many challenges to be addressed. One exception is the work from Kern et al. [3], but this work is at a preliminary stage and does not demonstrate a way to replace complex static help artifacts like videos. The automatic help generation offered by some MDEbased approaches could reduce development costs and anticipate help scenarios at runtime. However, in many cases, full automation might not be practical enough for developing complex help scenarios, which require the input of a domain expert. Hence, the approach presented in this paper offers human experts the tool support they need for developing model-driven contextual help. Modelling languages like the Unified Modelling Language (UML) [59], do not provide dedicated models for representing end-user help. The Cameleon Reference Framework [1] suggests multiple levels of abstraction for developing model-driven UIs: task model (e.g., ConcurTaskTrees [60]), abstract UI (AUI), concrete UI (CUI), and final UI (FUI). Yet, these levels of abstraction only represent a UI‟s content. This paper presents a novel approach for representing model-driven contextual help.
ACCEPTED MANUSCRIPT 7
AC
CE
PT
ED
M
AN US
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 3. A Class Diagram Representing the Meta-Model of CHAIN’s Concrete Concepts
2.8 Challenges for Help Systems Some challenges can be deduced after exploring the related work and the different approaches that were adopted for developing end-user UI help. Help systems can become more effective by overcoming these challenges. As
explained previously, some challenges were overcome by certain approaches but not others. The first challenge lies in providing technology independence, which can be overcome by adopting a modeldriven approach. The second challenge is the efficient development of a high quality help, by providing a bal-
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
CR IP T
8
M
Fig. 4. A Class Diagram Representing CHAIN Resources
CE
PT
ED
ance between human input on the help content and automation through tool support. The third challenge is involving end-users in the definition of help. Olsen suggests that UI systems could empower new design participants such as end-users by introducing them into the design process [61]. Crowdsourcing can be one way to achieve this suggestion. Finally, a major challenge is developing end-user help that can adapt and maintain its usefulness when its corresponding UI adapts. The latter is not addressed by any of the existing systems; hence, it will be the main focus of this paper without disregarding the other key challenges.
3 CHAIN CONCEPTS
AC
Contextual Help for Adaptive INterfaces (CHAIN) supports the definition of end-user contextual help, which maintains its usefulness across different adaptations of the same UI. Some systems such as Helpndoc [15] and Help and Manual [16] are used to develop tutorials and manuals for end-users. Other systems are not dedicated for developing help, but provide a form of guidance for endusers. For example, the UI development systems MacIDA and TRIDENT offer guidance related to task progression, so end-users can identify the tasks that were done and those that remain [62]. On the other hand, the work presented in this paper is a UI help system that is dedicated for the development of contextual help. In model-driven UIs that are based on the Cameleon Reference Framework, abstract UI (AUI) models are independent of any modality, e.g., graphical, while con-
crete UI (CUI) models are modality-dependent. Since help models are related to UIs, CHAIN encompasses abstract (Fig. 2) and concrete (Fig. 3) concepts. CHAIN currently supports the graphical UIs at the concrete level. Yet, if other modalities are added in the future the abstract help model will be mapped to the concrete help models representing the different modalities.
3.1 Abstract Concepts The class diagram shown in Fig. 2 represents CHAIN‟s abstract concepts. Abstract help models are composed of AbstractHelpModelElements, which can be one of four types: AbstractHelpContainer, AbstractHelpOperation, AbstractUserInput, and AbstractResponseToUserInput. These elements have a sortorder attribute that indicates the order in which an element is presented to the end-user. An AbstractHelpContainer groups related help model elements that will be presented together. For example, elements that explain a related part of the UI can be grouped under one container. AbstractHelpOperations embody the help that will be presented to the end-users. Each operation can be one of three types: explanation, demonstration, and emphasis. Explanation operations instruct end-users on how to perform a task. For example, on the concrete level, these operations can be mapped to a voice narration or a callout that displays text. On the other hand, demonstration operations show the end-users how to perform a task. For example, these operations can involve moving the cursor, typing text, selecting values, and so on. Finally, emphasis operations are used to make parts of the UI
ACCEPTED MANUSCRIPT 9
CR IP T
Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Fig. 5. A Class Diagram Representing Some Possible CHAIN Concrete User Inputs and Responses
them in the execution of a certain task. For example, in a line-of-business application, a Scenario can demonstrate how a sales invoice is created. A speed can be specified to indicate how fast a Scenario is presented. A Scene is a single step in a Scenario. For example, “select customer” can be a Scene in the “create sales invoice” Scenario. A ScenarioFlow indicates how a Scenario moves from one Scene to another. The flow can be automatic like a video, or it can be manual thereby end-user input would be required to proceed. A Scenario can also be used as a static annotation, where all Scenes are presented at once like a single snapshot. A SceneFlow indicates whether a Scene should run in parallel with other Scenes or sequentially after the previous one. Each Scene is composed of a number of SceneElements, each having one of the following four types: Action, Creation, Speech, and Animation. Action scene-elements invoke a change such as an input. TextInput simulates human-like typing at a specified speed, one letter at a time, in an input ConcreteUIElement (e.g., text box). This simulation gives data-entry a natural feel. A ValueChange can be used for changing the value of a selection ConcreteUIElement (e.g., combo box). The MouseMovement simulates cursor movement to guide end-users from one ConcreteUIElement to another. Mouse and Keyboard Commands can also be sent to the UI. Creation scene-elements create help visualizations. A Callout is created next to a UIElement to describe its purpose. A Marking can be used to attract the end-user‟s attention to part of the UI, by highlighting it with a different color or drawing a shape around it. A UIWindow can be used to show supporting UIs, e.g., those used for looking up information. Spoken narration is recommended by existing guidelines on producing help demonstrations [14]. Both Synthesized and Human Speech are supported by CHAIN. SynthesizedSpeech configures and invokes a text-to-speech engine to read a text, while HumanSpeech plays a prerecorded human voice.
PT
ED
M
AN US
more apparent to the end-users. For example, these operations can be realized by highlighting or magnifying a widget. AbstractHelpOperations are applied to AbstractUIElements, which are in turn mapped to ConcreteUIElements. Help models can become interactive by eliciting enduser input and providing responses. On the abstract level, inputs and responses are specified using AbstractUserInputs and AbstractResponsesToUserInputs. Examples of user input on the concrete level include: pressing a button, selecting an option, typing a question, and so on. As a response to an end-user‟s input, the system can, for example, present a relevant part of the help model. Each of the classes representing abstract concepts are assigned one of the following stereotypes: <
AC
CE
3.2 Concrete Concepts The class diagram shown in Fig. 3 represents CHAIN‟s concrete concepts for graphical UIs. Other modalities, e.g., voice and tactile, can be explored further in the future. For example, the current support for speech could form a basis for exploring fully vocal help in the future. Speech is currently supported in CHAIN as part of the help for graphical UIs. Yet, in the future, speech could be explored with purely vocal UIs. The concrete concepts shown in Fig. 3 map to their abstract counterparts in Section 3.1. For example, Callout, TextInput, and Marking are concrete implementations of the explanation, demonstration, and emphasis AbstractOperations, respectively. Classes representing the related abstract concepts (Fig. 2) are also presented in Fig. 3 (colored in gray) to show the relationship between CHAIN‟s abstract and concrete concepts. Each CHAIN Document comprises a help Scenario composed of several Scenes. A Scenario represents contextual help that is presented to end-users for assisting
ACCEPTED MANUSCRIPT 10
Pierre A. Akiki / The Journal of Systems and Software
Listing 1. An Example Excerpt of a Help Scenario Represented as a CHAINXML Document 1.
17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40.
PT
ED
M
AN US
CR IP T
2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.
AC
CE
41.
Fig. 6. An Example Excerpt of a Help Scenario Represented using CHAIN’s Visual Notation
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
11
(<
4 LANGUAGE AND VISUAL NOTATION
This section presents an XML-compliant language and a visual notation that can be used for defining CHAIN help models based on the concepts presented in Section 3.
M
AN US
Animations have been used with UIs in a variety of ways. Some researchers integrated animations as part of help systems [63], while others used animated transitions for conveying UI adaptations to end-users [64]. Animated Characters can in some applications enhance the end-users‟ interaction with the help system by serving as a façade to SynthehizedSpeech and Callouts. For example, the Companions system uses a human-like avatar to make conversations between end-users and applications seem more natural [65]. CHAIN supports the inclusion of animated Characters, which can be moved around the UI or simply remain fixed in a corner. Magnification can be used to make part of the UI more apparent, by zooming in on a particular point or a path. Each CHAIN document has a ResourceDictionary (Fig. 3). The possible entries for this dictionary are depicted by the class diagram presented in
CR IP T
Fig. 7. Implementation Architecture for Supporting the Development of End-User Help Using CHAIN (It is Also Possible to Export CHAINXML Documents from the Database to be used as Local Files)
AC
CE
PT
ED
Fig. 4. These resources include: Styles, Images, and MultilingualTexts, which are defined at design-time and may get updated at runtime. Additionally, these resources include collections of DynamicElements that are added at runtime. A Style can be either defined individually per SceneElement or globally in the ResourceDictionary to be shared among multiple elements. Multilingual Texts are used for on-screen display and for SpeechSynthesis. Having multilingual capabilities in a single help scenario is an advantage over approaches that create different help artifacts for each human language. Images can help in pointing out the use of some help model elements like indicating the type of a Callout. DynamicElementCollections are needed for applications that create DynamicElements. For example, a help Scenario might demonstrate how a modelling tool is used to draw shapes. The abstract concepts that were explained in Section 3.1 included elements that represent user input and responses to that input. Two concrete implementations of these abstract concepts are shown in Fig. 5. Classes representing the related abstract concepts (from Fig. 2) are also presented in Fig. 5 (colored in gray) to show the relationship between CHAIN‟s abstract user inputs and responses to user inputs and their concrete counterparts. A UserKeyPress indicates that a response is triggered when an end-user presses a certain key. The UserKeyPress is linked to one ConcreteUIElement or an entire UI. As a response to a UserKeyPress input, a Scene can be presented to provide an explanation. The UserKeyPress class inherits from UserInput, which is part of a Scenario
4.1 Language The Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) is an XML-compliant language for expressing help models. An excerpt of a document defined using this language and pertaining to the scenario illustrated in Fig. 1 is presented in Listing 1. A number of XML-style tags are supported for defining end-user help based on the concepts shown by the class diagrams in Fig. 2 to Fig. 5. The supported tags are summarized in the Appendix. 4.2 Visual Notation An example of a help model expressed using CHAIN‟s visual notation is presented in Fig. 6. This notation can be used to specify help models without having to explicitly write CHAINXML tags. The “cognitive dimensions” framework [66] was taken into consideration when designing CHAIN‟s visual notation as summarized in the following paragraphs. CHAIN‟s visual notation uses rectangles to consistently represent help model elements. In terms of diffuseness, each meaning (e.g., SceneElement) is expressed using one symbol and a text description. Hence, the notation is terse enough to represent large help models on the screen thereby improving visibility. The example shown in Fig. 6 is detailed and expanded, but zooming out reduces the height of the rectangles. As shown in Fig. 3, a SceneElement is applied to a ConcreteUIElement and thereby depends on it. In order to avoid hidden dependencies, the related ConcreteUIElment is highlighted in the supporting tool when a SceneElement is selected. The relation between SceneElements and ConcreteUIElements does not force a premature commitment since the SceneElements can be specified first and then linked to the UI. This way an early stage help model can be independent from the UI design. Progressive evaluation can be done by executing a help model even if it is not fully complete. Hence, novice designers and developers can evaluate their progress at frequent intervals. The viscosity of CHAIN‟s visual notation is low, since very little
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
Listing 2. CHAINXML Presenter - Speech Synthesis Code (Excerpt) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
//Get properties from CHAINXML document function SESynthesizedSpeech(XmlData) { this.ReadPropertiesFromXML(XmlData); } //Synthesize text using properties read from CHAINXML doc. SESynthesizedSpeech.prototype.Present = function () { var utterance = new SpeechSynthesisUtterance(); SetUtterProperties(utterance, this.Voice, this.Volume, this.Rate, this.Pitch, this.Text, this.Language); utterance.onend = function (event) { //Notify the scene that the text has been synthesized Scene.ElementPresentedSuccessfully(IndexInScene); }; window.speechSynthesis.speak(utterance); } //Cancel speech synthesis while in operation SESynthesizedSpeech.prototype.End = function () { window.speechSynthesis.cancel(); }
5 IMPLEMENTING HELP USING CHAIN
M
This section explains how CHAIN help models can be implemented based on the architecture shown in Fig. 7 and how these models are validated and fixed.
CE
PT
ED
5.1 Developing Help Models As depicted by the architecture in Fig. 7, CHAIN help models are stored in a relational database and can be accessed by software applications through a web-service. The CHAINXML Converter converts help models from relational data into CHAINXML documents. These documents contain XML-style tags that are based on the concepts discussed in Section 3. An excerpt of a CHAINXML document pertaining to the scenario illustrated in Fig. 1 is presented in Listing 1. A CHAINXML Presenter dynamically interprets and presents CHAINXML documents on the client-side as interactive contextual-help. Help models are technologyindependent. A presenter is dependent on a particular presentation technology such as: HTML, WPF, Swing, and so on. Nonetheless, it can be used across different software applications that adopt the same presentation technology. A Caching Engine caches CHAINXML documents on the client-side in order to improve performance, in case an end-user requested a help scenario more than once. To test CHAIN, different CHAINXML Presenters were developed using JavaScript for the web, C# with the .Net Framework for the desktop, and C# with Xamarin [67] for the mobile. The code excerpt shown in Listing 2 is taken from the JavaScript version of the presenter and is responsible for SpeechSynthesis. The code in Listing 3 shows how a CHAIN scenario is loaded and presented simply by calling a web-service using the Service Access API. The code excerpt shown in Listing 2 provides an
AC
1. function PresentScenario(ScenarioID) { 2. //Load a CHAINXML document from web-service using API 3. var doc = LoadChainXMLScenario(ScenarioID); 4. //Call CHAINXML Presenter to present the document 5. PresentChainXMLScenario(doc); 6. }
Listing 4. Load a Single CHAIN Scene to Explain Part of the UI When the End-User Requests Help by Pressing a Key (Excerpt) 1. 2. 3. 4. 5. 6. 7.
$("#UIElementID").keydown(function (e) { if (KeyCode == Code) //e.g., 112 indicates F1 Pressed { PresentScene(ScenarioID, SceneID); return false; //do not propagate the key press } });
example of how the presentation is done. The code shown in Listing 4 adds an event handler for the “keydown” event of a ConcreteUIElement, in case a UserKeyPress input is specified with a PresentScene response. Such event handlers are automatically added by the CHAINXML Presenter. The CHAINXML Presenter targeting HTML leverages the speech synthesis capabilities of HTML5 (Listing 2), which are supported by Google Chrome. The other versions of the presenter synthesize speech using the libraries provided by C#. Microsoft Agent (Double Agent) [68] is used to present Animated Characters, but CHAIN is not restricted to this technology.
AN US
effort is required to change a SceneElement. Developers can simply add a new element from the toolbox and delete the old one. The supporting tool also allows developers to right-click on an element and switch its type, thereby preserving common properties between the old type and the new one. Help models are stored in a database, thus it is possible to perform global changes across all models by writing queries. Packaging these queries into reusable transactions makes them executable without having to write code each time a change is needed.
Listing 3. Load and Present Complete CHAIN Scenario (Excerpt)
CR IP T
12
5.2 Transforming Help Models at Different Levels of Abstraction and Mapping these Models to the UI When multiple levels of abstraction are used in modeldriven development, transformations are applied to map a model on one level of abstraction to its counterpart on another level. For example, in model-driven UI development, transformations are used to perform mappings between models on the different levels of abstraction (task model, AUI, CUI, and FUI) defined by the Cameleon Reference Framework. The approach presented in this paper also applies transformations between the help models represented at different levels of abstraction. Abstract help models (corresponds to AUI) are transformed to concrete help models (corresponds to CUI), and the latter are transformed to working contextual help (corresponds to FUI) that is presented at runtime over the UI. Transformations between abstract and concrete help models are done using two techniques that are already supported by Cedar Studio for UI models. The first technique involves the definition of transformation rules through a visual mapping between the abstract and concrete help model elements. Cedar Studio shows a list of all the elements in a model and their possible counterparts on the other level of abstraction. This list can be used to map each element to the one that mostly fits a particular help scenario. For example, an Abstract Help Operation can be transformed to different types of Scene Elements such as: TextInput, Callout, SynthesizedSpeech, and so on. The second technique involves transforming one level of abstraction to another by defining scripts. Cedar Studio currently supports Iron Python, but it is
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Declare @SceneElemID int, @SceneElemType int Declare HMSECursor Cursor For --Get cursor movement scene elements GetSceneElements(@@HelpModelID, 'MouseMovement') Open HMSECursor --Iterate around the scene elements Fetch Next From HMSECursor Into @SceneElemID,@SceneElemType While @@FETCH_STATUS = 0 Begin --If previous element is a marking add both elements to result Select @PrevElemID = GetPreviousElement(@SceneElemID,'Marking') if @PrevElemID is not null Insert Into #ErroneousSceneElements Values(@@HelpModelID, @SceneElemID, @PrevElemID) Fetch Next From HMSECursor Into @SceneElemID,@SceneElemType End Close HMSECursor Deallocate HMSECursor Select * From #ErroneousSceneElements
Declare @SortOrder1 int, @SortOrder2 int Set @SortOrder1 = GetSortOrder(@@SceneElement1ID) Set @SortOrder2 = GetSortOrder(@@SceneElement2ID) /*Swap the sort orders to move the ‘Hightlight’ scene element after the ‘Cursor Movement’ one*/ Update SceneElements Set SortOrder = @SortOrder2 Where SceneElementID = @@SceneElement1ID and HelpModelID = @@HelpModelID Update SceneElements Set SortOrder = @SortOrder1 Where SceneElementID = @@SceneElement2ID and HelpModelID = @@HelpModelID
are with constraint languages such as OCL. In a previous work, SQL was used to perform similar validation on UI models [10]. The CHAINXML Validator, shown in Fig. 7, can be invoked from Cedar Studio to validate and fix help models. Syntactic errors can be avoided by using the provided visual development tool. However, different types of semantic errors of varying complexity can be validated. The validation can detect simple errors such as forgetting to assign a property, e.g., text, on a help model element. It can also detect more complex errors pertaining to the illogical ordering or use of certain elements within a scene. The following are two examples of validation operations that can be applied to help models: 1. SceneElements associated with UI elements that were deleted due to changes in requirements, e.g., a marking that is associated with a deleted text box 2. Illogical ordering of SceneElements, e.g., highlighting a text box before moving the cursor to it An excerpt of the SQL code that is used for realizing example 2 is presented in Listing 5. Validation SQL queries enable Cedar Studio to display errors and warnings based on the retrieved erroneous SceneElements. In case the validation needs to query the UI representation, which is the case of example 1, a model-driven UI representation is needed. UIs that are not model-driven can be reverse-engineered as shown in a previous work [70]. The reverse-engineering process is capable of generating complete UI models that can be viewed and modified using Cedar Studio. Help models can be developed for these UI models, which are dynamically presented to the end-users as working UIs. SQL queries can also be used for fixing errors by modifying help models. The SQL query presented in Listing 6 fixes the erroneous SceneElements retrieved by the validation query presented in Listing 5. In case the validation query detected errors, the developers can decide whether or not to automatically fix these errors by executing the related model fixing query. Although the validation and fixing technique is not directly related to runtime adaptation, it can circumvent human errors that might occur when help models are being developed and maintained.
AC
CE
PT
ED
M
AN US
possible to add support for other scripting languages or dedicated transformation languages in the future. The CHAINXML Presenters discussed in Section 5.1 are used to interpret the concrete help models and transform them to running contextual help. The mapping between the help models and the UI, depends on the nature of the latter. In case the UI was defined using Cedar Studio, then it follows the modeldriven approach that was presented in previous work [69]. In this case, the mapping is done between each help model element and its corresponding UI element at the appropriate level of abstraction. The abstract help model elements are mapped to their corresponding AUI element. As shown in Fig. 2, there is a one to many relationship connecting the AbstractUIElement to the AbstractHelpOperation. The mapping model in this case maps the instances of the participating classes using their ElementIDs. The concrete help model elements are mapped to their corresponding counterparts on the CUI. As shown in Fig. 3, there is a one to many relationship connecting the ConcreteUIElement to the SceneElement. The mapping model in this case also maps the elements based on their identifiers. Following the Cedar Architecture [69], UI models are interpreted, adapted, and presented at runtime. The help models presented in this paper follow a similar interpreted runtime model-driven approach and are presented by the CHAINXML Presenters over the FUI in order for the help to adapt with the UI. In case the UI was not model-driven, then the CHAINXML Presenters directly map the concrete help models to the FUI. There several advantages for adopting a model-driven approach for both the UI and its help. These advantages include: technology independence, supporting multiple modalities, and being able to validate whether a help model still conforms to a UI as the latter is modified due to changes in requirements.
Listing 6. Example of SQL-based Help Model Fix (Excerpt) Fixing Illogical Ordering by Swapping the Sort Order of a Marking and a Mouse Movement Scene-Element Pair
CR IP T
Listing 5. Example of SQL-based Help Model Validation (Excerpt) Illogical Ordering – Highlighting an Element before Moving the Cursor to it
13
5.3 Validating and Fixing Help Models Performing validation is necessary for uncovering errors, which might exist in help models due to human mistakes or changes that have occurred to the UI over time. As shown by the architecture in Fig. 7, the help models are stored in a relational database. This design choice makes it is feasible to perform model validation using SQL queries, which embody logic for validating if a help model has errors and retrieving these errors as relational data. The main reason for using SQL in this case, is that developers are generally more familiar with it than they
6 MAINTAINING HELP USEFULNESS AS UIS ADAPT This section provides an explanation with a few examples on how CHAIN help remains useful for end-users as UIs adapt. This explanation covers how the CHAINXML Pre-
ACCEPTED MANUSCRIPT 14
Pierre A. Akiki / The Journal of Systems and Software
(b) Customer UI Adapted Version (V2)
AN US
CR IP T
(a) Customer UI Initial Version (V1)
(c) Examples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
M
Date of Birth
The UI element is hidden. Hence, the help is not shown.
ED
Billing Cycle
Adapted Version (V2)
PT
Notes Formatting Toolbar
CE
The UI element is hidden. Hence, the help is not shown.
Fig. 8. Examples of UI Adaptation Operations that can be Accommodated by CHAIN Help Models while Remaining Useful
AC
senter presents the help over initial and adapted UIs and the abilities of CHAIN help to work with adaptive UI. UIs can adapt in different ways. Feature-set minimization and layout optimization are two types of UI adaptation [10]. The first type removes or disables UI elements that are not required in a particular context-of-use, while the second type adapts UI element properties such as size and grouping. The UI models are kept alive at runtime in order to perform the adaptation. The UI adaptation approach is comprehensively presented in another paper [69]. As shown by the class diagram in Fig. 3, SceneElements are associated with ConcreteUIElements. The CHAINXML Presenter maintains this association when the UI adapts. It plays a key role in enabling CHAIN help to work with a wide variety of UI adaptations. Presentation technologies support identifiers for UI elements. Two examples are “id” in HTML and “Name” in WPF. The CHAINXML Presenter obtains a ConcreteUIElement by its identifier (UIElementID) and
presents its related SceneElement. Hence, help scenarios are presented in a way that is independent from UI element properties that might change. A number of UI adaptation operations were explored to test the ability of CHAIN help models to adapt as the UI adapts. This exploration was done before conducting evaluation studies with participants. The adaptations were applied to prototype data entry UIs for managing: bank accounts, customers, inventory items, and sales invoices. The UIs were adapted in ways that could potentially occur in real-life enterprise applications. Applying adaptations to prototype UIs allowed the assessment of the help‟s adaptation alongside the UI. This assessment showed that CHAIN help remained useful when various UI adaptation operations were applied. What follows are examples of these adaptation operations and an explanation of how CHAIN help adapts accordingly. Nonetheless, other adaptation operations are also possible. Each operation is related to one or
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
tual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
ED
M
Date of Birth
PT
Billing Cycle
Notes Formatting Toolbar
CE
The UI element is hidden. Hence, the help is not shown.
AC
Fig. 8. The “Notes” text-box in the adapted version of the UI has additional buttons above it showing functionality that could have been accessed in the initial version through a context-menu. The “Date of Birth” example presented in (c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – c, shows a callout displayed next to the date of birth field in the initial UI. This callout is not displayed in the adapted UI, because the date of birth field is hidden. The inverse is shown in the “Notes Formatting Toolbar” example ((c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
AN US
6.1 Hiding or Disabling Widgets Widgets that are not required in a particular context-ofuse can be hidden or disabled in order to reduce UI bloat. For example, in enterprise applications, complex UIs can be simplified by hiding or disabling the fields that are not required by users with a certain role [10]. Another example is hiding unnecessary items from a bloated menu like the one in older versions of Microsoft Word [72]. Furthermore, adapting the accessibility-of-functions and information-density of a UI can also lead to widget hiding and disabling [73]. An example of adapting the accessibility-of-functions is shown in (c) Examples of Contex-
Notes Formatting Toolbar
CR IP T
more of the three context-of-use dimensions: user, platform, and environment. For example, the icons and colors can be adapted based on each end-user‟s culture [51]. Features can be hidden or disabled, for example, based on each end-user‟s job within an enterprise or computer literacy level [10]. The layout grouping and the widgets‟ types, sizes, and positions can be adapted based on the selected platform and each end-user‟s motor abilities [9]. The end-user‟s distance from a screen or mobility within a physical environment can also affect the layout grouping and widget sizes and positions [71].
15
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown. Adapted Version (V2)
Fig. 8. – c), where a callout is displayed over the formatThe UIUI. element is hidden. ting toolbar in the adapted This toolbar is not shown Hence,the thenotes help is notbox shown. in the initial UI, because text has a lower accessibility of functions. Hence, the callout does not appear in the initial UI. However, the formatting toolbar is part of the initial UI, but it is merely hidden. This example shows that hiding elements does not prevent a CHAIN scenario from being properly presented. The CHAINXML Presenter uses UIElementIDs to check whether a ConcreteUIElement was disabled, removed, or made invisible. For example, the JavaScript method “document.getElementById ('id')” and the HTML/CSS property “visibility” can be used to perform checking whether or not a UI element is visible. The CHAINXML Presenter disregards scene elements that are related to hidden or removed UI elements. For example, assume that a TextInput Action SceneElement is specified on a text box that gets hidden or disabled by a UI adaptation. In this case, the CHAINXML Presenter simply moves to the next SceneElement. Similarly, Animation, Creation and Speech, SceneElements are skipped if they are applied to a ConcreteUIElement that gets hidden by a UI adaptation. The Adapted Version (V2) ability of the CHAINXML Presenter to accommodate UI adaptations that hide or disable UI elements, allows it to Thethe UI contextual element is hidden. present end-users with help part that is Hence, theEnd-users help is notwill shown. relevant to the adapted UI. not have to see, for example, an explanation of a UI element that is no longer visible in their version of the UI. Here, the elements are removed at runtime for a particular context-of-use. In case some elements were removed at design-time for all contexts-of-use, validation and fixing can be applied to completely remove the associated help model SceneElements as discussed in Section 5.3.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
6.2 Changing the Layout Grouping Changing the layout grouping is performed for several reasons such as the accommodation of different screen sizes [9]. The UIs shown in Fig. 1 provide an example of a layout optimization adaptation that switches group-boxes with tab-pages. In Fig. 1 – a, the SceneElements are presented directly because all the related ConcreteUIElements are visible. On the other hand, in Fig. 1 – b, the CHAINXML Presenter first brings into view the relevant ConcreteUIElements by selecting their parent tab-page. Then, it presents the associated SceneElements. In some cases, a UI element such as a menu item cannot be located directly on the UI. In such cases, the path property of the ConcreteUIElement is used to indicate the UI elements that have to be loaded before accessing the target UI element. In all cases, when the layout grouping is changed the CHAINXML Presenter only has to locate the widget using its identifier, bring it into view, and present the relevant SceneElement whether its type is Action, Creation, Speech, or Animation.
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI
M
Initial Version (V1)
ED
Date of Birth
The UI element is hidden. Hence, the help is not shown.
CE
Notes Formatting Toolbar
PT
Billing Cycle
AC
Fig. 8, the widget representing the “Billing Cycle” field was adapted from a combo box ((c) Examples of Con-
textual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – b). The “Billing Cycle” example presented in (c)
AN US
6.3 Changing Widget Types One example of this adaptation is changing the type of the data selection widgets to accommodate different enduser motor abilities [9]. In the example shown in (c) Ex-
Fig. 8. – a) to a radio-button group ((c) Examples of
CR IP T
16
The UI element is hidden. Hence, the help is not shown.
Examples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Adapted Version (V2)
The UI element is hidden. Hence, the help is not shown.
The UI element is hidden. Hence, the help is not shown.
Fig. 8. – c, shows that in the initial UI the billing cycle combo box has both a callout displayed next to it and a marking (border) placed around it. We can see that adapting the combo box to a radio-button group, does not prevent the callout and the marking from being displayed. This is possible because both the callout and the marking are displayed with respect to the boundary of the target UI element whether it is a combo box, radio-button group or another type of widget. Adapted Version (V2) When changing a widget‟s type, the assumption is that the widget is being substituted with an alternative The UI element is hidden. that has an equivalent role. For example, a selection Hence,box the can helpbe is not shown. with a widget such as a combo substituted radio-button group or a list box. The post-adaptation type of a widget is identified by the CHAINXML Presenter (e.g., using the “GetType()” method on objects in C#) in order to decide how to present the SceneElements. Assume that ValueChange, Callout, and Marking SceneElements were specified on a combo box that was adapted to a radio-button group. The CHAINXML Presenter is able to present the SceneElements on the adapted widget if its identifier (UIElementID) does not change. First, the
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
Callout and Marking are positioned next and around the radio-button group respectively. Then, the radio button representing the value specified by the ValueChange action is selected. A similar example is shown in (c) Ex-
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
17
Notes Formatting Toolbar
The UI element is hidden. Hence, the help is not shown.
Fig. 8 – a, was relocated next to the “Personal Details” group-box as shown in Adapted (c) Examples Version of (V2)Contextual
Help as Shown on the Initial (V1) and Adapted (V2) Customer UI
Date of Birth
Billing Cycle
Notes Formatting Toolbar
Notes Formatting Toolbar
AN US
The UI element is hidden. Hence, the help is not shown.
Billing Cycle
CR IP T
The UI element is hidden. Initial Version (V1) Hence, the help is not shown.
Date of Birth
ED
M
Fig. 8 – c (billing cycle). Assume that a click MouseCommand was set on a checkbox that got adapted to a switch button (two-state checkbox alternative). The CHAINXML Presenter sends the MouseCommand to the new widget in order to switch its state just like the checkbox would have done. Speech SceneElements are presented in the same way for the different widget type. The only constraint is that the speech should, as a best practice, refer to the concept represented by a widget rather than the widget‟s type, e.g., “select a monthly billing cycle” rather than “click the monthly billing cycle radio button”.
PT
6.4 Resizing or Relocating Widgets With layout optimization, widgets can be resized or relocated for many reasons including filling the space that was left after hiding unnecessary widgets. For example, the “Financial Information” group-box, shown in (c) Ex-
CE
amples of Contextual Help as Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1)
AC
Date of Birth
Billing Cycle
The UI element is hidden. Hence, the help is not shown.
Fig. 8 – b. Hence, it occupied the space that was created by hiding the “Contact Information” group-box. Also, as shown in (c) Examples of Contextual Help as
Shown on the Initial (V1) and Adapted (V2) Customer UI Initial Version (V1) Date of Birth
Billing Cycle
Notes Formatting Toolbar
The UI element Adapted Version (V2)is hidden. Hence, the help is not shown.
The UI element is hidden. Hence, the helpwas is not shown. Fig. 8 – b, the “Notes” group-box resized to occupy the space that was created after relocating the “Financial Information” group-box.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
AN US
CR IP T
18
Fig. 9. CHAIN Components Work with Legacy Applications whether the Source Code is Available or Unavailable
Fig. 10. CHAIN Help Models Presented on Existing Desktop and Web Applications (a) OFBiz: the source-code is available, (b) Excel: the source code is not available
AC
CE
PT
ED
M
CHAIN help model Scene Elements are not affected by this type of adaptation, because the presentation is done with respect to the on-screen size and position of the ConcreteUIElements. The CHAINXML Presenter is able to locate a ConcreteUIElement using its UIElementID and thereby determine its post-adaptation location and size. For example, Markings are presented around the new borders of a widget, and Callouts are positioned with respect to the new position (left, right, top, and bottom) of the target widget. After determining the new location and size of a ConcreteUIElement, the CHAINXML Presenter is able to present Actions such as: TextInput and ValueChange (if the UI element type supports these operations), MouseMovement, MouseCommand, and KeyboardCommand. Animation SceneElements, whether CharacterAnimation or Magnification, depend on locations and paths that involve points on the UI. These points could be fixed, e.g., show a character in the corner at point (0, 0). Yet, these points could also depend on the location of certain ConcreteUIElements, e.g., to magnify a specific widget. In the latter case, the relevant UI element(s) is accessed using its identifier by the CHAINXML Presenter in order to obtain the post-adaptation location. Speech SceneElements are not directly affected by the resizing and relocation of UI widgets. Speech can be introduced anywhere in a Scene and is presented in the same way in both initial and adapted UIs. The CHAINXML Presenter just brings the relevant UI part into view, before presenting a speech SceneElement.
6.5 Changing Colors, Icons, and Fonts Adapting a UI‟s colors, icons, and fonts does not always have a direct effect on the way CHAIN Scenarios are pre-
sented. Changing the font-size of ConcreteUIElement might cause its size to change as well. Nonetheless, this change can be accommodated as explained in Section 6.4. Since CHAIN help is contextual, it remains useful when colors and icons are adapted without needing additional measures from the CHAINXML Presenter. On the other hand, static help causes discrepancies in such cases.
6.6 Direct and Indirect UI Adaptations Adaptive UIs can perform context-dependent adaptations upon loading (indirect adaptation) or while the end-user is working (direct adaptation). Performing an indirect adaptation and offering the adapted UI as an alternative can reduce user confusion [72]. However, in some cases, e.g., ubiquitous applications, it may be necessary to perform a direct adaptation while the user is working. For example, MASP detects changes in the environment and adapts the UIs of smart home systems accordingly [71]. Since CHAIN help models are interpreted and presented at runtime, UI adaptation operations such as the ones described in the previous sections can be detected in both direct and indirect adaptations. Hence, the help is presented in accordance with the adapted UI. CHAIN help models do not contain elements that are related to specific UI adaptation operations. This makes CHAIN useful in a wider variety of scenarios.
7 USING CHAIN WITH LEGACY SYSTEMS Olsen notes in his paper on evaluating UI systems research that legacy code can be a barrier to progress; hence rewriting applications is sometimes the price of progress
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
AC
CE
PT
ED
M
AN US
7.1 When there is Access to the Source-Code When developers have access to the source-code of a legacy software system, CHAIN help models can be leveraged by simply adding a few lines of code to integrate CHAIN‟s client components. These components become part of the application as shown in Fig. 9 – a. The ServiceAccess component calls a web-service to load help models, which are then presented by the CHAINXML Presenter. JavaScript examples of the lines-of-code to be added are shown in Listing 3 and Listing 4. The example in Fig. 10 – a shows a web-based and open-source ERP system called Apache Open for Business (OFBiz) [74]. OFBiz uses HTML to represent its UIs. In a previous work [70], the UI adaptation mechanism RBUIS [10] was integrated into OFBiz. RBUIS was used to adapt the “Product Store” UI of OFBiz by reducing its features and changing its layout grouping from collapsible group-boxes to tab-pages. The same CHAIN Scenario was presented over both the initial and adapted versions of the UI. The running Scenario shown in Fig. 10 – a, has Callouts that explain the purpose of the “visual theme” field. This field is presented as a combo box in the initial UI and as a set of radio buttons in the adapted UI. Nonetheless, the CHAINXML Presenter is able to show the field‟s respective Callout underneath it regardless of the adaptation.
7.2 When there is No Access to the Source-Code In some cases, there is a need to create help scenarios for a software system, without having access to its source-code. For example, IT personnel may need to create help Scenarios for commercial software such as enterprise resource planning (ERP) systems adopted by their companies. Another example can be general purpose software tools such as spreadsheet applications and email clients, which are used in many modern offices. An approach was devised for allowing the CHAINXML Presenter to function without requiring access to the target application‟s source-code. This approach uses existing UI automation libraries to support the presentation of CHAIN help models on top of the UIs of both desktop and web application. With UI automation, it is possible for a software application to access and manipulate the UI elements of another application using an Application Programming Interface (API). Hence, this technique can be used to present help models over the UIs of legacy applications. Some UI automation solutions are the outcome of academic research, while others are commercial or open-source products. One example from the literature is Sikuli, which was included in Section 2.3 under pixel analysis [28]. PUMA is another UI automation system that was used for dynamically analyzing mobile applications [75]. As shown in Fig. 9 – b, the Mediator hosts Client-Components and third-party UI automation components. This mediator has a Request Listener that relays help requests from the target applications to the CHAINXML Presenter, which in turn sends commands to the UI automation components. The Mediator and Presenter were developed using C#. The following paragraphs offer more details on how UI automation is used with CHAIN. Microsoft‟s UI Automation framework [76] was used to present help models over the UIs of desktop applications. This framework was chosen due to its compatibility with C# and its advanced capabilities. For example, some UI testing libraries such as TestStack [77] are based on it. The UI Automation framework can retrieve UI information. One example is retrieving a widget‟s position, which is useful for highlighting a widget or displaying callouts next to it. This framework also supports controlling UIs using commands such as mouse-clicks.
CR IP T
[61]. Yet, he also states that it is desirable to provide a new advance while maintaining legacy code. CHAIN aims to achieve the latter, by supporting help models for the UIs of legacy software systems. CHAIN help models can be used with both newly developed and legacy software systems. For the latter, two cases are considered. The first case is when the sourcecode of the legacy software system is available, and the second case is when this source-code is unavailable. The architecture previously shown in Fig. 7 is further detailed in Fig. 9 to show the interaction between the CHAINXML Presenter and the software applications that leverage UI help models. The approach addresses the two previously mentioned cases of source-code availability as explained in Sections 7.1 and 7.2. An example on each case is illustrated in Fig. 10.
19
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
M
AN US
CR IP T
20
ED
Fig. 11. CHAIN Help Model Visual-Design Tool in Cedar Studio
AC
CE
PT
The technique devised for web applications, uses the Selenium WebDriver [33] to access and control HTML pages through the web-browser. It provides access to an HTML page‟s elements and allows the modification of their properties, in addition to the injection and execution of Java Script code on the page. For example, Keyboard and Mouse commands can be sent to particular elements on the page, and Java Script code can be executed to create elements such as Callouts or modify properties such as colors. The need for cross-origin HTTP requests that are blocked due to security constraints is avoided by using the Selenium WebDriver. The SceneElements are presented with respect to ConcreteUIElements that are located using UIElementIDs. When the help models are developed, the UIElementIDs are specified using the same values found in the UIs. These IDs are the UI automation IDs in desktop applications and HTML IDs in web applications. The automation IDs and HTML element IDs are the primary choices since they uniquely identify a UIElement. Yet, other properties, e.g., text and widget type, can serve as plausible Alternative Identifiers (UIElement class in Fig. 3) in case IDs were not specified. The example in Fig. 10 – b shows CHAIN help presented on Excel‟s ribbon UI, which collapses and adapts its shape when the window is resized. The help is demonstrating how Excel‟s “Scenario Manager” tool is accessed. The same help model is used for both adapta-
tions, when the ribbon is expanded or collapsed.
8 TOOL SUPPORT A tool was developed to support technical stakeholders, e.g., developers and IT personnel, in defining and maintaining CHAIN help models. This tool provides a visual canvas for designing help models using CHAIN‟s visual notation. It can also generate CHAIN help models from macro-like activity recordings, which are done on the running UI using the Selenium IDE [33]. These two features complement each other and facilitate the process of developing help models.
8.1 CHAIN’s Visual Design Tool Cedar Studio is an IDE for developing adaptive modeldriven UIs [12]. It provides visual-design tools for defining UIs on different levels of abstraction including: task models, abstract UI (AUI), and concrete UI (CUI). It also supports visual-design and code editing tools for defining adaptive behavior, which can be executed at runtime to adapt UIs according to the context-of-use. In this paper, Cedar Studio was extended by adding the help model visual-design tool shown in Fig. 11. This tool supports the visual design of help models, which embody the CHAIN concepts presented in Section 3. By using the visual representation, help models can be managed more easily without writing CHAINXML tags
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
21
(b) Selenium IDE Recording Tool
(c) Selenium IDE Recorded File (HTML Format)
(d) CHAIN Help Model (Can be Modified)
11.
AN US
1.
CR IP T
(a) Personal Information UI (Basic Example)
Fig. 12. End-Users can Record their Activities on a UI Using the Selenium IDE (a, b) and then Save the Result (c), which is Automatically Converted to a CHAIN Help Model (d) Using Cedar Studio. The CHAIN Help Model can be modified by Adding More Elements.
AC
CE
PT
ED
M
(e.g., Listing 1). As explained in Section 3, CHAIN supports the representation of help models at two levels of abstraction: abstract (Fig. 2) and concrete (Fig. 3). When designing UIs, it would be better if a designer could manipulate a concrete object rather than its abstraction [78]. Cedar Studio already supports both a bottom up and a top down approach for UI design. Other UI design tools, e.g., Damask [79], support a bottom-up UI design approach. Hence, developers can design the concrete UI, and the abstract UI is generated in the background. This notion also applies to help models and is thereby supported by Cedar Studio. The help model design tool offered by Cedar Studio uses a canvas (Fig. 11 - 5), which visualizes CHAIN Scenes as columns (Fig. 11 - 2) containing rectangles representing SceneElements such as: Callout, Marking, Speech, and so on. Each SceneElement has an icon indicating its type. Icons represent concrete objects (SceneElement) and make diagrams more understandable [80]. Scenes and SceneElements can be added as needed from the toolbox (Fig. 11 - 1). Both Scenes and SceneElements can be dragged and dropped in order to change their order, and their properties can be modified using a property box (Fig. 11 - 6). Cedar Studio‟s visual paradigm is familiar to developers since it is similar to the one offered by commercial IDEs, e.g., Visual Studio and Eclipse. As shown in Fig. 11, Cedar Studio offers a split interface that shows both the concrete UI (Fig. 11 - 3) and help
model design tool (Fig. 11 - 5). This split interface helps in associating UI elements with SceneElements. One example is associating a Callout with a text box. It is possible to define a validation rule to check whether or not a SceneElement is associated with an adequate type of ConcreteUIElement, by using the approach discussed in Section 5.3. For example, it would not be appropriate to associate a TextInput SceneElement with a ConcreteUIElement that does not support typing. In the future, this validation rule could be integrated directly into the visual design tool to prevent non-adequate associations before they occur. This could be done by giving a visual indicator over the ConcreteUIElements that are adequate to choose from, when a SceneElement is selected. An error message could be shown if a wrong choice is made. Help models can be executed and tested against their corresponding UI (Fig. 11 - 4). Cedar Studio interprets and presents both the UI and the help model at runtime.
8.2 Recording CHAIN Scenarios A preliminary evaluation of CHAIN, which is discussed later in the paper, showed that developers would like to use Cedar Studio‟s help model design tool in combination with a recording tool. The latter produces a usage scenario by recording a task as it is being performed on the UI. They thought that the recording tool helps them in quickly creating an initial help model, which they can later adjust using the design tool.
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
CR IP T
22
Fig. 13. Participants’ Experience in the Software Industry
Fig. 14. Time taken by the Participants to Complete Two Tasks Each of which Required them to Define a CHAIN Help Model
*13 Participants, 8 males and 5 females, aged 21 to 30
*There is a noticeable improvement between the Task 1 and Task 2
Fig. 16. Participants’ Preference between Using Cedar Studio to Create Help Models from Scratch or Using it in Combination with a Recording Tool that is Mainly Intended for End-Users
CE
PT
ED
M
AN US
* (n) is the number of participants in each level of experience
Fig. 15. Hand-Drawn Sketch of CHAIN Model (Excerpt) *The icons are annotated to indicate the concepts that they represent in CHAIN’s metal-model.
AC
An existing tool called Selenium IDE [33] can record and playback activities on web-application UIs. This tool can be used to record activities and save them to a file in HTML format as illustrated in Fig. 12 (a, b, and c). An extension was added to Cedar Studio to support importing a Selenium IDE recording and automatically converting it to a CHAIN help model such as the one shown in Fig. 12 - d. This help model can then be modified in Cedar Studio by adding more elements, which provide further explanation on how to use the UI. This possibility overcomes some of the limitations of macro recorders that are mentioned in Section 2.4. It is also possible to specify general settings to configure the conversion process. For example, the general settings can be used to specify if UI elements should be highlighted when clicked, the default color for highlighting UI elements, and so on. In the example shown in Fig. 12 – d, Cedar Studio can automatically generate MouseCommand (click)
Fig. 17. Participants’ Preference between Using Basic Shapes (e.g., circle, triangle, and so on) to Represent Help Model Elements When Conducting Manual Sketches (e.g., on paper or whiteboard), or Using the Same Symbols (Icons) Used in Cedar Studio
and Marking (highlight) elements before the TextInput elements, based on the configuration specified in the general settings. This type of configuration reduces even further the number of elements that are required to make a help model complete. The example shown in Fig. 12 was kept simple for space considerations, but the tools are not limited to simple examples.
9 PRELIMINARY EVALUATION A focus group was held to perform a preliminary evaluation of CHAIN‟s visual notation. Assessing the visual notation is not directly related to CHAIN‟s primary objective, which is adaptive help. Nonetheless, conducting this focus group was essential to obtain input that can be used for tuning CHAIN if necessary, before assessing its technical capabilities in another study.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
CR IP T
Fig. 13 show the general experience of the participants, in addition to their experience in developing UIs and help materials. Their collective experience in the software industry included developing web, desktop, and mobile line-of-business applications related to: banking, retail, supply chain management, and so on. The participants had also developed UIs using different presentation technologies such as: HTML/CSS, Windows Forms, Oracle Forms, and so on. As for developing help materials, the participants who had done work in this area mentioned that they have used different types of tools. Some had used non-specialized help development tools such as Microsoft Word and PowerPoint for developing guidelines and tutorials. Others mentioned using specialized tools such as Snagit and Microsoft Steps Recorder for developing screenshot-based help materials.
9.3 Tasks Given to the Participants The participants were asked to create two help models using the visual notation shown in Fig. 6. They were asked to base these models on snapshots that depict what the outcome would look like if the help was presented over the UIs. The help models were sketched on papers as shown in Fig. 15. Although Cedar Studio was presented during the focus group session, sketching the help models on paper was enough for this preliminary evaluation. The first help model that the participants were asked to create represents the use of a few features from the “Item Maintenance” UI in the Great Plains ERP system [85]. The second model represents launching the scenario manager tool in Microsoft Excel (Fig. 10 – b). Each help model contains three different Scenes, and each of these Scenes contains 2 to 5 elements.
AC
CE
PT
ED
M
AN US
9.1 Study Design Focus groups are carefully planned discussions with a relatively small number of participants. Purposive sampling is usually used to select participants based on their individual characteristics as related to the session topic [81]. The main characteristic that was considered when recruiting participants for this focus group is their expertise in the software industry. The focus group setting enables participants to provide rich information and build on the ideas of others [82]. Focus groups are usually used to collect qualitative data. In this focus group, the discussion was guided to yield qualitative data, and the participants were also asked to answer a few questions that provide quantitative data. The combination of both types of data offers a richer outcome. The session was video-recorded and analyzed later. The focus group session took approximately one hour. In the week prior to this session, the students were introduced to model-driven UI development in a three hour session. Before running the focus group, the participants received an explanation on the different types of CHAIN help model elements. The participants were asked how they would like to approach the construction of a CHAIN help model. One option is to use Cedar Studio‟s visual-design tool, shown in Fig. 11, for building the model from scratch. Another option is to use a recording tool (e.g., Section 8.2) to create an initial help model, which can be extended with more elements using Cedar Studio. The participants were also asked if it would be easier for them to use basic shapes such as squares and circles to represent help model elements when drawing manual sketches. This question assessed the understandability of CHAIN‟s help model element symbols and the ease of drawing them manually if sketching was needed for having reflective conversations (e.g., [83], [84]). The participants were asked to sketch two CHAIN help models, and the completion time was recorded. This task familiarized participants with CHAIN before answering the two previously mentioned questions. It also provided an initial indication about the simplicity and learnability of CHAIN‟s visual notation. The participants had not previously attempted defining help models using CHAIN before being asked to sketch the two help models during this focus group.
23
9.2 Participants The participants in this focus group were 13 Master‟s Computer Science students enrolled in a graduate HCI course at Notre Dame University – Louaize. Out of the 13 participants, 12 are working professionally in the software industry. Hence, these participants are practitioners who hold a Bachelor of Science in Computer Science and are working in the software industry while pursuing their Master degree. The data in
9.4 Primary Results The primary results of this focus group cover the three points discussed in Section 9.1. Task Completion: As shown in Fig. 14, the participants completed tasks 1 and 2 in a mean time of 10.77 (SE=0.95, SD=3.44) and 5.46 (SE=0.47, SD=1.71) minutes respectively. The improvement in the task completion time, between the first task and the second one, can be attributed to the learning curve that was higher in the first task. Eleven out of the thirteen participants (85%) were able to complete the tasks without any mistakes. Two of the participants forgot to add a marking in one case and added a UI Window element in another case when it was not required. Some participants were slightly confused in the beginning between explicitly specifying that a UI element should be highlighted, and simply specifying that the mouse should move to a UI element. This confusion happened with a menu item that can natively highlight itself when the mouse hovers over it. Considering that none of the participants had prior knowledge of CHAIN, these results provide a positive preliminary indication about the simplicity and learnability of CHAIN‟s visual notation.
ACCEPTED MANUSCRIPT 24
Pierre A. Akiki / The Journal of Systems and Software
M
AN US
CR IP T
Desktop Version of Diet Plan User Interface
Fig. 18. Example from the Second Study Showing a Desktop User Interface with Contextual Help
AC
CE
PT
ED
Design Approach: Concerning the approach for designing CHAIN help models, the pie-chart in Fig. 16 shows that 70% (9 out of 13) of the participants answered that they would generally prefer to combine a recording tool with Cedar Studio, because building an initial help model by carrying out a task that is recorded automatically could be faster than manual construction. The other 30% (4 out of 13) of the participants who said that they would like to start by defining the models using Cedar Studio, thought that it is easier to think from the beginning about dividing the help models into Scenes and SceneElements. They also noted that the simplicity of CHAIN‟s visual notation and Cedar Studio‟s visual-design tool would facilitate their work. Visualization: The participants thought that in some cases it could be useful to draw manual sketches for having reflective conversations on what the help model should contain. The pie-chart in Fig. 17 shows that 70% (9 out of 13) of the participants prefer to use the proposed icons rather than basic shapes. They thought that the icons were not hard to identify and draw by hand. They also thought that even though it is simpler to draw the basic shapes, it is easier to mentally associate an icon with a help model element. For example, it is easier to mentally associate a loudspeaker rather than a circle with SynthesizedSpeech. Hence, it is possible to say that the symbols representing CHAIN‟s help model elements are semantically immediate indicating that a novice reader can infer their meaning simply from their appearance [86]. Furthermore, some of the participants (part of the 30%) who initially said that they would pre-
fer to use basic shapes when manually sketching a help model, re-agreed with their colleagues after hearing their rationale about how the icons are easier to remember. In case the icons were not semantically immediate, a text tooltip would have to be added to explain each one. One such case from the literature is the temporal operators of the ConcurTaskTrees [87].
9.5 Suggestions Given by the Participants Some suggestions were given by the participants regarding adding features to Cedar Studio. One participant suggested that it could be clearer if the visual-design tool eliminates the space between the SceneElements that are going to be presented over the UI in parallel. This way the developer can mentally associate these elements with each other. Another participant suggested supporting filtering (search) the SceneElements of a particular Scenario. A third participant suggested supporting the ability to define templates for a Scene. This way, developers can work faster by dragging a Scene template containing predefined Scene Elements. Then, they can modify it as necessary. For example, a template for a Scene that explains the purpose of a UI field can contain: MouseMove, MouseCommand (click), TextInput, Marking (highlight), Callout, and SpeechSynthesis. These suggestions are useful for improving CHAIN‟s tool support and were thereby added to Cedar Studio. The suggestion on adding templates proved to be particularly useful in the study that evaluated CHAIN in actual use and is explained in Section 10.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
25
PT
ED
M
AN US
CR IP T
Mobile Version of Diet Plan User Interface
CE
Fig. 19. Example from the Second Study Showing a Mobile User Interface with Contextual Help (adaptation of the desktop UI)
AC
9.6 Threats to Validity and Limitations The participant sample size (n=13) taken in this study might not be sufficient for claiming the generalizability of the results. Nonetheless, this sample size is considered typical for a focus group study that is intended to provide an indication about a certain product. The tasks that were given to the participants were sketched on paper. Yet, this was enough for getting insights from this preliminary study. A second study was conducted with developers who used Cedar Studio to carry out more detailed tasks. The results of this study included some quantitative data, which are presented in Fig. 14, Fig. 16, and Fig. 17. However, most of the results were qualitative. This is considered normal for a focus group. Nonetheless, other studies can be conducted in the future in order to elicit more quantitative data. Future studies could involve more participants in order to improve the generalizability of the results.
10 EVALUATION THROUGH ACTUAL USE Another study was conducted to evaluate CHAIN by acquiring feedback from software developers who were asked to develop CHAIN help models for real-life adaptive UIs.
10.1 Study Design This study primarily focused on asking developers to evaluate CHAIN‟s strengths and shortcomings after using it to develop help for two sample UIs from a nutrition application that automates dietitians‟ clinics. Each of the sample UIs has two different adaptations, one for the desktop and another for the mobile. The desktop UIs run on Windows using C# and the .Net Framework, while the mobile UIs run on Android using C# and Xamarin. CHAIN was used to create help for the UIs. One help model was developed for each of the two sample UIs, and was used across a UI‟s various adaptations. For example,
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
10.4 Lessons Learned The following are the main strengths of CHAIN that the participants observed and mentioned. The participants thought that the overall approach for creating UI help with CHAIN is simple enough to learn and use. They valued the possibility of composing help by dragging predefined concepts and templates from the toolbox. In some other tools, e.g., video editors, concepts like callouts and markings are created by putting together primitives such as shapes and labels. The ability to develop a single help model that can be presented over multiple adaptations of the same UI was perceived as a main strong point in CHAIN. The participants observed that this feature significantly reduces development time. For example, if video recording was used, multiple videos would have to be recorded. Then, these videos would be imported into a video editing tool like “Adobe After Effects”, in order to be edited and extended with overlays, e.g., callouts. Additionally, a video requires rendering in order to be previewed. This activity takes more time than running the UI from Cedar Studio, which can instantly present it with the respective help model. The participants also perceived that having one model provides better maintainability since changes are only done in one place. The participants appreciated the ability to record UI actions that can be used to automatically generate help models, because this feature also significantly reduces the development time. The appreciation of this feature complies with the result that was obtained in the preliminary study presented in Section 9. In that study, the participants stated that they prefer to create help models by using Cedar Studio in combination with a tool that records activities performed on a UI (refer to Fig. 16). In addition to showing some of CHAIN‟s strong points, this study produced insights, based on the participants‟ feedback, on how to improve CHAIN and Cedar Studio. The following paragraphs explain these insights and offer suggestions on how each point could be potentially addressed. Accessing part of a widget that does not have an ID: CHAIN works on the basis of locating widgets using their identifiers (IDs). However, in some cases there is a need to locate one or more widget parts that do not have IDs. One example is having to explain data presented by widgets such as charts and progress bars. In this case, the part of the widget that shows a particular datum of interest has to be highlighted and explained. Another example is that of multi-record data input using a data grid widget. In this case, the data-grid rows and cells are examples of parts of the data-grid that do not have IDs. A potential solution for this limitation is enabling more communication between the CHAINXML Presenter and the toolkit being used to render the UI. Hence, the parent widget, e.g., data grid, is accessed using its ID and used to access its child UI parts that are associated with help model elements. This can be done by enabling the CHAINXML Presenter to invoke methods on the parent widget that would return the desired child UI parts. For example, a function on the data-grid widget can be called to retrieve a row or a cell. Since different versions of the CHAINXML Presenter are used for each presenta-
AN US
the diet plan UI has two adaptations, namely desktop and mobile, but only requires one help model. The desktop and mobile adaptations of one of the sample UIs (diet plan) that were used in this study are shown in Fig. 18 and Fig. 19 respectively. The diet plan UI allows dietitians to plan the meals that their patients would consume if they were following a particular diet. The help model is presented all at once on top of the UIs in Fig. 18 and Fig. 19 in order to conserve space in the paper. Nonetheless, the Scenes can also be presented sequentially in order to chronologically demonstrate the steps to the end-users. An indirect mode of adaptation (refer to Section 6.6) was used in these UIs. The participants used Cedar Studio to develop and run the help models. These models were developed for the UIs with the least constraints (e.g., a complete set of features). In this study, the desktop versions of the UIs had a full feature-set, while the mobile adaptations had less features. Hence, the help models were developed for the desktop UIs and were used with both the desktop and mobile UI versions. Since the same help models are used for both platforms, the presenter ignores the help model elements that do not apply on the platform with more constraints. For example, a Mouse Movement is ignored on the mobile. Also, certain elements are presented based on what each platform supports. For example, a left button Mouse Command is simply translated to a tap on the mobile.
CR IP T
26
AC
CE
PT
ED
M
10.2 Participants Six software developers participated in this study. The participants included two females and four males, with ages ranging between 23 and 28 years. Two of these participants are doing freelance development on the nutrition software application from which the sample UIs were selected. The other participants are employed at software companies. The work experience of the participants in the software industry has a mean of five years (SE=0.73, SD=1.78) and collectively included developing desktop, web, and mobile line-of-business applications, and mobile games. The participants had also developed UIs using different technologies such as: HTML/CSS, Windows Forms, Windows Presentation Foundation (WPF), and Unity. The participants had also previously developed video tutorials for software applications using video recording and editing tools, primarily “Debut Video Capture” [88] and “Adobe After Effects” [89]. 10.3 Tasks Given to the Participants Each participant completed this study in around one and a half hours. During this time CHAIN and Cedar Studio were introduced and each participant was asked to develop a help scenario for each of the two sample UIs that were selected from the nutrition application. After developing the help models, participants were asked to provide feedback on what they thought were the main strengths of CHAIN and the points that can be improved in this approach and its supporting tool.
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
CR IP T
rently done by an external tool (Selenium IDE), in Cedar Studio for improving productivity. Currently, developers run and test UIs and their related help models from Cedar Studio. Developers run a UI in the browser and use Selenium to record activities, which they can save to a file and import it to Cedar Studio. If a tool like Selenium was incorporated in Cedar Studio, the UI actions would get immediately transformed to a help model without the intermediate steps of saving and importing. Although this is a minor feature, incorporating scenario recording in Cedar Studio could reduce the development time to some extent.
10.5 Threats to Validity and Limitations This study only involved six participants who mostly provided qualitative data. This is not sufficient for claiming a statistically significant result. However, the nature of the provided data was quite rich. Since the participants were software developers, they were able to give insightful feedback that showed the strengths of the approach and highlighted the points that can be improved. This study obtained the feedback of practitioners on the use of CHAIN for developing real-life UI help. The participants expressed their satisfaction with how the help got adapted and its post-adaptation usefulness, based on the two UIs that were used in the study. However, the number of participants and UIs is not sufficient to report a significant assessment. In the future, another study can be conducted to assess the usefulness of the help when it is presented on the adapted UI. This assessment could provide further confirmation of the help‟s ability to accommodate various types of UI adaptation. This point and others can be assessed quantitatively with a larger number of participants (practitioners and end-users) and UIs in order to add a different dimension to the results obtained in this study.
AC
CE
PT
ED
M
AN US
tion technology, adding this feature will not affect the ability to use CHAIN with different technologies. Some software systems, e.g., legacy systems, might not use unique identifiers in their UIs. This was mitigated in CHAIN by supporting alternative non-unique identifiers that might not be optimal, e.g., caption text and widget type, but are still plausible in many cases. Displaying help for external software: A help model is created in CHAIN for a single software application. Yet, in some cases it is useful to run a help model across several applications. For example, in web applications, the help model might need to cover UIs that are part of the browser, e.g., print dialog, as well as HTML UIs that are part of the website. The same case applies to desktop applications when a task requires switching to another application, e.g., an operating system dialog. A potential solution for this limitation is extending CHAIN‟s meta-model to support the development of help models that target multiple applications. Hence, every Scenario would have a target application assigned to it. Each Scenario could then be presented by a version of the CHAINXML Presenter, which works with the target application‟s presentation technology. Executing tasks with side-effects: With help models that are rendered at runtime over the UI, executing some tasks could have undesirable side-effects. For example, it might not be desirable to send an email while demonstrating to an end-user how to work with notifications. Another example can be not wishing to store data in a live database while demonstrating how to perform a type of activity, e.g., diet plan, appointment, patient record, and so on. One way to potentially address this limitation is by supporting annotations over help model Action elements to specify the ones that execute an operation with a sideeffect. Hence, the CHAINXML Presenter can highlight the associated UI element, e.g., button, and wait for the end-user‟s confirmation before proceeding. Displaying scenarios on a timeline: One suggested improvement for Cedar Studio is the addition of a view that represents Scenarios and Scenes on a timeline. Timelines are common in video editing tools like “Adobe After Effects”. Hence, the users of these tools might find timelines to be compact and capable of accommodating more elements on the screen. Timelines can also visually illustrate whether two or more Scenarios are going to run in parallel. This suggestion partially overlaps with a comment given by a participant in the preliminary study (refer to Section 9.5) on eliminating the space between parallel SceneElements. Cedar Studio supports displaying Scene Elements as narrow rectangles when the help model is zoomed out. These narrow rectangles resemble the sprites on a timeline view that is offered by video editors. Nonetheless, it could be useful to add a timeline as an alternative view that resembles video editing tools. Incorporating scenario recording (e.g., using Selenium) in Cedar Studio: Another suggestion was to incorporate the automated scenario recording, which is cur-
27
11 CONCLUSIONS AND FUTURE WORK This paper contributes a novel approach called Contextual Help for Adaptive INterfaces (CHAIN) for developing model-driven contextual UI help, which maintains its usefulness (usability and utility) when the UI adapts. A language called Contextual Help for Adaptive INterfaces eXtensible Markup Language (CHAINXML) is presented alongside a visual notation for representing help models. CHAIN has several advantages over existing help development techniques. The main advantage is that CHAIN help maintains its usefulness by adapting at runtime as the UI adapts. Furthermore, CHAIN help models can be modified to reflect new requirements, instead of having to create completely new help artifacts like videos. It is also possible to validate a CHAIN help model to see if it still conforms to the UI it is associated with and to check for the presence of human errors. End-users can navigate to different segments of a CHAIN help scenario and can interact with it by providing an input and awaiting a response.
ACCEPTED MANUSCRIPT 28
Pierre A. Akiki / The Journal of Systems and Software
TABLE 2 Summary of the CHAINXML Tags Tag Name
Icon
+ChainXML +ResourceDictionary -Style -Text -Image
Description Highest level tag in a CHAINXML document A collection of static and dynamic resources Specifies an element‟s formatting in terms of: color, font, and so on Any alphanumeric string that is displayed or read during a scene Any picture or icon that is used as part of a help model
-DynamicElementCollection
Elements are added at runtime that the help scenario needs to access Helps the end-users in understanding how to perform a certain task Part of a scenario that shows how to perform part of a task
CR IP T
+Scenario +Scene -TextInput
Simulates human-like text input in a UI widget such as a text box
-ValueChange
Changes the value in a UI widget such as a combo box
-MouseMovement
Moves the mouse around a UI at a certain speed
-MouseCommand
Sends specified mouse button clicks to a UI
, Mouse Double Click
, Mouse Up
, Mouse Down
AN US
Mouse Click
-KeyboardCommand
Sends specified keystrokes to a UI
-Marking
Emphasizes parts of a UI, by highlighting or drawing shapes
Highlight Marking
, Square Marking
, Ellipse Marking
Shows a text message in a pop-up balloon
-UIWindow
Loads a helper UI window
-SynthesizedSpeech
Uses a speech synthesis engine to read text
-HumanSpeech
Plays back a prerecorded human voice
-CharacterAnimation
Shows and manages an animated character that can explain the UI
-Magnification
Magnifies an important or recently updated area in the UI
PT
-UnregisterDynamicElement +UserInteractions -UserKeyPress (input) -PresentScene (response)
ED
-RegisterDynamicElement
M
-Callout
Adds dynamically created elements (e.g., drawings) to a collection Removes dynamically created elements from a collection User inputs asking for help and responses to those inputs Detects a user‟s key press (e.g., F1) on part of a UI to ask for help Triggers the presentation of a help scene as an answer to a user input
AC
CE
Legacy software applications were also taken into consideration. Hence, it is possible to present CHAIN help models on the UIs of existing applications, whether or not their source-code is available. If the source-code is available, CHAIN components like the CHAINXML Presenter are integrated as part of the target application. Otherwise, these components become part of a mediator that links CHAIN help models to target applications by using UI automation. Cedar Studio [12] was extended with a visual design tool that supports the development of CHAIN help models. This tool is complemented by the Selenium IDE [33], which is used for automatically recording activities that are performed on a UI. These activities can be saved to a file and imported to Cedar Studio, which automatically converts them to a help model. This work was evaluated in two separate studies. The first study was conducted as a focus group to acquire feedback primarily on CHAIN‟s visual notation. The results of this focus group provided a good indication about the technical qualities of this notation and offered some insights on the expectations of developers. The
second study evaluated CHAIN by asking developers to use it to develop help for real-life adaptive UIs. This study provided insights about CHAIN‟s strengths and highlighted points that inform future work aiming towards improving its existing features. In addition to the points that were highlighted by the second study, CHAIN can be used as a basis for future work targeting a variety of contextual help topics. The following paragraphs provide two examples of these topics. Future work targeting end-user development and usage of help systems, can be based on the outcome of this paper. One interesting direction is providing specialized end-user development tools. Support for crowdsourcing would also be an interesting trait, since it could serve as a means of collaboration and sharing among end-users and expert communities. The work presented in this paper can also form the basis for building help systems that follow an advanced cooperative model, which engages end-users in a dialog with the system. End-users would be able to ask advanced questions about certain points in which they
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
[7]
ACKNOWLEDGMENT
[12]
[8]
[9]
[10]
[11]
AN US
This work is supported in part by ERC Advanced Grant 291652. Many thanks go to all the participants for taking the time to participate in the studies and for their valuable feedback.
[13]
APPENDIX
PT
ED
M
A summary of the tags supported by CHAINXML is provided in Table 2. This summary includes the tag name supported by the XML-compliant syntax, the icon used as part of the visual notation that is supported by Cedar Studio, and a short description explaining the purpose of the tag. When manually sketching a help model using the visual notation, it possible to draw variations of the icons that are presented in Table 2. These variations aim at making the sketching process easier, and include for example drawing only the contour of an icon, tilting it differently, eliminating minor details, and so on.
[3]
[4]
[5]
[6]
G. Calvary, J. Coutaz, D. Thevenin, Q. Limbourg, L. Bouillon, and J. Vanderdonckt, “A Unifying Reference Framework for Multi-Target User Interfaces,” Interact. Comput., vol. 15, no. 3, pp. 289–308, 2003. J. Nielsen, “A model of attributes of system acceptability,” in Usability Engineering, 1st ed., Morgan Kaufmann, 1994, p. 25. M. Kern, F. Trollmann, M. Blumendorf, and S. Albayrak, “Adaptive User Interface Assistance in Smart Environments,” in Proceedings of the Workshop on Meaning and Matching (AISB 2010), 2010. T. Yeh et al., “Creating Contextual Help for GUIs Using Screenshots,” in Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology, 2011, pp. 145–154. J. Matejka, T. Grossman, and G. Fitzmaurice, “Ambient Help,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 2751–2760. B. H. C. Cheng et al., “Software Engineering for Self-Adaptive Systems: A Research Roadmap,” in Software Engineering for SelfAdaptive Systems, vol. 5525, B. H. C. Cheng, R. Lemos, H. Giese, P. Inverardi, and J. Magee, Eds. Springer, 2009, pp. 1–26.
AC
[2]
CE
REFERENCES [1]
P. A. Akiki, A. K. Bandara, and Y. Yu, “Adaptive Model-Driven User Interface Development Systems,” ACM Comput. Surv., vol. 47, no. 1, p. 64:1-64:33, 2014. A. Demeure, G. Calvary, and K. Coninx, “COMET(s), A Software Architecture Style and an Interactors Toolkit for Plastic User Interfaces,” in Proceedings of the 15th International Workshop on Interactive Systems Design Specification and Verification, Kingston, Canada, 2008, pp. 225 – 237. K. Z. Gajos, D. S. Weld, and J. O. Wobbrock, “Automatically Generating Personalized User Interfaces with Supple,” Artif. Intell., vol. 174, no. 12, pp. 910–950, Aug. 2010. P. A. Akiki, A. K. Bandara, and Y. Yu, “RBUIS: Simplifying Enterprise Application User Interfaces through Engineering Role-Based Adaptive Behavior,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 3–12. M. Allbee, “Lawson Smart Office brings WPF Goodness to the Enterprise,” 2008. [Online]. Available: http://bit.ly/1mdhYJ0. [Accessed: 01-Aug-2013]. P. A. Akiki, A. K. Bandara, and Y. Yu, “Cedar Studio: An IDE Supporting Adaptive Model-Driven User Interfaces for Enterprise Applications,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 139–144. H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond Human-Computer Interaction, 2nd ed. Wiley, 2007. C. Plaisant and B. Shneiderman, “Show Me! Guidelines for Producing Recorded Demonstrations,” in Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, 2005, pp. 171–178. “HelpNDoc.” [Online]. Available: http://www.helpndoc.com. [Accessed: 03-Nov-2014]. “Help & Manual.” [Online]. Available: http://www.helpandmanual.com. [Accessed: 03-Nov-2014]. Techsmith, “Snagit.” [Online]. Available: http://snagit.en.softonic.com. [Accessed: 22-Dec-2015]. Microsoft, “Steps Recorder.” [Online]. Available: http://windows.microsoft.com/en-us/windows7/how-do-iuse-problem-steps-recorder. [Accessed: 22-Dec-2015]. “ScreenSteps.” [Online]. Available: http://www.screensteps.com. [Accessed: 03-Nov-2014]. “Add Contextual Help to Salesforce.” [Online]. Available: https://www.youtube.com/watch?v=5aiyuReI7dU. [Accessed: 17-Jun-2015]. D. E. Caldwell and M. White, “CogentHelp: A Tool for Authoring Dynamically Generated Help for Java GUIs,” in Proceedings of the 15th Annual International Conference on Computer Documentation, 1997, pp. 17–22. D. K. Farkas, “The Role of Balloon Help,” ACM SIGDOC Asterisk J. Comput. Doc., vol. 17, no. 2, pp. 3–19, 1993. M. Terry and E. D. Mynatt, “Side Views: Persistent, OnDemand Previews for Open-Ended Tasks,” in Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology, Paris, France, 2002, pp. 71–80. T. Grossman and G. Fitzmaurice, “ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2010, pp. 1515–1524. B. A. Myers, D. A. Weitzman, A. J. Ko, and D. H. Chau, “Answering Why and Why Not Questions in User Interfaces,” in Proceedings of the SIGCHI conference on Human Factors in compu-
CR IP T
require assistance, and the system could respond for example by playing a relevant help Scenario. Hence, CHAIN can be extended to support different types of UserInputs and their related Responses such as questions and answers. Another possible direction is to monitor end-user behavior in order to determine if they require assistance. For example, the help system could detect if an end-user is confused and use the CHAIN help models, which are kept alive at runtime, to propose assistance and provide guidance. Plasticity denotes a UIs ability to adapt to the contextof-use, while preserving its usability [90]. The help developed using CHAIN preserves its usefulness as the UI adapts, since it is presented over the adapted version of the UI. In the future, it would interesting to research the plasticity of UI help when alternate interaction modalities, e.g., voice, tactile, and so on, are considered depending on changes in the context-of-use.
29
[14]
[15] [16] [17] [18]
[19] [20]
[21]
[22] [23]
[24]
[25]
ACCEPTED MANUSCRIPT Pierre A. Akiki / The Journal of Systems and Software
[44]
[45]
[46]
[47]
[48]
[49]
oriented Help,” in Proceedings of the 8th Annual ACM Symposium on User interface and Software Technology, 1995, pp. 181–187. J. Eisenstein and C. Rich, “Agents and GUIs from Task Models,” in Proceedings of the 7th International Conference on Intelligent User Interfaces, 2002, pp. 47–54. E. A. DeKoven, Help Me Help You: Designing Support for PersonProduct Collaboration. TU Delft, Delft University of Technology, 2004. F. García, J. Contreras, P. Rodríguez, and R. Moriyón, “Help generation for task based applications with HATS,” in Engineering for Human-Computer Interaction, Springer, 1999, pp. 149–167. V. Gribova, “Automatic Generation of Context-Sensitive Help Using a User Interface Project,” in Proceedings of the 8th International Conference” Knowledge-Dialogue-Solution, 2007, vol. 2, pp. 417–422. J. Nielsen and R. Molich, “Heuristic Evaluation of User Interfaces,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1990, pp. 249–256. A. Ramachandran and R. M. Young, “Providing Intelligent Help Across Applications in Dynamic User and Environment Contexts,” in Proceedings of the 10th International Conference on Intelligent User Interfaces, 2005, pp. 269–271. D. R. Olsen Jr, S. E. Hudson, T. Verratti, J. M. Heiner, and M. Phelps, “Implementing Interface Attachments Based on Surface Representations,” in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 1999, pp. 191–198. K. Reinecke and A. Bernstein, “Improving Performance, Perceived Usability, and Aesthetics with Culturally Adaptive User Interfaces,” ACM Trans. Comput.-Hum. Interact., vol. 18, no. 2, pp. 1–29, Jun. 2011. K. Mao, L. Capra, M. Harman, and Y. Jia, “A Survey of the Use of Crowdsourcing in Software Engineering,” J. Syst. Softw., vol. 126, pp. 57–84, 2017. P. A. Akiki, A. K. Bandara, and Y. Yu, “Crowdsourcing User Interface Adaptations for Minimizing the Bloat in Enterprise Applications,” in Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, London, UK, 2013, pp. 121–126. A. G. Frey, G. Calvary, and S. Dupuy-Chessa, “Users need your models!: Exploiting Design Models for Explanations,” in Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers, 2012, pp. 79–88. R. Moriyon, P. Szekely, and R. Neches, “Automatic Generation of Help from Interface Design Models,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1994, pp. 225–231. C. Rich and C. L. Sidner, “DiamondHelp: A Generic Collaborative Task Guidance System,” AI Mag., vol. 28, no. 2, p. 33, 2007. C. Rich, C. Sidner, N. Lesh, A. Garland, S. Booth, and M. Chimani, “DiamondHelp: A Graphical User Interface Framework for Human-Computer Collaboration,” in IEEE International Conference on Distributed Computing Systems Workshops, 2005, pp. 514–519. C. Rich, N. Lesh, A. Garland, and J. Rickel, “A Plug-in Architecture for Generating Collaborative Agent Responses,” in Proceedings of the 1st International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2, 2002, pp. 782–789. M. Fowler, UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd ed. Addison-Wesley Professional, 2004. F. Paternò, C. Mancini, and S. Meniconi, “ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models,” in Pro-
AC
CE
PT
ED
M
AN US
ting systems, Montreal, Quebec, Canada, 2006, pp. 397–406. [26] T. Babaian, B. J. Grosz, and S. M. Shieber, “A Writer‟s Collaborative Assistant,” in Proceedings of the 7th International Conference on Intelligent User Interfaces, 2002, pp. 7–14. [27] M. Dixon, D. Leventhal, and J. Fogarty, “Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 969–978. [28] T. Yeh, T.-H. Chang, and R. C. Miller, “Sikuli: Using GUI Screenshots for Search and Automation,” in Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, 2009, pp. 183–192. [29] R. Mansfield, Mastering VBA for Microsoft Office 2013. Sybex, 2013. [30] T. Myer, Apple Automator with AppleScript Bible. Wiley, 2009. [31] “JitBit Macro Recorder.” [Online]. Available: https://www.jitbit.com/macro-recorder. [Accessed: 21-Jun2015]. [32] “Mini Mouse Macro.” [Online]. Available: http://sourceforge.net/projects/minimousemacro. [Accessed: 21-Jun-2015]. [33] “SeleniumHQ.” [Online]. Available: http://www.seleniumhq.org. [Accessed: 10-Oct-2015]. [34] G. Leshed, E. M. Haber, T. Matthews, and T. Lau, “CoScripter: Automating & Sharing How-To Knowledge in the Enterprise,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2008, pp. 1719–1728. [35] T. Babaian and W. T. Lucas, “Leveraging Usage History to Support Enterprise System Users,” in Proceedings of the 14th International Conference on Enterprise Information Systems, Wroclaw, Poland, 2012, pp. 35–44. [36] F. Linton, D. Joy, H.-P. Schaefer, and A. Charron, “OWL: A Recommender System for Organization-Wide Learning,” Educ. Technol. Soc., vol. 3, no. 1, pp. 62–76, 2000. [37] G. Greco, A. Guzzo, G. Manco, and D. Sacca, “Mining and Reasoning on Workflows,” Knowl. Data Eng. IEEE Trans. On, vol. 17, no. 4, pp. 519–534, 2005. [38] Y. A. Chakravarthi, C. Lutteroth, and G. Weber, “AIMHelp: Generating Help for GUI Applications Automatically,” in Proceedings of the 10th International Conference NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction, 2009, pp. 21–28. [39] P. K. Chilana, A. J. Ko, and J. O. Wobbrock, “LemonAid: Selection-based Crowdsourced Contextual Help for Web Applications,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2012, pp. 1549– 1558. [40] G. Stevens and T. Wiedenhöfer, “CHIC- A pluggable solution for community help in context,” in Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, 2006, pp. 212–221. [41] A. García Frey, “Self-Explanatory User Interfaces by ModelDriven Engineering,” in Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Berlin, Germany, 2010, pp. 341–344. [42] A. García Frey, G. Calvary, and S. Dupuy-Chessa, “Xplain: An Editor for Building Self-Explanatory User Interfaces by ModelDriven Engineering,” in Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Berlin, Germany, 2010, pp. 41–46. [43] S. Pangoli and F. Paternò, “Automatic Generation of Task-
CR IP T
30
[50]
[51]
[52]
[53]
[54]
[55]
[56] [57]
[58]
[59] [60]
ACCEPTED MANUSCRIPT Pierre A. Akiki / CHAIN: Developing Model-Driven Contextual Help for Adaptive User Interfaces
[67] [68] [69]
[70]
[71]
[72]
[73] [74] [75]
[76] [77]
CR IP T
[78] A. Demeure, J. Meskens, K. Luyten, and K. Coninx, “Design by Example of Graphical User Interfaces Adapting to Available Screen Size,” in Proceedings of the 7th International Conference on Computer-Aided Design of User Interfaces, Albacete, Spain, 2009, pp. 277–282. [79] J. Lin and J. A. Landay, “Employing Patterns and Layers for Early-Stage Design and Prototyping of Cross-Device User Interfaces,” in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy, 2008, pp. 1313– 1322. [80] M. Petre, “Why Looking Isn‟t Always Seeing: Readership Skills and Graphical Programming,” Commun. ACM, vol. 38, no. 6, pp. 33–44, 1995. [81] J. Kontio, L. Lehtola, and J. Bragge, “Using the Focus Group Method in Software Engineering: Obtaining Practitioner and User Experiences,” in Empirical Software Engineering, 2004. ISESE ’04. Proceedings. 2004 International Symposium on, 2004, pp. 271–280. [82] J. Langford and D. McDonagh, Focus Groups: Supporting Effective Product Development. CRC press, 2003. [83] N. Mangano, T. D. LaToza, M. Petre, and A. Van der Hoek, “How Software Designers Interact with Sketches at the Whiteboard,” Softw. Eng. IEEE Trans. On, vol. 41, no. 2, pp. 135–156, 2015. [84] Q. Chen, J. Grundy, and J. Hosking, “An E-whiteboard Application to Support Early Design-Stage Sketching of UML Diagrams,” in Human Centric Computing Languages and Environments, 2003. Proceedings. 2003 IEEE Symposium on, 2003, pp. 219– 226. [85] “Dynamics GP Inventory.” [Online]. Available: http://bit.ly/1N8bE5x. [Accessed: 23-Nov-2015]. [86] D. L. Moody, “The „Physics‟ of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering,” Softw. Eng. IEEE Trans. On, vol. 35, no. 6, pp. 756–779, 2009. [87] G. Mori, F. Paternò, and C. Santoro, “CTTE: Support for Developing and Analyzing Task Models for Interactive System Design,” IEEE Trans. Softw. Eng., vol. 28, no. 8, pp. 797–813, 2002. [88] “Debut Video Capture.” [Online]. Available: www.nchsoftware.com/capture. [Accessed: 15-Dec-2016]. [89] “Adobe After Effects.” [Online]. Available: www.adobe.com/products/aftereffects.html. [Accessed: 15Dec-2016]. [90] G. Calvary et al., “Plasticity of User Interfaces: A Revisited Reference Framework,” in In Task Models and Diagrams for User Interface Design, 2002, pp. 127–134.
AN US
[66]
M
[65]
ED
[64]
PT
[63]
CE
[62]
AC
[61]
ceedings of the 6th International Conference on Human-Computer Interaction, Sydney, Australia, 1997, vol. 96, pp. 362–369. D. R. Olsen,Jr., “Evaluating User Interface Systems Research,” in Proceedings of the 20th ACM SIGCHI Symposium on User Interface Software and Technology, Newport, Rhode Island, USA, 2007, pp. 251–258. F. Bodar, A.-M. Hennebert, J.-M. Leheureux, I. Provot, J. Vanderdonckt, and G. Zucchinetti, “Key Activities for a Development Methodology of Interactive Applications,” in Critical Issues in User Interface Systems Engineering, D. Benyon and P. Palanque, Eds. London: Springer London, 1996, pp. 109–134. P. Sukaviriya and J. D. Foley, “Coupling a UI Framework with Automatic Generation of Context-sensitive Animated Help,” in Proceedings of the 3rd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology, New York, NY, USA, 1990, pp. 152–166. C.-E. Dessart, V. Genaro Motti, and J. Vanderdonckt, “Showing User Interface Adaptivity by Animated Transitions,” in Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, New York, NY, USA, 2011, pp. 95–104. A. Dingli, Y. Wilks, R. Catizone, and W. Cheng, “The Companions: Hybrid-world Approach,” in Proceedings of the 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, 2009, p. 60. T. R. G. Green and M. Petre, “Usability Analysis of Visual Programming Environments: A „Cognitive Dimensions‟ Framework,” J. Vis. Lang. Comput., vol. 7, no. 2, pp. 131–174, 1996. “Xamarin.” [Online]. Available: https://www.xamarin.com/. [Accessed: 14-Apr-2017]. “Double Agent on SourceForge.net.” [Online]. Available: http://doubleagent.sourceforge.net. [Accessed: 05-Jun-2015]. P. A. Akiki, A. K. Bandara, and Y. Yu, “Engineering Adaptive Model-Driven User Interfaces,” IEEE Trans. Softw. Eng., vol. 42, no. 12, pp. 1118–1147, Apr. 2016. P. A. Akiki, A. K. Bandara, and Y. Yu, “Integrating Adaptive User Interface Capabilities in Enterprise Applications,” in Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India, 2014, pp. 712–723. S. Feuerstack, M. Blumendorf, and S. Albayrak, “Bridging the Gap between Model and Design of User Interfaces,” in GI Jahrestagung (2), Dresden, Germany, 2006, vol. P-94, pp. 131– 137. J. McGrenere, R. M. Baecker, and K. S. Booth, “An Evaluation of a Multiple Interface Design Solution for Bloated Software,” in Proceedings of the 20th SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, USA, 2002, pp. 164–170. K. Reinecke, “Culturally Adaptive User Interfaces,” PhD Thesis, University of Zurich, Zurich, 2010. “Apache Open For Business (OFBiz).” [Online]. Available: http://ofbiz.apache.org. [Accessed: 29-Jul-2013]. S. Hao, B. Liu, S. Nath, W. G. Halfond, and R. Govindan, “PUMA: Programmable UI-Automation for Large-Scale Dynamic Analysis of Mobile Apps,” in Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, 2014, pp. 204–217. Microsoft, “Microsoft UI Automation.” [Online]. Available: http://bit.ly/1Z1xvO5. [Accessed: 13-May-2015]. “TestStack/White.” [Online]. Available: https://github.com/TestStack/White. [Accessed: 13-May2015].
31
ACCEPTED MANUSCRIPT 32
Pierre A. Akiki / The Journal of Systems and Software
AC
CE
PT
ED
M
AN US
CR IP T
Pierre A. Akiki is an assistant professor of computer science at Notre Dame University – Louaize, Lebanon. He received a Ph.D. in Computing from The Open University, U.K. in 2014. His research interests include adaptive model-driven interactive software systems, enterprise applications, and the internet of things. His work on adaptive model-driven user interfaces was published in several venues including IEEE TSE, ACM Computing Surveys, and ICSE, and it received the best paper award at ACM SIGCHI EICS’13. Further information about Pierre's research work and publications can be found at: http://www.pierreakiki.com.