- Email: [email protected]
Adapting applications in handheld devices using fuzzy context information
Interacting with Computers 15 (2003) 521–538 www.elsevier.com/locate/intcom
Adapting applications in handheld devices using fuzzy context information Jani Ma¨ntyja¨rvia,*, Tapio Seppa¨nenb a
VTT Electronics, Technical Research Centre of Finland, P.O. Box 1100, Oulu FIN-90571, Finland b University of Oulu, P.O. Box 4500, Oulu FIN-90014, Finland
Abstract Context-aware devices are able to take advantage of fusing sensory and application specific information to provide proper information on a situation, for more flexible services, and adaptive user interfaces (UI). It is characteristic for handheld devices and their users that they are continuously moving in several simultaneous fuzzy contexts. The dynamic environment sets special requirements for usability and acceptance of context-aware applications. Context-aware applications must be able to operate sensibly even if the context recognition is not 100% reliable and there are multiple contexts present at the same time. We present an approach for controlling context-aware applications in the case of multiple fuzzy contexts. This work has several potential applications in the area of adaptive UI application control. Our study is focused on the adaptation of applications representing information in handheld devices. The design of controllers and experiments with real context data from user scenarios are presented. Experimental results show that the proposed approach enhances the capability of adapting information representation in a handheld device. User reactions indicate that they accept application adaptation in many situations while insisting on retaining the most control over their device. Moreover, user feedback indicates that abrupt adaptations and instability should be avoided in the application control. q 2003 Elsevier B.V. All rights reserved. Keywords: Context aware computing; Context recognition; Mobile devices
1. Introduction Context-aware computing has been proposed to enhance the usability of handheld devices such as personal digital assistants (PDAs) and mobile phones by enabling the adaptation of various applications and UIs according to the context of a device and its user. The development of context-aware properties for handheld devices requires sensing of * Corresponding author. E-mail addresses: [email protected] (J. Ma¨ntyja¨rvi), [email protected] (T. Seppa¨nen). 0953-5438/03/$ - see front matter q 2003 Elsevier B.V. All rights reserved. doi:10.1016/S0953-5438(03)00038-9
522
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
a physical environment and the device’s user via low cost on-board sensors, exploiting context information sources located in the Internet, as well as acquiring terminal specific knowledge of applications and operating networks. The exact definition of context has so far been elusive but has been described as “Any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves” (Dey and Abowd, 2000). This definition of context, however, remains abstract, as the abstraction in the definition of context has been shifted to the definition of information. The definition does not indicate how context can be processed. Neither does it indicate how the relevancy of context for applications can be acquired. Exploiting context-awareness in handheld devices set challenges for system architectures since there are several feasible context information sources, such as sensors, location, applications, operating networks, and sources located over the Internet. Some of these sources might be distributed over various communication networks while some sources may be located on-board, for example, sensors. Due to the device’s mobility characteristics, a context information obtained from these sources is highly dynamic, thus making the exploitation of context information difficult. Salber et al. (1999) introduced an idea of a context toolkit for supporting the development of context-aware applications. Their architecture is reported to handle a distribution of components, dynamic context data and communication between heterogeneous components. The architecture introduces context widgets mediating between the environment and application. Widgets handle the sensing of the environment and they provide context information for applications. Other architecture solutions for various needs have been reported. The study made by McCarthy and Anagnost (2000) presents a tool for supporting peripheral awareness in workplaces. The tool allows users to be notified when a predefined event occurs. In this case a user sets a triggering value for context, not an application designer. An important issue in processing context information from sensors is how to obtain relevant context information. Handheld devices and their users are continuously moving in several simultaneous and overlapping contexts. A number of research efforts concerning the development of context recognition have been carried out. The context recognition from multidimensional sensor signals can be carried out, for example, using algorithms based on neural networks (Schmidt et al., 1999). Signals from multiple sensors can be processed into multidimensional context vectors, from which relevant context information can be extracted and compressed by using statistical methods (Himberg et al., 2001). To extract higher-level descriptions of contexts, the time series segmentation and clustering methods can be utilized (Ma¨ntyja¨rvi et al., 2001; Flanagan et al., 2002). In modeling of context states over time to provide most probable current context Hidden Markov Models (HMMs) can be exploited. Context recognition methods for wearable computers have been studied with the means of a wearable camera, environmental audio signal processing and HMMs (Clarkson et al., 2000). There are several studies concerning implementation of sensing systems for handheld devices to facilitate the development of context-aware applications. It has been proposed that there should be triggering mechanisms for executing context-aware applications. The triggering should occur for example, when sensor values match values defined in the triggering conditions (Brown, 1998). A similar approach has been presented in
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
523
the work of Schmidt et al. (Schmidt et al., 1999). They propose scripting primitives for performing three types of actions: when entering, leaving and while in a context. Their approach also exploits triggering mechanisms. Moreover, their work presents the implementation of several sensors into a cell phone and into a PDA to recognize the context of a handheld device and its user. The study by Hinckley et al. (2000) presents the integration of sensors into a PDA for developing novel applications and for examining implementation. Both studies, by Hinckley et al. and by Schmidt et al., propose that carefully implemented sensor fusion into handheld device provides a direct sensor-based application control and potential sources for context information. A central problem in the development of context-aware systems is how to compose the context representation. Moreover, there exists a lack of knowledge about the structure of context information in changing environments. It has been proposed that separation of how context is acquired from how it is used, and categorization of context information might help the development of these systems (Dey et al., 2001). More information for studying context information might be obtained by examining humans’ behavior in social contexts. For example, we lower our voice in public libraries and during lectures, and yell loudly in soccer games. Our actions are based on continuously learned adaptation models. Humans’ perceptions from environment are fuzzy. We recognize quantities of our environment to be for instance, loud, bright, warm, etc. It is reasonable to assume that the development of context dependent behavior for handheld device applications might benefit from fuzzy context representation. Due to the uncertainty related to context information, the formulation of exact definition explaining how to process context data and how to define its relevancy to applications has so far been elusive. The use of fuzzy context representation introduced in this paper provides a method to exploit sensor-based context information in handheld device applications. Moreover, by utilizing fuzzy logic methods for processing context information, the fuzzy control methods can be exploited in designing control mechanisms for adapting various applications and UIs. Dynamic environments set special requirements related to usability issues and social acceptance of novel functionalities of context-aware handheld device applications that must be able to operate according to multiple fuzzy context information with a minimum misinterpretation of current contexts. For context information extracted from sensor signals describe approximations of current contexts at best, the use of context information in application control still maintains the risk for launching applications in unsuitable situations and applications to behave in an unwanted manner. In most of the studies concerning the development of context-aware systems, the action performed based on context information is an execution of an application or a feature of an application. Usually, the execution is initiated by triggering. However, in some cases, it would be feasible to provide continuous context-based control signals. In this paper, we experiment with context information to control handheld device applications. We concentrate on one particular category of context sources located in handheld devices— sensors. An approach for adapting applications according to fuzzy context information is introduced. In our approach context representation is obtained by applying fuzzy membership functions. We show that the use of a fuzzy control for adapting applications according to fuzzy context representation benefits from the uncertainty in context information produced by
524
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
context recognition, thus more flexible continuous behaviour of applications can be achieved. A control mechanism is presented as a part of system architecture designed for handling context information. Section 2 explains how context representation is composed, and how the application adaptation is carried out by directly controlling different features of UI applications. The adaptation exploits fuzzy logic controllers (FLCs) in handling multiple fuzzy contexts. In Section 3 the design of FLCs is presented. The performance of the application adaptation is examined with concrete examples with real user data in Section 4. User experiences about the presented method are also provided. Section 5 concludes the work.
2. Fuzzy control of applications This section explains a procedure for converting raw signals obtained from sensors into a context representation, and a framework for fuzzy control of context-aware applications. 2.1. Fuzzy sets The theory of fuzzy sets defines the relations and operations between variables and fuzzy sets. Both basics and advanced explanation of fuzzy sets can be found, for example in (Cox, 1998; Zadeh, 1965). Fuzzy logic control has been widely used for control purposes in various fields of research. A good overview of fuzzy sets and systems is provided in (Cox, 1998). In the concept of fuzzy set a variable can be considered as an input x; which is related to a fuzzy set A by a membership function m: The relation is expressed as mA ðxÞ: The value of membership function can be something between zero and one, mA ðxÞ [ ½0; 1: Zero denotes a null membership and one denotes a full membership. An example of fuzzy sets with membership functions related to a variable x is presented in Fig. 2. Basic operations for fuzzy sets are complement, intersection and union. The membership of the complement of a fuzzy set A is
m:A ðxÞ ¼ 1 2 mA ðxÞ
ð1Þ
When defining fuzzy rule using logical operation NOT, the membership of the complement of a fuzzy set is utilized. The membership of the intersesction between two fuzzy sets A and B for aggregating two membership functions can be expressed as
mA>B ðxÞ ¼ minðmA ðxÞ; mB ðxÞÞ
ð2Þ
When defining fuzzy rule using logical operation AND, the membership of the intersection between fuzzy sets is utilized. The membership of the union between two fuzzy sets A and B is
mA
ð3Þ
When defining fuzzy rule using logical operation OR, the membership of the union between fuzzy sets is utilized.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
525
Fuzzy sets form the basics for developing the format for context representation. Using above-mentioned operations several fuzzy sets can be combined to a control signal. The operations of fuzzy sets are used in designing rule bases for FLCs. 2.2. Context representation A central problem in utilizing context information in application development is a difficulty to add meaning of concepts from real world to extracted features. For example, when setting a volume of ringing tone to behave context dependently, it would be easier to use representation: ‘When environment loudness is SILENT, set ringing tone volume LOW’, instead of the representation: ‘When environment loudness is 2 20 dB, set ringing tone volume to 2.5’. The first task in context extraction is to abstract raw sensor signals and compress information by using different signal processing and feature extraction methods. The features to be extracted are chosen according to how well they describe some parts of the real world context. We call extracted features context atoms since they contain the smallest amount of context information. The illustration of the process for increasing the abstraction level of raw context information is presented in Fig. 1. After sampling, raw signals are preprocessed, for instance, in the case of sensor measurements, signal values
Fig. 1. Illustration of context information processing procedure. Context information fusion produces higher-level context information.
526
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
are calibrated and rescaled. Preprocessed signals are used as inputs to various feature extraction methods in time and frequency domains producing features to describe context information. For example, the root mean square (RMS) value of an audio signal describes the loudness of environment audio. We have chosen to apply fuzzy quantization for extracted features resulting in a more expressive representation of context information. Fuzzy quantization can be viewed as a granulation of information (Zadeh, 1996). Two different methods can be used to quantize features to represent context information: (i)
Set crisp limits for features. The result is a logic style true false labeling for fuzzy membership functions. For example, in the recognition of loudness of environment audio, the processed feature is divided into three quantities labeled Silent, Modest, and Loud, corresponding to three membership functions as presented in Fig. 2a. If one of these is true, the others are false. (ii) Apply a fuzzy set for features resulting in a more expressive fuzzy labeling, as Fig. 2b presents. For example: mL ðxÞ ¼ 0.7/Silent þ 0.3/Modest þ 0/Loud. The context information processing scheme combines quantized features into a context atom vector CðnÞ ¼ ½C1 ; C2 ; …; Ck T at each quantization time step n; k ¼ maximum number of context atoms. The values of individual context atoms are between zero and one. The context representation used here has many advantages:
Fig. 2. Examples of crisp (a) and fuzzy (b) quantizations.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
527
(i)
It allows one to compose the ontology for sensor based context information that facilitates the development of context-aware applications, for example, a direct control of various types of applications, as shown in experiments (Section 3). (ii) Multidimensional context atom vector C enables the utilization of e.g. explorative data analysis methods for examining higher-level contexts as Fig. 1 depicts (Himberg et al., 2001; Ma¨ntyja¨rvi et al., 2001; Flanagan et al., 2002). (iii) It is extendable, i.e. it is straightforward to add new context atoms into a vector C, thus extending the ontology for sensor based context information. 2.3. Application control The implementation of context-awareness into a handheld device requires novel approach for developing system architectures to support both heterogeneous sets of context information sources and applications. Several promising approaches have been presented (McCarthy and Anagnost, 2000; Salber et al., 1999). We propose an additional block for handling context-aware applications that utilizes context representation. The approach differs from others because the block consists of FLCs, which are designed to use predefined sets of the elements of context atom vector C as inputs. Usually in architecture models, an application control is located between an application and information source, for instance, a server. The purpose of the application control is to process information from the information source into the functionality of an application. Applications can be controlled according to user context by adjusting various features, such as font size, spatial resolution, size of images, colors and contrast of images shown on a display, and volume and tunes of operating sounds. Thus, the information content and compression of information presented on the display can be controlled according to context of a user. The adaptation of these features improves the capabilities of presenting only the task relevant information to a user, for example, when he/she is browsing a service. We propose an additional block called Context based application control to be inserted into the architecture. The overview of the proposed architecture is presented in Fig. 3. The control scheme is influenced by the context representation, the requirements set by the usability models of applications, and the requirements set by the service information content domain. The Context based application control block resides between the UI application and the source of context information. The block consists of number of FLCs. Each FLC uses a predefined set of contexts as inputs and a particular rule base for handling information from multiple fuzzy contexts. A predefined set of contexts of a particular controller can be defined as a group of the most important contexts from the viewpoint of the functionality of a certain feature of the UI or the application itself. The functionality of the Context based application control block includes the capability to deliver control information for the Application control block and directly for the UI application. This ensures that it is possible to launch applications according to context information, and at the same time, to continuously control the functionality of the features of applications. Service adaptation covers the handling of physical, semantic and functional entities of media content of the service, and it provides functional scaling for the content
528
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 3. Overview of architecture for controlling applications based on fuzzy context information. Service provides information to a terminal with indexed content. ‘Context information’ block consists of context information available in a terminal, e.g. low and higher level context information.
and representation (Metso et al., 1998). The service adaptation can be exploited in representing only task relevant highlighted information according to context. Information provided by a service, for example, a bus timetable, includes information content with representation indexing. The indexing enables context-aware control of the content and the format of service information presented on a display. In representation indexing, the content of a service is indexed with k ¼ 1; 2; 3 information representation levels. Lower levels are sublevels of higher levels. For example, a sentence {A, B, C, D, E, F, G, H, I} is indexed with three levels as {A1, B3, C1, D2, E1, F2, G1, H3, I2}. Thus information representation is {A, C, E, G þ X} on level 1, {A, C, D, E, F, G, I þ Y} on level 2 and {A, B, C, D, E, F, G, H, I þ Z} on level 3. Variables X; Y; Z denote additional context dependent content such as ‘…next bus…‘. The service adaptation can be realized by matching indexing information with appropriate control signals from the FLCs. Adaptation of applications takes place in the Application control block, while the adaptation of application features takes place in the UI application block. 3. Design of fuzzy controllers This section explains the contents of Context based application control block. The design of FLCs for adjusting applications according to contexts is presented. Controlled application features include: Volume of operating tunes, font size of text presented on the display, display illumination, and the compression of information content presented as text on the display. Inputs to controllers are presented in Fig. 4, which shows categorized context variables with labeling. The context types and variables are chosen empirically based on tests with real sensor measurements. The chosen context types turned out to be the most important ones for these controlling purposes.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
529
Fig. 4. Categorization and labeling of context atoms.
3.1. Controller for operating tune volume One FLC is designed for controlling operating tunes. Inputs to the controller are context types ‘User activity’ and ‘Loudness level of environment’. Context type ‘User activity’ contains context atoms ‘Movements’, ‘Walking’, ‘Running’ while context type ‘Loudness level of environment’ includes context atoms ‘Silent’, ‘Modest’, ‘Loud’, as presented in Fig. 4. Moreover, a control variable represented as fuzzy sets is used in the design of the controller. Membership functions of fuzzy sets of context atoms and control variable used in the design of the controller are presented in Fig. 5. The output of the controller guides
Fig. 5. Piecewise continuous trapezoidal membership functions of context atoms used in designing the FLC. (a) User activity, (b) Loudness of environment (c) Volume of operating sounds. Values of membership functions are on the range of [0, 1].
530
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Table 1 Fuzzy rule base for the volume of operating tunes-controller
R1 R2 R3 R4 R5 R6 R7 R8 R9
IF User activity
AND Loudness of environment
THEN Volume of operating tunes
Movements Movements Movements Walking Walking Walking Running Running Running
Silent Modest Loud Silent Modest Loud Silent Modest Loud
Level 1 Level 2 Level 3 Level 2 Level 2 Level 3 Level 3 Level 3 Level 3
one specific feature of a UI application: The volume of operating tunes including ringing tones, message alerts, and other application specific tunes. Membership functions of context features and control variable are used in generating the fuzzy rule base for the controller. The rule base describes how a controller maps combinations of context features into control purposes according to the predefined rules, which are set by using basic operations of fuzzy sets described in Eqs. (1) – (3). The fuzzy rule base for the controller of the volume of operating tunes is presented in Table 1. The fuzzy rule base can be presented as a 3D-decision surface, which is presented in Fig. 6. Inputs to a controller are on the horizontal axes and the output of a controller is on the vertical axis. The decision surface illustrates, for example, that when either ‘User activity’ or ‘Loudness level of environment’ is at a maximum, the volume of operating
Fig. 6. The decision surface of fuzzy sound alert controller. Input features are on the horizontal axes and output of a controller is on vertical axis.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
531
tunes is at maximum. Furthermore, the minimum volume of operating tunes is obtained when both ‘User activity’ and ‘Loudness level of environment’ are at a minimum. 3.2. Controller for information representation An FLC utilizing two inputs and producing two outputs is designed for controlling the representation of information. Inputs to the controller are context types ‘User activity’ and ‘Illumination level of environment’. The context type ‘Illumination level of environment’ contains context atoms ‘Dark’, ‘Normal’, ‘Bright’. In this case there are two outputs from the FLC for adjusting different features of two UI applications: The one is the display illumination of a terminal and the other is the font size of text presented on the display. In addition, the font size control signal is used for guiding the compression of the information content. Information content adaptation is realized by exploiting the service content indexing, implemented in mobile services. Fuzzy rule bases for controlling features of information representation are illustrated as 3D-decision surfaces in Fig. 7a and b. The decision surface in Fig. 7a illustrates the behavior of the font size control. When ‘User activity’ is at a maximum, the font size is large, whereas when ‘User activity’ is at a minimum and ‘Illumination level of environment’ is quite bright, the font size is small. The decision surface presented in Fig. 7b illustrates the behavior of display illumination control. The brightest display is obtained when ‘Illumination level of environment’ is low, and also when ‘User activity’ is at a maximum. The display is always dark when ‘Illumination level of environment’ is high, except when ‘User activity’ is at a maximum. Both controllers are designed to together maintain the maximum contrast of information presented on the display.
4. Experiments and results This section presents experiments with real context data from user scenarios and they demonstrate the adaptive control of information representation application and application features. Moreover, this section presents results and discussion of experiments with the method, and a summary of the examination of user’s reactions to the system. 4.1. Context data Sensor data is logged from a user scenario with a sampling frequency of 256 Hz by using a portable laptop a user carries. Analog signals from accelerometers and illumination sensor are A/D-converted and sampled with National Instruments DaqCard 1200 measurement board, which is connected to a PC-CARD slot in a laptop PC. Audio data is sampled at 16 kHz by using the standard audio measurement board of the laptop. Data is stored in a file by using a measurement program made in LabVieww graphical programming environment. Conversion from raw sensor signals to context atoms is performed according to the procedure described in Section 2. It is carried out by using a self-programmed context recognition software, which produce a context atom vector C ¼ ½C1 ; C2 ; …; C9 T ; for describing two main context categories: ‘Environment’
532
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 7. (a) The decision surface of font size FLC. (b) The decision surface of display illumination FLC. Input features are on the horizontal axes and output of a controller is on the vertical axis.
and ‘User’ and for describing three context types: ‘User activity’, ‘Loudness level of environment’ and ‘Illumination level of environment’. Each context type contains three context atoms resulting in a nine-dimensional context atom vector. The categorization and labeling of context information is presented in Fig. 4. Context atom values are calculated every second and a sliding window with the length of four seconds is used in calculating averaged context atoms to prevent the occurrence of transients in control signals. The experiments with a user scenario are performed offline in a Matlabw scientific programming environment. First, FLCs are designed using a Matlabw Fuzzy Logic Toolboxw that provides graphical UIs to design fuzzy rules and controllers. Secondly, generated FLCs are tested offline with logged context data using a Matlabw Simulinkw simulating environment.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
533
Table 2 Description of the user scenario Segment n:o
Explanation of the action
1
User is at the office, it is late, he is planning to catch a bus from the nearest bus stop, and he is browsing bus timetable Walks along a hallway (browsing bus timetable) Walks down the stairs (browsing bus timetable) Opens the front door, goes out and walks on the street (browsing bus timetable) Notices that a bus is coming, begins to run to a bus stop (browsing bus timetable) Catches the bus, steps in to it, pays, and sits down
2 3 4 5 6
4.2. Experiments with user scenario Performance of designed controllers is evaluated with real context data recorded from a scenario where a user browses a bus timetable service by using a handheld device while he is leaving the office and is trying to catch a bus. A scenario is described in Table 2. The context information extracted from the user scenario recordings is presented in Fig. 8. The duration of a scenario is about three minutes. Experiments with a user scenario are performed as follows: 1. A user scenario presented in Table 2 is recorded and processed into context atom vectors. 2. Inputs are assigned to each FLC. 3. Outputs of the FLCs are gathered. 4. The output of the font size controller is used in adapting the information content of a text presented on the display. An information content indexing of a service is mapped to the control signal of the font size controller.
Fig. 8. Multiple fuzzy context signals from user scenario recordings. Values of context atoms are represented as grey level bars, black means 1 and white means 0 activity. Numbers of segments correspond to certain activities of a user.
534
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 9. Operating tune volume control, display illumination control and font size control signals from the user scenario. Numbers correspond the segments of the scenario. Outputs of FLCs are illustrated with continuous lines. Adapted information contents of bus timetable service with illumination control in different phases of the scenario are also presented.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
535
Fig. 9 presents outputs of FLCs as continuous lines. The numbers of various segments correspond to actions explained in Table 2. Adapted information content of a bus timetable service in different phases of the scenario is also presented. The behavior of control signals, presented in Fig. 9, can be examined by comparing them to user actions expressed as context atoms in Fig. 8. Operating tune volume control signal is at a minimum when a user is inside (Segment 1), that is when ‘User activity’ is partly ‘Movements’ and partly ‘Walking’ while ‘Loudness of environment’ is ‘Silent’. Also, operating tune control signal is at a minimum in some points when a user is in a bus (Segment 6) and it is silent. The control signal starts to rise when a user begins to walk (Segment 2). The control signal rises considerably when a user walks down the stairs (Segment 3), reaching the maximum when a user is running (Segment 5), and in a noisy bus (Segment 6). Signals of font size and display illumination controllers behave in the same manner, except when a user is in a bus (Segment 6) when the font size control is at a minimum. This is because the ‘User activity’ is at a minimum that is, ‘Movements’. Display illumination control is a minimum when the user is under the bright light when going outside (between Segments 2 and 3). In the adapted information content of the bus timetable service the basic information representation (Level 3 representation) is provided when font size is small. That is, when ‘User activity’ is ‘Movements’. Correspondingly, the most compressed information representation (Level 1 representation) is provided when ‘User activity’ is ‘Running’. It demonstrates the information representation adaptation of service content that combines the benefits of the content and representation indexing of a service and context based application control in a terminal. It is required that the indexing is delivered via the service, or a terminal is able to index the content. Furthermore, in the screenshot showing the Level 1 information representation (Fig. 9), the one-touch dial to next bus timetable is presented. This functionality enables user to browse next bus timetable. Control signals indicate that the applications are able to adapt to the current context of a user’s handheld device by using FLCs and chosen context information. However, control signals tend to fluctuate when the values of individual contexts change rapidly. The variability of context signals can be considered as the main problem of a system’s performance since it is typical in rapidly changing environments, in which handheld devices operate. The fluctuation of control signals may lead to oscillation of various information representation levels on a display. The fluctuation of control signals can be eliminated using moving averaging methods for smoothing the context information or in this case, using smoothing in changing font sizes. The control system can be extended adding new types of contexts and context atoms into the control scheme. This requires the implementation of distinct FLCs or adding new inputs to existing ones. However, in the design of FLCs the amount of important features can be larger, but the amount of fuzzy rules, which determine the functionality of a controller, grows rapidly. This may complicate the development of a control system. 4.3. User reactions We gathered users’ reactions, particularly their subjective opinions and preferences concerning the adaptive control of handheld device applications according to
536
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
the environment and user activity. Ten cell phone users were interviewed. Five of them were normal users, that is, they use basic communication applications. The rest of them were active cell phone users who use actively most of the cell phone applications and features. The interview process started by explaining the scenario presented in Table 2 to a user. Secondly, we explained to a user how adaptive applications operate in general and, particularly during the user scenario. At this phase, we presented control signals and screenshots of adapted service contents (Fig. 9). Thirdly, users were interviewed by asking their opinions about selected topics. In actual interviews, we avoided making leading questions to obtain subjective opinions and preferences. Questions and a summary of results are presented in Table 3. The results from user interviews show that in general application adaptation is an acceptable feature, but mistakes in adaptation are not accepted. Moreover, the users want to have control over the device’s operations. Particularly, to have a control over the device’s operations users refer to three main issues: (i) Device should ask user’s acceptance for adaptation when adapting major applications/application features. (ii) Device should provide notification before adaptation when adapting minor applications/application features. (iii) After adaptation user should be able to set original settings. Information representation adaptation is acceptable for certain types of applications and services, for example, timetables benefit from this feature, but time information should Table 3 Results of interview. Number in braces indicate how many similar answers appeared Q1: What is your opinion about adaptive behavior of applications in handheld devices? If it works properly without mistakes it is a good feature (7) I want to have a control over adapted actions (7) Simple features could be adaptive, but no execution of applications without my control (3) I don’t like if it makes mistakes, can cause embarrassing situations (3) Not good idea, user can get confused about adaptation (2) Q2: What is your opinion concerning the adaptation of information content of a service? To represent only compressed information is a good feature (6) It should not change information representation suddenly since it discontinues reading when some words disappear (6) System should adapt representation how I want. No mistakes in adaptation (3) Timetable service should take time information into account: there is no use to show past events (2) Some other services, for instance news, should not be adapted, current representation is enough (2) Q3: What is your opinion about font size adaptation of text represented on a screen according to user actions? It is a useful feature (6) It clarifies information representation (4) Font size should be bigger in all cases (3) There is no need for this feature when I type information (2) There is no need for this feature (2) Q4: What is your opinion about operating tune volume adaptation according to environment loudness? It is useful feature, particularly in loud environments (10) It should be possible to shut off adaptation/set silent profile in environments where it is appropriate to be silent (4)
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
537
also be involved to avoid presenting useless information. Continuous adaptation should be carried out without discontinuations in information representation. Also, social acceptance is critical; a device should not lead their users into embarrassing situations.
5. Conclusions and future work An approach for adapting applications in a handheld device according to multiple fuzzy contexts is presented. FLCs, which use predetermined sets of multiple fuzzy contexts, are designed for adapting specific features of UI applications. The context representation and the design of FLCs are presented in detail. The adaptation of the content and the representation of information according to multiple fuzzy contexts are demonstrated with real user data experiments. Context representation with context ontology for sensor based context information can be obtained by applying fuzzy quantization to features extracted from sensor signals. We have showed that this facilitates the development of application control. The examination of control signals suggests that control of applications with FLCs enhances the capabilities of a handheld device for representing information according to context. The rapid fluctuation of control signals is a problem since too fast adaptation may confuse users. Users’ reactions to adaptive control of applications are positive. They accept the adaptivity of minor application features while they want to have control over major application adaptation. The behavior of handheld device applications should follow the rules of humans’ social behavior. Users found the adaptation of information representation suitable for certain types of applications and services. The next phase of the work is to develop and test the application control by using highlevel contexts. Furthermore, to provide context-aware application adaptation, the presented mechanisms should be developed further and suitable UIs solutions for end user needs should be designed and evaluated.
References Brown, P.J., 1998. Triggering Information by Context, Personal Technologies, vol. 1. Springer, Berlin, pp. 18 –27. Clarkson, B., Mase, K., Pentland, A., 2000. Recognizing user context via wearable sensors. Fourth International Symposium on Wearable Computers (ISWC), IEEE, 69–75. Cox, E., 1998. The Fuzzy Systems Handbook; A Practinioner’s Guide to Building, Using and Maintaining Fuzzy Systems, Second ed, AP Professional, NewYork. Dey, A.K., Abowd, G.D., 2000. Towards a Better Understanding of Context and Context-awareness, CHI 2000 Workshop on The What, Who, Where, When, Why and How of Context-awareness, ACM, pp. 1–6. Dey, A.K., Salber, D., Abowd, G.D., 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications, anchor article of a special issue on context-aware computing. Human–Computer Interaction Journal 16 (2–4), 97–166.Lawrence Erlbaum Associates, New Jersey. Flanagan, J.A., Ma¨ntyja¨rvi, J., Himberg, J., 2002. Unsupervised clustering of symbol strings and context recognition. International Conference on Data Mining (ICDM), IEEE, 171–178. Himberg, J., Ma¨ntyja¨rvi, J., Korpipa¨a¨, P., 2001. Using PCA and ICA for exploratory data analysis in situation awareness. International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), IEEE, 127 –131.
538
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Hinckley, K., Pierce, J., Sinclair, M., Horwitz, E., 2000. Sensing techniques for mobile interaction. Symposium on User Interface Software and Technology (UIST), ACM, 91–100. Ma¨ntyja¨rvi, J., Himberg, J., Korpipa¨a¨, P., Mannila, H., 2001. Extracting the context of a mobile device user. 8th Symposium on Human–Machine Systems, 445–450.IFAC, Du¨sseldorf. McCarthy, J., Anagnost, T., 2000. EventManager: support for the peripheral awareness of events, lecture notes in computer science n:o 1927. Second International Symposium on Handheld and Ubiquitous Computing (HUC), Berlin, 227–235. Metso, M., Koivisto, A., Sauvola, J., 1998. Multimedia adaptation for dynamic environments. Second Workshop on Multimedia Signal Processing, IEEE, 203–208. Salber, D., Dey, A.K., Abowd, G.D., 1999. The context toolkit: aiding the development of context-enabled applications. Conference on Human Factors in Computing Systems (CHI), ACM, 434–441. Schmidt, A., Aidoo, K.A., Takaluoma, A., Tuomela, U., Van Laerhoven, K., Van de Velde, W., 1999. Advanced interaction in context, Lecture notes in computer science n:o 1927. Second International Symposium on Hand Held and Ubiquitous Computing (HUC), Berlin, 89– 101. Zadeh, L.A., 1965. Fuzzy sets. Information and Control 8 (3), 338 –353. Zadeh, L.A., 1996. Fuzzy logic and the calculi of fuzzy rules and fuzzy graphs: a precis. Journal of Multiple Valued Logic 1, 1–38.Taylor and Francis.
Adapting applications in handheld devices using fuzzy context information Jani Ma¨ntyja¨rvia,*, Tapio Seppa¨nenb a
VTT Electronics, Technical Research Centre of Finland, P.O. Box 1100, Oulu FIN-90571, Finland b University of Oulu, P.O. Box 4500, Oulu FIN-90014, Finland
Abstract Context-aware devices are able to take advantage of fusing sensory and application specific information to provide proper information on a situation, for more flexible services, and adaptive user interfaces (UI). It is characteristic for handheld devices and their users that they are continuously moving in several simultaneous fuzzy contexts. The dynamic environment sets special requirements for usability and acceptance of context-aware applications. Context-aware applications must be able to operate sensibly even if the context recognition is not 100% reliable and there are multiple contexts present at the same time. We present an approach for controlling context-aware applications in the case of multiple fuzzy contexts. This work has several potential applications in the area of adaptive UI application control. Our study is focused on the adaptation of applications representing information in handheld devices. The design of controllers and experiments with real context data from user scenarios are presented. Experimental results show that the proposed approach enhances the capability of adapting information representation in a handheld device. User reactions indicate that they accept application adaptation in many situations while insisting on retaining the most control over their device. Moreover, user feedback indicates that abrupt adaptations and instability should be avoided in the application control. q 2003 Elsevier B.V. All rights reserved. Keywords: Context aware computing; Context recognition; Mobile devices
1. Introduction Context-aware computing has been proposed to enhance the usability of handheld devices such as personal digital assistants (PDAs) and mobile phones by enabling the adaptation of various applications and UIs according to the context of a device and its user. The development of context-aware properties for handheld devices requires sensing of * Corresponding author. E-mail addresses: [email protected] (J. Ma¨ntyja¨rvi), [email protected] (T. Seppa¨nen). 0953-5438/03/$ - see front matter q 2003 Elsevier B.V. All rights reserved. doi:10.1016/S0953-5438(03)00038-9
522
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
a physical environment and the device’s user via low cost on-board sensors, exploiting context information sources located in the Internet, as well as acquiring terminal specific knowledge of applications and operating networks. The exact definition of context has so far been elusive but has been described as “Any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves” (Dey and Abowd, 2000). This definition of context, however, remains abstract, as the abstraction in the definition of context has been shifted to the definition of information. The definition does not indicate how context can be processed. Neither does it indicate how the relevancy of context for applications can be acquired. Exploiting context-awareness in handheld devices set challenges for system architectures since there are several feasible context information sources, such as sensors, location, applications, operating networks, and sources located over the Internet. Some of these sources might be distributed over various communication networks while some sources may be located on-board, for example, sensors. Due to the device’s mobility characteristics, a context information obtained from these sources is highly dynamic, thus making the exploitation of context information difficult. Salber et al. (1999) introduced an idea of a context toolkit for supporting the development of context-aware applications. Their architecture is reported to handle a distribution of components, dynamic context data and communication between heterogeneous components. The architecture introduces context widgets mediating between the environment and application. Widgets handle the sensing of the environment and they provide context information for applications. Other architecture solutions for various needs have been reported. The study made by McCarthy and Anagnost (2000) presents a tool for supporting peripheral awareness in workplaces. The tool allows users to be notified when a predefined event occurs. In this case a user sets a triggering value for context, not an application designer. An important issue in processing context information from sensors is how to obtain relevant context information. Handheld devices and their users are continuously moving in several simultaneous and overlapping contexts. A number of research efforts concerning the development of context recognition have been carried out. The context recognition from multidimensional sensor signals can be carried out, for example, using algorithms based on neural networks (Schmidt et al., 1999). Signals from multiple sensors can be processed into multidimensional context vectors, from which relevant context information can be extracted and compressed by using statistical methods (Himberg et al., 2001). To extract higher-level descriptions of contexts, the time series segmentation and clustering methods can be utilized (Ma¨ntyja¨rvi et al., 2001; Flanagan et al., 2002). In modeling of context states over time to provide most probable current context Hidden Markov Models (HMMs) can be exploited. Context recognition methods for wearable computers have been studied with the means of a wearable camera, environmental audio signal processing and HMMs (Clarkson et al., 2000). There are several studies concerning implementation of sensing systems for handheld devices to facilitate the development of context-aware applications. It has been proposed that there should be triggering mechanisms for executing context-aware applications. The triggering should occur for example, when sensor values match values defined in the triggering conditions (Brown, 1998). A similar approach has been presented in
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
523
the work of Schmidt et al. (Schmidt et al., 1999). They propose scripting primitives for performing three types of actions: when entering, leaving and while in a context. Their approach also exploits triggering mechanisms. Moreover, their work presents the implementation of several sensors into a cell phone and into a PDA to recognize the context of a handheld device and its user. The study by Hinckley et al. (2000) presents the integration of sensors into a PDA for developing novel applications and for examining implementation. Both studies, by Hinckley et al. and by Schmidt et al., propose that carefully implemented sensor fusion into handheld device provides a direct sensor-based application control and potential sources for context information. A central problem in the development of context-aware systems is how to compose the context representation. Moreover, there exists a lack of knowledge about the structure of context information in changing environments. It has been proposed that separation of how context is acquired from how it is used, and categorization of context information might help the development of these systems (Dey et al., 2001). More information for studying context information might be obtained by examining humans’ behavior in social contexts. For example, we lower our voice in public libraries and during lectures, and yell loudly in soccer games. Our actions are based on continuously learned adaptation models. Humans’ perceptions from environment are fuzzy. We recognize quantities of our environment to be for instance, loud, bright, warm, etc. It is reasonable to assume that the development of context dependent behavior for handheld device applications might benefit from fuzzy context representation. Due to the uncertainty related to context information, the formulation of exact definition explaining how to process context data and how to define its relevancy to applications has so far been elusive. The use of fuzzy context representation introduced in this paper provides a method to exploit sensor-based context information in handheld device applications. Moreover, by utilizing fuzzy logic methods for processing context information, the fuzzy control methods can be exploited in designing control mechanisms for adapting various applications and UIs. Dynamic environments set special requirements related to usability issues and social acceptance of novel functionalities of context-aware handheld device applications that must be able to operate according to multiple fuzzy context information with a minimum misinterpretation of current contexts. For context information extracted from sensor signals describe approximations of current contexts at best, the use of context information in application control still maintains the risk for launching applications in unsuitable situations and applications to behave in an unwanted manner. In most of the studies concerning the development of context-aware systems, the action performed based on context information is an execution of an application or a feature of an application. Usually, the execution is initiated by triggering. However, in some cases, it would be feasible to provide continuous context-based control signals. In this paper, we experiment with context information to control handheld device applications. We concentrate on one particular category of context sources located in handheld devices— sensors. An approach for adapting applications according to fuzzy context information is introduced. In our approach context representation is obtained by applying fuzzy membership functions. We show that the use of a fuzzy control for adapting applications according to fuzzy context representation benefits from the uncertainty in context information produced by
524
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
context recognition, thus more flexible continuous behaviour of applications can be achieved. A control mechanism is presented as a part of system architecture designed for handling context information. Section 2 explains how context representation is composed, and how the application adaptation is carried out by directly controlling different features of UI applications. The adaptation exploits fuzzy logic controllers (FLCs) in handling multiple fuzzy contexts. In Section 3 the design of FLCs is presented. The performance of the application adaptation is examined with concrete examples with real user data in Section 4. User experiences about the presented method are also provided. Section 5 concludes the work.
2. Fuzzy control of applications This section explains a procedure for converting raw signals obtained from sensors into a context representation, and a framework for fuzzy control of context-aware applications. 2.1. Fuzzy sets The theory of fuzzy sets defines the relations and operations between variables and fuzzy sets. Both basics and advanced explanation of fuzzy sets can be found, for example in (Cox, 1998; Zadeh, 1965). Fuzzy logic control has been widely used for control purposes in various fields of research. A good overview of fuzzy sets and systems is provided in (Cox, 1998). In the concept of fuzzy set a variable can be considered as an input x; which is related to a fuzzy set A by a membership function m: The relation is expressed as mA ðxÞ: The value of membership function can be something between zero and one, mA ðxÞ [ ½0; 1: Zero denotes a null membership and one denotes a full membership. An example of fuzzy sets with membership functions related to a variable x is presented in Fig. 2. Basic operations for fuzzy sets are complement, intersection and union. The membership of the complement of a fuzzy set A is
m:A ðxÞ ¼ 1 2 mA ðxÞ
ð1Þ
When defining fuzzy rule using logical operation NOT, the membership of the complement of a fuzzy set is utilized. The membership of the intersesction between two fuzzy sets A and B for aggregating two membership functions can be expressed as
mA>B ðxÞ ¼ minðmA ðxÞ; mB ðxÞÞ
ð2Þ
When defining fuzzy rule using logical operation AND, the membership of the intersection between fuzzy sets is utilized. The membership of the union between two fuzzy sets A and B is
mA
ð3Þ
When defining fuzzy rule using logical operation OR, the membership of the union between fuzzy sets is utilized.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
525
Fuzzy sets form the basics for developing the format for context representation. Using above-mentioned operations several fuzzy sets can be combined to a control signal. The operations of fuzzy sets are used in designing rule bases for FLCs. 2.2. Context representation A central problem in utilizing context information in application development is a difficulty to add meaning of concepts from real world to extracted features. For example, when setting a volume of ringing tone to behave context dependently, it would be easier to use representation: ‘When environment loudness is SILENT, set ringing tone volume LOW’, instead of the representation: ‘When environment loudness is 2 20 dB, set ringing tone volume to 2.5’. The first task in context extraction is to abstract raw sensor signals and compress information by using different signal processing and feature extraction methods. The features to be extracted are chosen according to how well they describe some parts of the real world context. We call extracted features context atoms since they contain the smallest amount of context information. The illustration of the process for increasing the abstraction level of raw context information is presented in Fig. 1. After sampling, raw signals are preprocessed, for instance, in the case of sensor measurements, signal values
Fig. 1. Illustration of context information processing procedure. Context information fusion produces higher-level context information.
526
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
are calibrated and rescaled. Preprocessed signals are used as inputs to various feature extraction methods in time and frequency domains producing features to describe context information. For example, the root mean square (RMS) value of an audio signal describes the loudness of environment audio. We have chosen to apply fuzzy quantization for extracted features resulting in a more expressive representation of context information. Fuzzy quantization can be viewed as a granulation of information (Zadeh, 1996). Two different methods can be used to quantize features to represent context information: (i)
Set crisp limits for features. The result is a logic style true false labeling for fuzzy membership functions. For example, in the recognition of loudness of environment audio, the processed feature is divided into three quantities labeled Silent, Modest, and Loud, corresponding to three membership functions as presented in Fig. 2a. If one of these is true, the others are false. (ii) Apply a fuzzy set for features resulting in a more expressive fuzzy labeling, as Fig. 2b presents. For example: mL ðxÞ ¼ 0.7/Silent þ 0.3/Modest þ 0/Loud. The context information processing scheme combines quantized features into a context atom vector CðnÞ ¼ ½C1 ; C2 ; …; Ck T at each quantization time step n; k ¼ maximum number of context atoms. The values of individual context atoms are between zero and one. The context representation used here has many advantages:
Fig. 2. Examples of crisp (a) and fuzzy (b) quantizations.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
527
(i)
It allows one to compose the ontology for sensor based context information that facilitates the development of context-aware applications, for example, a direct control of various types of applications, as shown in experiments (Section 3). (ii) Multidimensional context atom vector C enables the utilization of e.g. explorative data analysis methods for examining higher-level contexts as Fig. 1 depicts (Himberg et al., 2001; Ma¨ntyja¨rvi et al., 2001; Flanagan et al., 2002). (iii) It is extendable, i.e. it is straightforward to add new context atoms into a vector C, thus extending the ontology for sensor based context information. 2.3. Application control The implementation of context-awareness into a handheld device requires novel approach for developing system architectures to support both heterogeneous sets of context information sources and applications. Several promising approaches have been presented (McCarthy and Anagnost, 2000; Salber et al., 1999). We propose an additional block for handling context-aware applications that utilizes context representation. The approach differs from others because the block consists of FLCs, which are designed to use predefined sets of the elements of context atom vector C as inputs. Usually in architecture models, an application control is located between an application and information source, for instance, a server. The purpose of the application control is to process information from the information source into the functionality of an application. Applications can be controlled according to user context by adjusting various features, such as font size, spatial resolution, size of images, colors and contrast of images shown on a display, and volume and tunes of operating sounds. Thus, the information content and compression of information presented on the display can be controlled according to context of a user. The adaptation of these features improves the capabilities of presenting only the task relevant information to a user, for example, when he/she is browsing a service. We propose an additional block called Context based application control to be inserted into the architecture. The overview of the proposed architecture is presented in Fig. 3. The control scheme is influenced by the context representation, the requirements set by the usability models of applications, and the requirements set by the service information content domain. The Context based application control block resides between the UI application and the source of context information. The block consists of number of FLCs. Each FLC uses a predefined set of contexts as inputs and a particular rule base for handling information from multiple fuzzy contexts. A predefined set of contexts of a particular controller can be defined as a group of the most important contexts from the viewpoint of the functionality of a certain feature of the UI or the application itself. The functionality of the Context based application control block includes the capability to deliver control information for the Application control block and directly for the UI application. This ensures that it is possible to launch applications according to context information, and at the same time, to continuously control the functionality of the features of applications. Service adaptation covers the handling of physical, semantic and functional entities of media content of the service, and it provides functional scaling for the content
528
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 3. Overview of architecture for controlling applications based on fuzzy context information. Service provides information to a terminal with indexed content. ‘Context information’ block consists of context information available in a terminal, e.g. low and higher level context information.
and representation (Metso et al., 1998). The service adaptation can be exploited in representing only task relevant highlighted information according to context. Information provided by a service, for example, a bus timetable, includes information content with representation indexing. The indexing enables context-aware control of the content and the format of service information presented on a display. In representation indexing, the content of a service is indexed with k ¼ 1; 2; 3 information representation levels. Lower levels are sublevels of higher levels. For example, a sentence {A, B, C, D, E, F, G, H, I} is indexed with three levels as {A1, B3, C1, D2, E1, F2, G1, H3, I2}. Thus information representation is {A, C, E, G þ X} on level 1, {A, C, D, E, F, G, I þ Y} on level 2 and {A, B, C, D, E, F, G, H, I þ Z} on level 3. Variables X; Y; Z denote additional context dependent content such as ‘…next bus…‘. The service adaptation can be realized by matching indexing information with appropriate control signals from the FLCs. Adaptation of applications takes place in the Application control block, while the adaptation of application features takes place in the UI application block. 3. Design of fuzzy controllers This section explains the contents of Context based application control block. The design of FLCs for adjusting applications according to contexts is presented. Controlled application features include: Volume of operating tunes, font size of text presented on the display, display illumination, and the compression of information content presented as text on the display. Inputs to controllers are presented in Fig. 4, which shows categorized context variables with labeling. The context types and variables are chosen empirically based on tests with real sensor measurements. The chosen context types turned out to be the most important ones for these controlling purposes.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
529
Fig. 4. Categorization and labeling of context atoms.
3.1. Controller for operating tune volume One FLC is designed for controlling operating tunes. Inputs to the controller are context types ‘User activity’ and ‘Loudness level of environment’. Context type ‘User activity’ contains context atoms ‘Movements’, ‘Walking’, ‘Running’ while context type ‘Loudness level of environment’ includes context atoms ‘Silent’, ‘Modest’, ‘Loud’, as presented in Fig. 4. Moreover, a control variable represented as fuzzy sets is used in the design of the controller. Membership functions of fuzzy sets of context atoms and control variable used in the design of the controller are presented in Fig. 5. The output of the controller guides
Fig. 5. Piecewise continuous trapezoidal membership functions of context atoms used in designing the FLC. (a) User activity, (b) Loudness of environment (c) Volume of operating sounds. Values of membership functions are on the range of [0, 1].
530
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Table 1 Fuzzy rule base for the volume of operating tunes-controller
R1 R2 R3 R4 R5 R6 R7 R8 R9
IF User activity
AND Loudness of environment
THEN Volume of operating tunes
Movements Movements Movements Walking Walking Walking Running Running Running
Silent Modest Loud Silent Modest Loud Silent Modest Loud
Level 1 Level 2 Level 3 Level 2 Level 2 Level 3 Level 3 Level 3 Level 3
one specific feature of a UI application: The volume of operating tunes including ringing tones, message alerts, and other application specific tunes. Membership functions of context features and control variable are used in generating the fuzzy rule base for the controller. The rule base describes how a controller maps combinations of context features into control purposes according to the predefined rules, which are set by using basic operations of fuzzy sets described in Eqs. (1) – (3). The fuzzy rule base for the controller of the volume of operating tunes is presented in Table 1. The fuzzy rule base can be presented as a 3D-decision surface, which is presented in Fig. 6. Inputs to a controller are on the horizontal axes and the output of a controller is on the vertical axis. The decision surface illustrates, for example, that when either ‘User activity’ or ‘Loudness level of environment’ is at a maximum, the volume of operating
Fig. 6. The decision surface of fuzzy sound alert controller. Input features are on the horizontal axes and output of a controller is on vertical axis.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
531
tunes is at maximum. Furthermore, the minimum volume of operating tunes is obtained when both ‘User activity’ and ‘Loudness level of environment’ are at a minimum. 3.2. Controller for information representation An FLC utilizing two inputs and producing two outputs is designed for controlling the representation of information. Inputs to the controller are context types ‘User activity’ and ‘Illumination level of environment’. The context type ‘Illumination level of environment’ contains context atoms ‘Dark’, ‘Normal’, ‘Bright’. In this case there are two outputs from the FLC for adjusting different features of two UI applications: The one is the display illumination of a terminal and the other is the font size of text presented on the display. In addition, the font size control signal is used for guiding the compression of the information content. Information content adaptation is realized by exploiting the service content indexing, implemented in mobile services. Fuzzy rule bases for controlling features of information representation are illustrated as 3D-decision surfaces in Fig. 7a and b. The decision surface in Fig. 7a illustrates the behavior of the font size control. When ‘User activity’ is at a maximum, the font size is large, whereas when ‘User activity’ is at a minimum and ‘Illumination level of environment’ is quite bright, the font size is small. The decision surface presented in Fig. 7b illustrates the behavior of display illumination control. The brightest display is obtained when ‘Illumination level of environment’ is low, and also when ‘User activity’ is at a maximum. The display is always dark when ‘Illumination level of environment’ is high, except when ‘User activity’ is at a maximum. Both controllers are designed to together maintain the maximum contrast of information presented on the display.
4. Experiments and results This section presents experiments with real context data from user scenarios and they demonstrate the adaptive control of information representation application and application features. Moreover, this section presents results and discussion of experiments with the method, and a summary of the examination of user’s reactions to the system. 4.1. Context data Sensor data is logged from a user scenario with a sampling frequency of 256 Hz by using a portable laptop a user carries. Analog signals from accelerometers and illumination sensor are A/D-converted and sampled with National Instruments DaqCard 1200 measurement board, which is connected to a PC-CARD slot in a laptop PC. Audio data is sampled at 16 kHz by using the standard audio measurement board of the laptop. Data is stored in a file by using a measurement program made in LabVieww graphical programming environment. Conversion from raw sensor signals to context atoms is performed according to the procedure described in Section 2. It is carried out by using a self-programmed context recognition software, which produce a context atom vector C ¼ ½C1 ; C2 ; …; C9 T ; for describing two main context categories: ‘Environment’
532
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 7. (a) The decision surface of font size FLC. (b) The decision surface of display illumination FLC. Input features are on the horizontal axes and output of a controller is on the vertical axis.
and ‘User’ and for describing three context types: ‘User activity’, ‘Loudness level of environment’ and ‘Illumination level of environment’. Each context type contains three context atoms resulting in a nine-dimensional context atom vector. The categorization and labeling of context information is presented in Fig. 4. Context atom values are calculated every second and a sliding window with the length of four seconds is used in calculating averaged context atoms to prevent the occurrence of transients in control signals. The experiments with a user scenario are performed offline in a Matlabw scientific programming environment. First, FLCs are designed using a Matlabw Fuzzy Logic Toolboxw that provides graphical UIs to design fuzzy rules and controllers. Secondly, generated FLCs are tested offline with logged context data using a Matlabw Simulinkw simulating environment.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
533
Table 2 Description of the user scenario Segment n:o
Explanation of the action
1
User is at the office, it is late, he is planning to catch a bus from the nearest bus stop, and he is browsing bus timetable Walks along a hallway (browsing bus timetable) Walks down the stairs (browsing bus timetable) Opens the front door, goes out and walks on the street (browsing bus timetable) Notices that a bus is coming, begins to run to a bus stop (browsing bus timetable) Catches the bus, steps in to it, pays, and sits down
2 3 4 5 6
4.2. Experiments with user scenario Performance of designed controllers is evaluated with real context data recorded from a scenario where a user browses a bus timetable service by using a handheld device while he is leaving the office and is trying to catch a bus. A scenario is described in Table 2. The context information extracted from the user scenario recordings is presented in Fig. 8. The duration of a scenario is about three minutes. Experiments with a user scenario are performed as follows: 1. A user scenario presented in Table 2 is recorded and processed into context atom vectors. 2. Inputs are assigned to each FLC. 3. Outputs of the FLCs are gathered. 4. The output of the font size controller is used in adapting the information content of a text presented on the display. An information content indexing of a service is mapped to the control signal of the font size controller.
Fig. 8. Multiple fuzzy context signals from user scenario recordings. Values of context atoms are represented as grey level bars, black means 1 and white means 0 activity. Numbers of segments correspond to certain activities of a user.
534
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Fig. 9. Operating tune volume control, display illumination control and font size control signals from the user scenario. Numbers correspond the segments of the scenario. Outputs of FLCs are illustrated with continuous lines. Adapted information contents of bus timetable service with illumination control in different phases of the scenario are also presented.
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
535
Fig. 9 presents outputs of FLCs as continuous lines. The numbers of various segments correspond to actions explained in Table 2. Adapted information content of a bus timetable service in different phases of the scenario is also presented. The behavior of control signals, presented in Fig. 9, can be examined by comparing them to user actions expressed as context atoms in Fig. 8. Operating tune volume control signal is at a minimum when a user is inside (Segment 1), that is when ‘User activity’ is partly ‘Movements’ and partly ‘Walking’ while ‘Loudness of environment’ is ‘Silent’. Also, operating tune control signal is at a minimum in some points when a user is in a bus (Segment 6) and it is silent. The control signal starts to rise when a user begins to walk (Segment 2). The control signal rises considerably when a user walks down the stairs (Segment 3), reaching the maximum when a user is running (Segment 5), and in a noisy bus (Segment 6). Signals of font size and display illumination controllers behave in the same manner, except when a user is in a bus (Segment 6) when the font size control is at a minimum. This is because the ‘User activity’ is at a minimum that is, ‘Movements’. Display illumination control is a minimum when the user is under the bright light when going outside (between Segments 2 and 3). In the adapted information content of the bus timetable service the basic information representation (Level 3 representation) is provided when font size is small. That is, when ‘User activity’ is ‘Movements’. Correspondingly, the most compressed information representation (Level 1 representation) is provided when ‘User activity’ is ‘Running’. It demonstrates the information representation adaptation of service content that combines the benefits of the content and representation indexing of a service and context based application control in a terminal. It is required that the indexing is delivered via the service, or a terminal is able to index the content. Furthermore, in the screenshot showing the Level 1 information representation (Fig. 9), the one-touch dial to next bus timetable is presented. This functionality enables user to browse next bus timetable. Control signals indicate that the applications are able to adapt to the current context of a user’s handheld device by using FLCs and chosen context information. However, control signals tend to fluctuate when the values of individual contexts change rapidly. The variability of context signals can be considered as the main problem of a system’s performance since it is typical in rapidly changing environments, in which handheld devices operate. The fluctuation of control signals may lead to oscillation of various information representation levels on a display. The fluctuation of control signals can be eliminated using moving averaging methods for smoothing the context information or in this case, using smoothing in changing font sizes. The control system can be extended adding new types of contexts and context atoms into the control scheme. This requires the implementation of distinct FLCs or adding new inputs to existing ones. However, in the design of FLCs the amount of important features can be larger, but the amount of fuzzy rules, which determine the functionality of a controller, grows rapidly. This may complicate the development of a control system. 4.3. User reactions We gathered users’ reactions, particularly their subjective opinions and preferences concerning the adaptive control of handheld device applications according to
536
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
the environment and user activity. Ten cell phone users were interviewed. Five of them were normal users, that is, they use basic communication applications. The rest of them were active cell phone users who use actively most of the cell phone applications and features. The interview process started by explaining the scenario presented in Table 2 to a user. Secondly, we explained to a user how adaptive applications operate in general and, particularly during the user scenario. At this phase, we presented control signals and screenshots of adapted service contents (Fig. 9). Thirdly, users were interviewed by asking their opinions about selected topics. In actual interviews, we avoided making leading questions to obtain subjective opinions and preferences. Questions and a summary of results are presented in Table 3. The results from user interviews show that in general application adaptation is an acceptable feature, but mistakes in adaptation are not accepted. Moreover, the users want to have control over the device’s operations. Particularly, to have a control over the device’s operations users refer to three main issues: (i) Device should ask user’s acceptance for adaptation when adapting major applications/application features. (ii) Device should provide notification before adaptation when adapting minor applications/application features. (iii) After adaptation user should be able to set original settings. Information representation adaptation is acceptable for certain types of applications and services, for example, timetables benefit from this feature, but time information should Table 3 Results of interview. Number in braces indicate how many similar answers appeared Q1: What is your opinion about adaptive behavior of applications in handheld devices? If it works properly without mistakes it is a good feature (7) I want to have a control over adapted actions (7) Simple features could be adaptive, but no execution of applications without my control (3) I don’t like if it makes mistakes, can cause embarrassing situations (3) Not good idea, user can get confused about adaptation (2) Q2: What is your opinion concerning the adaptation of information content of a service? To represent only compressed information is a good feature (6) It should not change information representation suddenly since it discontinues reading when some words disappear (6) System should adapt representation how I want. No mistakes in adaptation (3) Timetable service should take time information into account: there is no use to show past events (2) Some other services, for instance news, should not be adapted, current representation is enough (2) Q3: What is your opinion about font size adaptation of text represented on a screen according to user actions? It is a useful feature (6) It clarifies information representation (4) Font size should be bigger in all cases (3) There is no need for this feature when I type information (2) There is no need for this feature (2) Q4: What is your opinion about operating tune volume adaptation according to environment loudness? It is useful feature, particularly in loud environments (10) It should be possible to shut off adaptation/set silent profile in environments where it is appropriate to be silent (4)
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
537
also be involved to avoid presenting useless information. Continuous adaptation should be carried out without discontinuations in information representation. Also, social acceptance is critical; a device should not lead their users into embarrassing situations.
5. Conclusions and future work An approach for adapting applications in a handheld device according to multiple fuzzy contexts is presented. FLCs, which use predetermined sets of multiple fuzzy contexts, are designed for adapting specific features of UI applications. The context representation and the design of FLCs are presented in detail. The adaptation of the content and the representation of information according to multiple fuzzy contexts are demonstrated with real user data experiments. Context representation with context ontology for sensor based context information can be obtained by applying fuzzy quantization to features extracted from sensor signals. We have showed that this facilitates the development of application control. The examination of control signals suggests that control of applications with FLCs enhances the capabilities of a handheld device for representing information according to context. The rapid fluctuation of control signals is a problem since too fast adaptation may confuse users. Users’ reactions to adaptive control of applications are positive. They accept the adaptivity of minor application features while they want to have control over major application adaptation. The behavior of handheld device applications should follow the rules of humans’ social behavior. Users found the adaptation of information representation suitable for certain types of applications and services. The next phase of the work is to develop and test the application control by using highlevel contexts. Furthermore, to provide context-aware application adaptation, the presented mechanisms should be developed further and suitable UIs solutions for end user needs should be designed and evaluated.
References Brown, P.J., 1998. Triggering Information by Context, Personal Technologies, vol. 1. Springer, Berlin, pp. 18 –27. Clarkson, B., Mase, K., Pentland, A., 2000. Recognizing user context via wearable sensors. Fourth International Symposium on Wearable Computers (ISWC), IEEE, 69–75. Cox, E., 1998. The Fuzzy Systems Handbook; A Practinioner’s Guide to Building, Using and Maintaining Fuzzy Systems, Second ed, AP Professional, NewYork. Dey, A.K., Abowd, G.D., 2000. Towards a Better Understanding of Context and Context-awareness, CHI 2000 Workshop on The What, Who, Where, When, Why and How of Context-awareness, ACM, pp. 1–6. Dey, A.K., Salber, D., Abowd, G.D., 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications, anchor article of a special issue on context-aware computing. Human–Computer Interaction Journal 16 (2–4), 97–166.Lawrence Erlbaum Associates, New Jersey. Flanagan, J.A., Ma¨ntyja¨rvi, J., Himberg, J., 2002. Unsupervised clustering of symbol strings and context recognition. International Conference on Data Mining (ICDM), IEEE, 171–178. Himberg, J., Ma¨ntyja¨rvi, J., Korpipa¨a¨, P., 2001. Using PCA and ICA for exploratory data analysis in situation awareness. International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), IEEE, 127 –131.
538
J. Ma¨ntyja¨rvi, T. Seppa¨nen / Interacting with Computers 15 (2003) 521–538
Hinckley, K., Pierce, J., Sinclair, M., Horwitz, E., 2000. Sensing techniques for mobile interaction. Symposium on User Interface Software and Technology (UIST), ACM, 91–100. Ma¨ntyja¨rvi, J., Himberg, J., Korpipa¨a¨, P., Mannila, H., 2001. Extracting the context of a mobile device user. 8th Symposium on Human–Machine Systems, 445–450.IFAC, Du¨sseldorf. McCarthy, J., Anagnost, T., 2000. EventManager: support for the peripheral awareness of events, lecture notes in computer science n:o 1927. Second International Symposium on Handheld and Ubiquitous Computing (HUC), Berlin, 227–235. Metso, M., Koivisto, A., Sauvola, J., 1998. Multimedia adaptation for dynamic environments. Second Workshop on Multimedia Signal Processing, IEEE, 203–208. Salber, D., Dey, A.K., Abowd, G.D., 1999. The context toolkit: aiding the development of context-enabled applications. Conference on Human Factors in Computing Systems (CHI), ACM, 434–441. Schmidt, A., Aidoo, K.A., Takaluoma, A., Tuomela, U., Van Laerhoven, K., Van de Velde, W., 1999. Advanced interaction in context, Lecture notes in computer science n:o 1927. Second International Symposium on Hand Held and Ubiquitous Computing (HUC), Berlin, 89– 101. Zadeh, L.A., 1965. Fuzzy sets. Information and Control 8 (3), 338 –353. Zadeh, L.A., 1996. Fuzzy logic and the calculi of fuzzy rules and fuzzy graphs: a precis. Journal of Multiple Valued Logic 1, 1–38.Taylor and Francis.