Intelligent multimedia layout: A reference architecture for the constraint-based spatial layout of multimedia presentations

cllNPumlSUUlOAROS IilNlifdCES ELSEVIEIR

Computer

Standards

& Interfaces

18 (1997)

515-524

Intelligent multimedia layout: a reference architecture for the constraint-based spatial layout of multimedia presentations Winfried H. Graf * German

Research

Center-for

Artificial

Intelligence-DFKI

GmbH,

Stuhlsaizenhausweg

3, Saarbriicken

66123,

Gutnag

Abstract With increases in the complexity and dynamics of multimedia information communicated by current applications, there arises a corresponding need towards a standard technology for intelligent multimedia interfaces. In this article, we address those components of an intelligent multimedia presentation system (IMMPS) which deal with the design and the realization of spatial Layout. We treat multimedia layout as a multidimensional constraint problem and propose a reference architecture for a general-purpose intelligent multimedia layout manager (IMMLM) that is based on a dedicated constraint solver kernel. 0 1997 Elsevier Science B.V. Keywords:

Intelligent

multimedia

presentation

systems;

Automated

layout;

1. Introduction Due to the tremendously growing amount and sophistication of daily electronic (tele)communication, humans demand more flexible, adaptive, and effective multimedia presentation systems which communicate specific information whenever and wherever they need it, as well as in the most adequate format to perform certain tasks. Especially in most industrially marketed applications, such as multimedia .presentation generation, flowchart design, electronic publishing, and Internet access, the inherent layout design tasks establish crucial points. Although multimedia technology is so popular today that virtually everybody is using it more or less, the commercially available tools suffer from fundamental shortcomings. For example, they are frequently restricted to canned presentation segments with hard* E-mail:

[email protected]

0920-5489,/97/$17.00 0 1997 Elsevier PII SO920-5489(97)00016-O

Science

B.V.

All rights

reserved.

Constraint

processing

techniques;

Standardization

wired layout patterns, limited in flexibility regarding adaptation, interaction, and media coordination, do not provide high-level authoring tools, intent-based layout facilities, and content-based reuse strategies, as well as do not consider the semantics and pragmatics of the presentation. As graphics infrastructure became more and more sophisticated during the last decade, a new generation of so-called intelligent multimedia presentation systems fIMMPSs), which meet the current drawbacks by incorporating advanced AI techniques, entered the area between multimedia user interfaces and knowledge-based systems (cf. Refs. [ 1,2]). While these system addressed nearly all aspects of multimedia presentation generation, currently there seems to be no principled intelligent multimedia layout manager (ZMMLM) that makes use of the structural properties of the underlying information. Most previous work on automated layout has only concentrated on single media types-for instance, in

516

W.H.

Graf/

Computer

Standards

the context of graphics generation-or specialized mechanisms and does not support the interplay between layout design and the other layers of an IMMPS (e.g., content planning). These approaches cover a broad spectrum of heterogeneous layout mechanisms ranging from traditionally encoded special-purpose algorithms to general AI-based tools. Procedural pagination and layout techniques comprise the large set of graph drawing algorithms (cf. Ref. [3]) and dynamic programming (e.g., Ref. [4]). Search and optimization mechanisms include numeric optimization (e.g., linear and integer programming), heuristic and stochastic search, constraint satisfaction (cf. Ref. [5]), genetic algorithms and physical simulations (e.g., simulated annealing) (cf. Ref. [6]). Grammar-driven approaches formulate layout requirements and styles as rules of a spatial relational grammar (e.g., Ref. [7]) or a graph layout grammar (e.g., Ref. [8]). Knowledge-based layout systems encode general layout expertise by means of a declarative knowledge representation using rulebased (e.g., Refs. [9,10]), constraint-based (e.g., Ref. [ 11,12]), and case-based formalisms (e.g., Ref. [ 131) and exploit efficient problem solving mechanisms for the computation of the final layout. Many of the aforementioned layout problems seem to go back to the lack of a generic model for an IMMLM. Especially, the construction of larger and commercially marketed IMMPSs requires a fundamental common basis for their development and evaluation such as benchmark tests. According to the proposal of a Standard Reference Model for Intelligent Multimedia Presentation Systems by the ERCIM Computer Graphics Network [ 141 which treats layout as a black-box, here, we refine the layout design component of its design layer by describing a standard reference model for an IMMLM. In Section 2, we describe the major design rationales of an IMMLM and we provide the required definitions and terminology and specify an IMMLM from the functional point of view. Thereafter, we propose a corresponding reference architecture for an IMMLM. Based on the expertise we have gained during the development of the constraint-based multimedia layout manager LuyLub [15,11,12], we provide a formalization of the knowledge-intensive layout design process as a multidimensional constraint satisfaction problem (CSP) that can be addressed by means of

& Interjaces

18 (1997)

515-524

advanced constraint processing techniques. In Part 3 of this volume, we will show how the proposed model was successfully exploited in the realization of the IMMLM LayLab [ 161.

2. A generic IMMLM 2.1. Definitions and terminology In practice, the design of an efficient pagination strategy for text-graphics documents-treated as the combination of search and optimization-has been proven as a complex computational task (cf. Ref. [4]). In addition to the computation of any single layout, multimedia presentation layout also makes strong demands for flexibility, including needs for multiple pagination strategies, varying and prioritized optimization criteria, and heterogeneous user requirements as well as application-specific graphical, semantic-pragmatic, and contractual layout constraints. In this respect, we define the term multimedia layout as follows. Definition 1. Multimedia Layout. Multimedia layout is the non-overlapping arrangement of multimedia items of constant size on a plain layout space as well us the determination qf graphical and ~pographical attributes regarding aesthetical and functional criteria. Since the physical format and layout of a presentation often conveys the structure, intention, and significance of the underlying information and therefore plays an important role in presentation consistency and coherency, automatic layout facilities have been included increasingly in presentation systems and multimedia interfaces. That is, besides aesthetical and space minimization criteria, multimedia layout also influences the rhetorical and intentional as well as attentional dimension of a presentation and thus can be treated as a communicative resource itself just as graphics or text. There is no other medium that has more impact on the attentional structure-i.e., the objects which are focused by the viewer-than layout. For instance, the visual clarity of a presentation can be enhanced strongly by using a graphical arrangement of its elements that empha-

W.H. Grf/

Computer

Standards

sizes the underlying logical and rhetorical structure of the information to be presented. So the visual organization of a sequence of objects representing an instruction has major impact on their decoding and interpretation (cf. Refs. [ 17,18]). Therefore, the intent of a presentation, that can be specified as a communicative goal, should partially be communicated through its layout. Unfortunately, previous investigations have mainly been concerned with compactness and stylistic aspects of the layout while its communicative function has only been regarded rudimentarily. Intelligent multimedia layout tools striving for effective: presentations involve the generation of intent-based layouts under constraints, which reflect the logical presentation structure and the presentation intention. Definition 2. Intent-based layout under constraints.. Given: set of layout objects, 0,) . . . ,O,: layout space, R; set of constraints, C; intentional goal, I; layout result, L; and user group, U; perceives (LI,L) means: user u perceives layout result, L; believes (u. I) means: user u believes the presentation intention I. An intent-based layout under constraints holds if: Vc, E C, c;(L) = true, and Vu, E U (i = 1,. . . ,n) perceives (cr,,L) * believes (u,,O. 2.2. Layout as constraint satisfaction problem As with many other interesting AI design and configuration tasks, spatial layout can be viewed as a complex combinatorial search and optimization problem in a finite discrete search space. It can be formalized as a constraint satisfaction problem (CSP) and can be solved efficiently using advanced constraint processing techniques. Here, the main problem consists of finding one, all, or the best solution that satisfies all topological and geometrical restrictions while regarding certain aesthetic criteria. In the following. we view layout as a Boolean CSP. That is, one has a set V of variables I’,, . . ,I:,,, each to be instantiated in an associated domain Di and a set of Boolean constraints limiting the set of allowed values for specified subsets of the variables. We stay to the general definition of a CSP given in Ref. [ 191 as

& Interjiicrs

18 (1997)

515-524

517

a formula in first-order predicate logic restricted to unary and binary predicates:

Here, Pi, is only included in the wff if i = P,,(,;,, I’,). Intuitively, a constraint definition comprises a declarative representation of the relation and a set of procedures that can be invoked by the underlying constraint solver to fufill it. The declarative semantics of constraint languages allows one to specify graphical objects while avoiding extraneous concerns about the realization of the drawing algorithms. Another major advantage is their ability to describe complex objects simply and naturally. So, constraint networks provide an elegant mechanism to declaratively state design-relevant knowledge about heterogeneous geometrical and topological relationships, characterizing properties between different kinds of multimedia objects that can be maintained by the underlying system. They can be easily used to specify layout requirements in graphical environments in order to guarantee local circumscriptions of the presentation like format restrictions, margins, distances. and non-overlapping. or to maintain consistency among objects on the whole document. Furthermore, the constraint programming approach allows the declarative representation of aesthetic knowledge, such as basic design principles expressing perceptual criteria. With regards to the definition above, layout design as a CSP can be formalized in the following manner: The position of each layout item is given by two variables-usually s and y coordinate-with associated domains that contain an (in)finite number of n-tuples of real or discrete values. For example. those domains can be described intensionally by constraints, e.g., specifying the boundaries of the connected subspaces permitted for that item. Constraints, e.g., ‘The header has to be placed in the upper left comer’, must also be expressed intensionally by algebraic equations and inequalities on the

518

W.H. Graf/Computer

Standards

values of the constraint variables. Moreover, one might have p-ary relations such as ‘text Tl must be placed between two graphics G2 and G3’. 2.3. The functional

view on an IMMLM

The knowledge-based layout design process can be viewed as a function that automatically computes a multimedia layout from a set of input data including multimedia objects and their relations, as well as the intent of the presentation by following certain layout-specific design parameters and exploiting application domain knowledge and common-sense knowledge about layout design. The schema in Fig. 1 illustrates the functional behaviour of an IMMLM. Its processing modules expect as input a set of multimedia components, such as text, graphics, animation, hypermedia, and video, in the internal representation of the corresponding medium. The input components can be delivered by media-specific generators of the generation layer within an IMMPS or can be retrieved from an application system (e.g., a multimedia database). The interi’intra-relationships between presentation parts are given by a rhetorical presentation structure. These comprise spatial and graphical relationships includ-

Fig. 1. Functional

& Interfaces

18 (19971 515-524

ing topological, geometrical, and visualization constraints. Moreover, we can distinguish local relations between objects that are semantically connected and global relations regarding the physical arrangement. Semantic and pragmatic relationships between multimedia fragments can be represented, for instance, by means of so-called rhetorical relations based on the rhetorical structure theory (RST) proposed in Ref. [20] for text planning. Typical examples of RST relations are ‘graphics-text’, ‘sequence’, ‘contrast’, ‘elaboration’, ‘organization’, and ‘motivation’. Besides the description of presentation acts on the content level, the design of interactive multimedia presentations requires the definition of relations such as ‘comply-with-request’, ‘ela.borate-exchange’ on the exchange level to describe the rhetorical structure of a multimedia dialogue. Furthermore, in the case of dynamic presentations (e.g., animation or video) temporal relationships have to be considered in the layout design process, which can be visualized by mapping them onto spatial or graphical constraints. The intention of the presentation is formalized as a presentation goal. The presentation structure and presentation goals may be established by a presentation planner at the content layer of an IMMPS or can

view on an IMMLM.

W.H. Graf/Computer

Standards

Fig. 2. The coordinated

& Interfirces

multimedia

be specified explicitly by the user or designer of the system. Application-specific layout criteria as well as common-sense layout design knowledge have to be formulated in a knowledge representation formalism, e.g., a constraint programming language. Then a multimedia layout can be tailored to an individual user and a specific context or situation by selecting among a set of design parameters. Thus, a large variety of layouts can be generated for one and the same injput data. The large set of layout relevant design parameters includes,: (a) Document type, e.g., technical manuals, overhea’d slides, scientific publications, newspapers and magazines, electronic directories, and display/web presentations; (b) Design conventions, e.g., reading direction, ordering, (weighted) grid structures, horizontal vs. vertical alignment, and leftto-right vs. top-down placement; (c) Resource limitations, e.g., hardware availability such as ASCII vs. graphics display, monochrome vs. color monitor, and display resolution, space restriction, time restriction (cf. anytime generation), expense limitation, and media restrictions; (d) Optimization criteria, e.g., space minimization, efficiency, graphics-text balancing, aesthetics, layout requirements, effectiveness, and user adaptivity; (e) User status, e.g., normal vs. short-sighted; (f) Presentation type, e.g., marketing vs. technical; (g) Output mode. batch vs. incremental; (h) Output medium, e.g., printer (portrait vs. landscape), display, CD-ROM, and WWW; (i) Layout representation language, e.g., PostScript, SGML (Standard Generalized Markup Language), HTML (Hypertext Markup Language), PDF (Portable Document Format), and LATEX; (i) Viewing time and distance. The physical manifestation of a multimedia presentation is essentially characterized by functional dependencies between its different document parts. In order to achieve a coherent output. an IMMLM must be able to reflect certain semantic and pragmatic relations through the visual appearance of a

layout

18 (19971515-524

design

519

process.

mixture of multimedia fragments. In this respect. the semantics of a multimedia layout is defined by an expressive mapping which relates the intentional and rhetorical presentation structure onto effective and adaptive layout and visualization techniques. Fig. 2 sketches the different phases in the multimedia presentation design process which derives a meaningful layout from the specified presentation intent as output for the presentation layer. For example, in the IMMPS WIP (cf. Ref. [21]), the presentation planner considers the production of multimedia output as a sequence of communicative acts to achieve certain goals. Here, the user specifies the communicative intent of a planned presentation as a presentation goal, e.g., the request of an instruction how to prepare espresso. This goal is transformed into a refinement-style plan, called rhetorical presentation structure, that contains the propositional contents, intentional goals, and rhetorical relations. This structure is formalized by means of semantic-pragmatic relations between multimedia elements. The essence of the layout design is the mapping of rhetorical relations onto graphical constraints: Layout-relevant rhetorical relations are compiled into visual techniques and Gestalt principles. These design heuristics are encoded in terms of a knowledge representation language, e.g., arithmetic constraints inequalities, or dis-stated as equations. equations-which allow for an efficient computation of a meaningf~d multimedia layout. The resulting layout plan is realized by means of a layout description language at the realization layer of an IMMPS. 2.4. General requirements The central guiding design rationales for a knowledge-based multimedia layout manager are flexibility, adaptivity, and interactivity in order to account for consistency, effectiveness, and expressiveness. In

520

W.H. Graf/Computer

Standards

& Interfaces

18 (1997)

515-524

the following, we will summarize the major design issues of an IMMLM.

interaction in order to achieve an effective and consistent output with high coherence.

2.4.1. Flexibility A generic IMMLM should guarantee domain independence and should be applicable to a broad spectrum of layout problems in many different environments. In a number of situations, it is an important feature to be able to produce customized layouts of multimedia items with minimal effort. Especially, in dynamic environments such as the world-wide web, displays must be flexible enough to accommodate varying numbers and sizes of objects on the fly.

2.4.5. Incrementalit?, Since IMMLMs will not rely on pre-defined links between pre-stored multimedia information items, presentation design will be performed during runtime in order to decrease the response time and react more promptly to the application. As a significant change in the application requires a complete redesign of the interface, we encourage an incremental update of the presentation to improve its quality and efficiency, i.e., the immediate realization of parts of a stepwise provided input, such as the beautification of lines in a flowchart and positions of display objects.

2.4.2. Adaptitiity By setting design parameters, an IMMLM should be able to generate a large variety of user-dedicated layouts as well as to adapt layouts on-line with regard to changing presentation situations. The fact that communication is always situated can be approximated by making all decision processes sensitive to design parameters. 2.4.3. Effectiveness A fundamental aspect of multimedia layout is the generation of effective presentations, which convey a specific communicative intent as well as human perceptual abilities in the layout. An IMMLM has to focus on the intent of the presentation in order to generate syntactically correct as well as semantically adequate and effective layouts that enhance the presentation and interpretation of the multimedia information. In this respect, layout itself is as an important carrier of meaning. IMMLMs should deal with page layout as a rhetorical force, influencing the intentional and attentional state of the reader. 2.4.4. Interactiuity IMMLMs must be able to handle the automated positioning of generated material as well as dynamic layout and graphical editing tasks in interactive settings. The quest for user interaction is based on the fact that it is impossible to anticipate all needs and requirements of each potential user in an infinite number of presentation situations. Therefore, systems using dynamic multimedia presentations have to adjust their design on-line in response to user

2.4.6. Declaraticitity An IMMLM should allow a natural-i.e., declarative and intuitive-formulation of layout-specific requirements in a sound and complete representation and specification language by the system designer as well as the end user. Furthermore, it should provide high transparency of the layout decisions (e.g., by means of explanation and debugging facilities). 2.4.7. Consistency and coherence A layout generation on the basis of a firm knowledge representation formalism and reasoning from first principles guarantees consistent and coherent presentations.

3. The reference architecture 3.1. Structuring the layout process The general layout process can be divided into two major tasks: high-level layout design versus low-level layout realization in a document representation language, such as LATEX [22], SGML [23], HTML [24], or PDF [25], which are partially performed within the design layer or the realization layer of the standard IMMPS model (cf. Fig. 6 and Fig. 8 in Ref. [14]). Although a full-fledged intelligent multimedia layout manager subsumes all modules for layout design, layout realization, and layout coordination, in the following we will focus on the crucial design part.

W.H. Graf/Computer

3.2. The architecture

Standards

& Interfaces

of an IMMLM

In general, the architecture of an IMMLM is composed of the processing modules and the knowledge base. The processing modules comprise (cf. Fig. 3): layout engine; pre- and post-processing modules (e.g., for grid generation and layout assessment); interface:s to the content, design, and realization layers; and, task-specific layout modules (e.g., for display management, grid-based placement, typography editing, beautification, navigation, and visualization). Furthemlore, a full-fledged version of an interactive IMMLM (e.g., in a standalone version) may also require: a compiler for the layout specification language; and, explanation and debugging facilities. In Section 3.2.1, we will concentrate on the central layout engine. 3.2.1. The layout engine The layout engine establishes the kernel of an IMMLM. It provides the problem solving mechanism-layout algorithm(s)-that is responsible for the efficient computation and optimization of the layout. Since multimedia layout of heterogeneous

Fig. 3. The architecture

18 i1997)

515-524

521

objects is a complex task, no single layout algorithm seems to be sufficient. Therefore, we propose a hybrid approach that is based on a spectrum of special-purpose problem solvers (PSI, ,PS, in Fig. 3) which are triggered at a control level (see also Refs. [ 11,26-281) As it has been shown in previous work, many graphical automation tasks can be facilitated efficiently using constraint processing techniques (cf. Ref. [5]). Recent general constraint (logic) programming languages have also demonstrated their adequacy and versatility for solving highly combinatorial problems (e.g., Oz [29]). An AI approach based on constraint technology provides the following strong advantages for automated multimedia layout: Constraint languages allow a flexible and intuitive formulation of complex layout restrictions as well as a declarative representation of graphical design knowledge. Hence, a constraint-based layout system is easy to maintain, extend, and customize by the user. Moreover, it is easy to adapt to various presentation media and to tailor to a specific user profile by setting design parameters. Constraint formalisms can also cope with semantic aspects of the layout.

of an IMMLM.

522

W.H. Cruf/

Computer

Standards

Furthermore, constraint-based approaches can easily be combined with other mechanisms such as genetic algorithms and simulated annealing (e.g., for layout optimization). Therefore, we propose a constraint-based methodology for representing and processing design-relevant layout knowledge as a basic paradigm for the knowledge-based operationalization of multimedia layout design. As general constraint systems are not mostly directed at the representation of complex layout knowledge, and seem too inefficient to compete with specialized layout algorithms, we recommend a set of constraint solvers (cf. Ref. [30]), which are dedicated to specific layout problems, as the central problem solving mechanism of a multimedia layout engine. Regarding the heterogeneous requirements of static geometric placement versus interactive on-line layout, a generic model has to treat the computation of a multimedia layout as a combination of constraint satisfaction (CS)-problem solving on constraints (search for solutions)and constraint-based inference (CBI) problemsproblem solving using constraints (modification of (sub)solutions). ’ 3.2.2. The structure of the knowledge base The processing modules of an IMMLM have at its disposal a large set of different knowledge sources representing expertise from different disciplines which can, in general, be divided into knowledge bases for layout design knowledge, common-sense knowledge, and context-sensitive domain knowledge (see also Ref. [3 II>. Layout design knowledge: Stylistic and legibility rules. grid structures, knowledge about presentation, interaction and visualization techniques, standards and guidelines. Commonsense knowledge: General layout strategies, design heuristics, empirical knowledge about perceptual and organizational Gestalt principles, and evaluation criteria.

’ This classification reflects the two major constraint solving approaches: the perturbation model vs. the refinement model (cf. Ref. [22]).

& Interfaces

I8 (19971 515-524

Context-sensitive knowledge: Application model, domain model, user model, and modalities/media model. 3.3. Coordination aspects of an IMMLM It is an important goal of an IMMLM not simply to map the multimedia generation results onto the display, but to carefully coordinate content planning and layout in such a way that the rhetorical and intentional content structure is reflected in an aesthetically pleasing and meaningful layout (see Refs. [32,33]). As already mentioned in Ref. [14], the layout design can be performed before, after, or interleaced with the production of the multimedia material. Here, one has to distinguish one-pass layout-if the presentation material has already been designed completely-and multi-pass layout, in the case of incrementally generated multimedia presentations. The latter may require further negotiations between the layout manager and the media design modules of the design layer as well as the content and realization layer of an IMMPS, such as for the determination of cross-media references. Frequently, a draft layout has to be revised because the output supplied by the media-specific realization components does not fit into the previously planned layout frames (e.g., a text or graphics does not fit on the page). In this case, a dependency-based layout revision process is initiated. On the other hand, also cases may arise in which formatting restrictions influence the selection of the contents. Such restrictions may be given a priorie.g., when a certain format is required-or result during the generation process-e.g., when the system has to follow the format of previously designed presentation parts to ensure syntactic coherence. For instance, after text realization the layout manager discovers that the produced text exceeds the margins of the document. Due to the severe format restrictions, there is no chance neither to increase the size of text frame nor to modify its typography. Therefore, the layout manager requests the text design component to shorten the corresponding sentence. If the text generator is not able to produce significantly shorter paraphrases, and is not allowed to revise the contents specification, it informs the presentation planner that the required task cannot be accom-

W.H. Graf/Computer

Standards

plished. Then the presentation planner decides which contents reduction will have the least impact on the intended communication goal.

4. Conclusion In this article, we have described a proposal for a new standard reference model for intelligent multimedia layout. We have described a constraint-based methodology as a general paradigm for the knowledge-based operationalization of multimedia layout design. This approach will be detailed by the example of I.VZP’s layout manager LuyZ,ab in Part 3 of this volume.

Acknowledgements Supported by a grant from The German Ministry for Research and Technology (BMBF) under contract ITW 9701 0.

References [I] [2] [3]

[4]

[5]

[6]

[7]

[8]

[9]

M.T. Maybury (Ed.), Intelligent Multimedia Interfaces, AAAI Press, Menlo Park, CA, 1993. M.T. Maybury, W. Wahlster (Eds.), Readings in Intelligent User Interfaces, Morgan Kaufmann, Los Altos, CA, 1997. G. Di Battista, P. Eades, R. Tamassia, I.G. Tollis, Algorithms for drawing graphs: an annotated bibliography, J. Comput. Geom. Theory Appl. 4 (5) (1994) 235-282, Also available via http://www.cs.brown.edu/people/rt/gd-biblio.html. M.F. Plass, Optimal Pagination Techniques for Automatic Typesetting Systems, PhD. thesis, Dept. of Computer Science, Stanford University, Stanford, CA, USA, 198 I. W. Hewer, W.H. Graf, A bibliographical survey of constraint-based approaches to CAD, graphics, layout, visualization, and related topics, Knowledge-Based Syst. 9 (7) (1996) 449-464. J. Christensen, J. Marks, S. Shieber, An empirical study of algorithms for point feature label placement, ACM Trans. Graplhics 14 (3) (1995) 203-232. L. Weitzman, K. Wittenburg, Grammar-based articulation for multimedia document design, Multimedia Syst. 1 (4) (1996) 99-111. F.J. Brandenburg, Designing graph drawings by layout graph grammars, in: R. Tamassia, LG. Tollis (Eds.), Graph Drawing: DIMACS International Workshop, LNCS 894, Springer, Berlin, 1994, pp. 416-427. S. Feiner, A grid-based approach to automating display

& Interfaces

18 (19%‘~ 515-524

523

layout, in: Proceedings of the Graphics Interface ‘88, Edmonton, Canada, Morgan Kaufman& Los Altos, CA, June 1988, pp. 192-197. [IO] C. Kosak, J. Marks, S. Shieber, Automating the layout of network diagrams with specified visual organization, IEEE Trans. Syst., Man. Cybernetics 24 (3) (1994) w-454. [I 11 W.H. Graf, Constraint-based graphical layout of multimodal presentations, in: M.T. Maybury, W. Wahlster (Eds.), Readings in Intelligent User Interfaces, Morgan Kaufmann, Los Altos, CA, 1997. [12] W.H. Graf, Constraint-based graphical layout of multimodal presentations, in: T. Catarci, M.F. Costabile, S. Levialdi (Eds.), Advanced Visual Interfaces, World Scientific Press, Singapore, 1992, pp. 365-385. [I31 R. MacNeil, Generating multimedia presentations automatically using TYRO, the constraint, case-based designer’s apprentice, in: Proceedings of the 1991 IEEE Workshop on Visual Languages (VL ‘911, Kobe, Japan, 1991, pp. 74-79. [I41 M. Bordegoni, G. Faconti, S. Feiner, M. Maybury, T. Rist, S. Ruggieri, P. Trahanias, M. Wilson, A standard reference model for intelligent multimedia presentation systems, Computer Standards and Interfaces, 1997. [I51 W.H. Graf, The constraint-based layout framework LayLab and its applications, in: I.F. Cruz, J. Marks, K. Wittenburg (Eds.), Proceedings of the Workshop on Effective Abstractions in Multimedia Layout, Presentations, and Interaction in conjunction with ACM Multimedia ‘95, San Francisco, CA, USA, November 4, 1995. Also available as Electronic Proceedings via http://www.cs.tufts.edu/ * isabel/mmwsproc.html. [I61 W.H. Graf, LayLab from the perspective of the IMMPS standard. In present volume, 1997. [I71 S. van Mulken, Reasoning about the user’s decoding of presentations in an intelligent multimedia presentation system, in: Proceedings of the Fifth Inremational Conference on User Modeling (UM-96), Kailua-Kona, Hawaii, January 2-5, 1996, pp. 67-74. 181 J. Nordbotten, M. Crosby, Reading strategies for graphic models from an experiment in data model perception, in: Proceedings of the Fifth International Conference on User Modeling (UM-96). Kailua-Kona, Hawaii, January 2-5, 1996, pp. 43-49. 191 A. Mackworth, Constraint satisfaction, in: S. Shapiro, (Ed.), Encyclopedia of Artificial Intelligence, Vol. 1, 2nd edn., Wiley, Chichester, England, 1992, pp. 285-293. . _ 1201 W. Mann, S. Thompson, Rhetorical structure theory: towards a functional theory of text organization, Discourse Processes 9 (1986) 57-90. [21] W. Wahlster, E. Andri, W. Finkler, H.-J. Profitlich, T. Rist, Plan-based integration of natural language and graphics generation, Artif. Intel]., Special Issue on Natural Language Processing 63 (I) (1993) 387-427. [22] L. Lamport, LaTeX: A Document Preparation System, User’s Guide and Reference Guide, 2nd edn., Addison-Wesley, Reading, MA, 1994. [23] CF. Goldfarb, The SGML handbook, Oxford Univ. Press, Oxford, England, 1990.

524 [24]

1251 [26]

[27]

[28]

[29]

[30]

[3l]

W.H.

Graf/

Computer

Standurds

T. Berners-Lee, D. Connolly, Hypertext Markup Language 2.0. MIT/W3C, Proposed ISO/IEC International Standard for HTML. Cambridge, MA, Sept., 199.5. Adobe (Ed.), Portable Document Format Reference Manual, Addison-Wesley, Reading, MA, 1995. G. Di Battista, G. Liotta, F. Vargiu, Diagram server, In: I.F. Cruz. P. Eades (Eds.), J. Vis. Languages Computing, Vol. 6, No. 3, Academic Press, New York, NY. Sept.. 1995, Special Issue on Graph Visualization. M. Himsolt, Comparing and evaluating algorithms with GraphEd, In: IF. Cruz, P. Eades (Eds.), J. Vis. Languages Computing, Vol. 6, No. 3, Academic Press, New York. NY, Sept., 1995, Special Issue on Graph Visualization. IF. Cruz, P. Eades (Eds.), J. Vis. Languages Computing. Vol. 6, No. 3, Academic Press, New York, NY, Sept., 1995. Special Issue on Graph Visualization. T. Miiller, K. Popov, C. Schulte. J. Wiirtz, Constraint programming in Oz, in: G. Smolka. R. Treinen (Eds.). DFKI Oz Documentation Series, German Research Center for Artificial Intelligence (DFKI GmbH), Saarbrlicken, Germany, 1997. Also available via http://ps-www.dfki.unisb.de/oz/documentation/. B. Freeman-Benson, M. Wilson, DeltaStar, how I wonder what you are: a general algorithm for incremental satisfaction of constraint hierarchies. Technical Report 90-05-02, Department of Computer Science and Engineering. University of Washington, 1990. S. Roth. W. Heffley, Intelligent multimedia presentation systems. Research and Principles, In: M.T. Maybury, W.

& Interfaces

18 (19971

515-524

Wahlster (Eds.). Readings in Intelligent User Interlaces, Morgan Kaufmann, Los Altos. CA, 1997. W. Graf, T. Rist. [Xl W. Wahlster, E. Andre, S. Bandyopadhyay, WIP: the coordinated generation of multimodal presentations from a common representation. in: A. Ortony, J. Slack, 0. Stock (Eds.). Communication from an Artificial Intelligence Perspective: Theoretical and Applied Issues, Springer. Berlin, 1992. pp. 121-144. layout in interactive multimedia [331 W.H. Graf, Intent-based communication, in: Proceedings of the First International Workshop on Intelligence and Multimodality in Multimedia Interfaces (IMMII J, Edinburgh, Scotland, July, 1995.

Winfried Graf received his diploma in Computer Science from the RWTH Aachen in 1988. Between 1988 and 1989, he worked at the Fraunhofer Institute for Biomedical Engineering (IBMT). In 1990, Winfried Graf joined the German Research Center for Artificial Intelligence (DFKI GmbHJ in Saarbriicken. where he is a senior researcher and project manager since 1995. Dr. Graf received a PbD. from the University of the Saarland in 1996 with a thesis on automated multimedia layout. He has published more than 50 technical papers in the area of intelligent user interfaces. His current research interests include intelligent multimedia systems, internet agents, constraint/casebased programming. and technology management.