Composing customized web 3D animations with semantic queries

Composing Customized Web 3D Animations with Semantic Queries

Journal Pre-proof

Composing Customized Web 3D Animations with Semantic Queries ´ Jakub Flotynski, Krzysztof Walczak, Marcin Krzyszkowski PII: DOI: Reference:

S1524-0703(19)30042-6 https://doi.org/10.1016/j.gmod.2019.101052 YGMOD 101052

To appear in:

Graphical Models

Received date: Revised date: Accepted date:

15 November 2018 18 April 2019 22 November 2019

´ Please cite this article as: Jakub Flotynski, Krzysztof Walczak, Marcin Krzyszkowski, Composing Customized Web 3D Animations with Semantic Queries, Graphical Models (2019), doi: https://doi.org/10.1016/j.gmod.2019.101052

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier Inc.

Graphical Abstract

1

Composing Customized Web 3D Animations with Semantic Queries Jakub Floty´nski, Krzysztof Walczak, Marcin Krzyszkowski Pozna´n University of Economics and Business Niepodległo´sci 10, 61-875 Pozna´n, Poland {flotynski,walczak}@kti.ue.poznan.pl, [email protected]

Abstract Creation of customized animated 3D content is an important issue in web-based VR/AR environments that involve users with various requirements for the content, e.g., in education, training, tourism, cultural heritage and e-commerce. 3D content customization is a complex task, as it typically covers various content features such as geometry, structure and appearance. Although a number of tools enable modeling of 3D animations, they do not support on-demand animation customization to users’ requirements specified using semantic (application or domain-specific) concepts. Available solutions permit creation of 3D content in its final form, require technical knowledge and skills or are limited to a particular application domain, without possibilities to easily change the target application or domain. In this paper, we present a pipeline for creation of customized animated 3D content on the web. The pipeline uses rule-based representation of 3D content activities, which are composed into more complex animations. Animation composition is performed on-demand, using queries describing requirements in an arbitrary application domain. The use of knowledge representation adapts the approach to the current trends in the evolution of the web, including the semantic web. It enables exploration by reasoning on the content generated with users’ queries. Furthermore, it permits inference of tacit properties of animations from explicitly specified properties, thereby liberating the authors from specifying all necessary animation properties. The pipeline leverages an advanced motion capture system, 3D modeling tools and game engines. We have implemented and evaluated a web-based environment for on-demand animation composition. The proposed approach can be successfully used in multiple application areas on the web, in which 3D content components are intensively reconfigured according to requirements of different users. Keywords: web 3D, VR/AR, 3D content, semantic web, logic programming, animation composition

1

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

1. Introduction Virtual reality (VR) and augmented reality (AR) applications become increasingly popular in various application domains, which has been enabled by the possibility of realistic, immersive presentation of virtual objects as well as the availability of diverse inexpensive presentation and interaction devices, such as glasses, headsets, haptic interfaces, motion tracking and capture systems. The primary element of any VR/AR application is 3D content, which typically includes various animated geometrical, structural and presentational elements. Multiple approaches and tools have been developed to permit creation of animated 3D content for VR/AR applications. The first group encompasses libraries (e.g., Java3D [1] and Direct3D [2]) for programming 3D content animations with imperative languages (e.g., Java and C++). The second group are declarative 3D content formats (e.g., VRML [3], X3D [4] and XML3D [5]), which allow for content animation based on key frames and interpolation as well as animation described by imperative scripts. The third group are tools for visual 3D modeling (e.g., MotionBuilder [6] and SketchUp [7]), which also enable the use of key Preprint submitted to Computers & Graphics

22 23 24 25 26 27 28

29 30 31 32 33 34 35 36 37 38 39

40 41

frames and interpolation. In addition, some tools permit programming of animations (e.g., Blender [8] and 3ds Max [9]). However, the aforementioned approaches have not been designed to build customized 3D animations of reusable 3D components. This is an especially important deficiency in the context of web-based VR/AR environments based on 3D content repositories. In particular: 1. The available approaches require knowledge of technical concepts specific to 3D graphics and animation, or they are limited to a particular application/domain without the possibility to be used in other applications/domains. Wide use of VR and AR demands the involvement of domain experts, who are usually not IT-specialists, in the 3D modeling process. 3D content creation methods should be general (application/domain independent) and capable of using semantic (i.e., domain- or applicationrelated) concepts across a variety of possible application domains. 2. The available approaches require imperative programming of animations, covering steps that must be completed to December 16, 2019

42 43 44 45 46 47

48 49 50 51 52 53 54

55 56 57 58 59 60 61 62 63 64 65 66 67

68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

achieve desirable goals. Instead, 3D content creation should focus on goals to be achieved (like in declarative programming) and hide technical steps that must be completed. Such an approach liberates content authors from developing algorithms, which is a complex technical task. 3. The available approaches require specification of all 3D content properties to be presented. Efficient 3D content creation methods should liberate users from determining all content properties by enabling inference of tacit properties on the basis of explicitly specified properties, e.g., to infer assignment of objects to domain-specific classes based on visual objects’ properties. 4. The available approaches do not enable efficient customization of animated 3D content with queries. Generally, 3D content formats and programming languages are difficult for query-oriented processing, while the available tools are designed for visual modeling of content. 3D content creation methods, especially these used on the web, should permit on-demand creation of animated 3D content. Queries could select 3D objects and animations from repositories to compose 3D scenes, e.g., animated urban areas with avatars, cars and natural elements. In addition, the results of queries should be suitable for further indexing, searching and processing by search engines and analytical tools. These requirements can be satisfied by applying knowledge representation technologies to 3D content creation, including logic programming and description logics. The use of knowledge representation gains increasing attention in the semantic web, which aims at the evolutionary development of the current web towards a distributed database linking structured content with descriptions of the semantics of its components and properties. In contrast to the available 3D content formats, knowledge-based representation may be used in any application and domain, in a form intelligible to domain experts or end users, with possibilities of inferring tacit content properties from the explicitly specified properties. There are multiple possible applications of knowledge representation to 3D content, such as content retrieval [10], design of industrial spaces [11], archaeology [12] and molecular visualization [13, 14]. Nonetheless, the available approaches to knowledge-based 3D content creation do not enable efficient on-demand customization according to requirements of different content consumers (users or applications). First, the available methods and tools produce 3D content in its final form without inference of tacit content properties. Second, the available knowledge-based 3D content representations are static (time-independent) and do not evolve over time – even if the content does. Therefore, dynamic (time-dependent) 3D content properties cannot be customized and explored by consumers’ queries. For example, a running 3D machine model could be queried about the presentation and description of states and fuel consumption at different points in time. 3D football games could be queried about the presentation and number of goals scored by a player in a period of

time. From a 3D crowd simulation only avatars with suspicious behaviors could be selected for presentation. Temporal knowl99 edge representation has been used so far in AI and robotics, 100 but it can be used also in VR/AR. Temporal knowledge-based 101 representation of 3D content can be achieved using logic pro102 gramming [15], which has been addressed in our approach. 103 The main contribution of this paper is a content creation 104 pipeline for animated 3D web content. The key stage of the 105 pipeline is query-based composition of 3D content animations, 106 which uses the proposed rule-based representation of anima107 tions. In the composition, basic activities of 3D objects, created 108 with animation modeling tools and motion capture systems, 109 are represented by application- or domain-specific concepts and 110 composed on-demand into more complex animations in a goal111 oriented fashion. The composed animations are provided to 112 users in a form that can be further explored with queries about 113 different properties in different points in time. The results of 114 queries reflect past, current or potential future states of the an115 imated content, which enables the use of content as a respon116 sive, dynamically changing database. The used time-dependent 117 3D content representation is based on Prolog, which is a well118 established language for programming in logic with facts and 119 rules. Facts and rules permit inference of tacit content prop120 erties from explicitly specified properties. The syntax of the 121 rule-based representations can be more intelligible to non-IT122 specialists in comparison to imperative instructions. 123 The remainder of this paper is structured as follows. Sec124 tion 2 provides an overview of the current state of the art in 125 knowledge-based representation of 3D content with focus on 126 behavior representation. The proposed pipeline of animated 127 3D web content creation is described in Section 3. The rule128 based representation of animations, which enables query-based 129 composition of animations within the pipeline, is presented in 130 Section 4. In Section 5, the developed animation composition 131 environment is outlined. It is followed by the description of an 132 example pipeline with query-based generation of customized 133 3D content animations as well as examples of exploration of 134 the animations in Section 6. Evaluation and discussion of the 135 approach are provided in Sections 7 and 8. Finally, Section 9 136 concludes the paper and indicates possible future research. 97 98

137

2. Related Works

The background of the proposed content creation pipeline covers aspects related to knowledge-based representation of 3D 140 content, adaptation of 3D content, and languages for querying 141 semantically described resources. 138 139

142

2.1. Knowledge-based 3D Content

The largest group of knowledge-based solutions for representing 3D content use the semantic web. A comprehensive 145 review of such solutions has been presented in [16]. The main 146 standards in this field used to represent 3D content are the Re147 source Description Framework (RDF) [17], the RDF Schema 148 (RDFS) [18] and the Web Ontology Language (OWL) [19]. 143 144

3

RDF is a data model based on triples (subject – property – object). RDFS and OWL are languages built upon RDF, provid151 ing higher expressiveness—classes and properties that enable 152 comprehensive description of content. These standards permit 153 design of ontologies, which are specifications of conceptual154 ization [20] for a domain. Ontologies may describe arbitrary 155 objects as well as classes and properties of objects. Ontologies 156 constitute the foundation of the semantic web. The Semantic 157 Web Rule Language (SWRL) [21] is an extension to OWL with 158 rules, which permits enrichment of ontologies with rule sets. 159 Following [16], a knowledge-based 3D content representa160 tion is a document that specifies 3D content components and 161 properties—in particular, geometrical, structural, spatial, pre162 sentational and domain-specific—using knowledge representa163 tion standards. Hence, a knowledge-based 3D content repre164 sentation may be a 3D scene, encoded as an ontology in RDF, 165 RDFS and OWL, or a rule set, encoded in SWRL. The pro166 cess of creating knowledge-based 3D content representations is 167 referred to as knowledge-based 3D content creation. The es168 sential activity of knowledge-based 3D content creation, which 169 is also the main advantage over other modeling approaches, is 170 the inference of tacit knowledge. Knowledge about 3D content, 171 which may be expressed at the low or high abstraction levels, 172 can be used to infer implicit content properties from explicit 173 properties. Therefore, content authors are liberated from speci174 fying all necessary content properties. 175 A number of works have been devoted to knowledge-based 176 representation of 3D content including a variety of geometrical, 177 structural, spatial, presentational and domain-specific elements. 178 However, the majority of the approaches are used like typical 179 3D content formats, in which all content elements have to be ex180 plicitly specified by the authors, without capabilities of knowl181 edge inference. The available methods are summarized in Table 182 1. Five methods address the low (graphics-specific) abstraction 183 level, while eight methods address high (general, application184 or domain-specific) abstraction levels. Three of those methods 185 may be used with different application or domain ontologies.

instance, a mapping links particular meshes in 3D scenes to particular pieces of furniture in a virtual shop to allow for intuitive 195 search and presentation to customers [26]. The approach also 196 permits combination of primitive actions (e.g., move, turn, ro197 tate, etc.) to composite behavior intelligible to end users with198 out the knowledge of computer graphics. The method requires 199 authors to specify all components and properties of 3D content. 200 The method proposed in [27, 28] also enables 3D content 201 creation at both low and high abstraction levels. Ontologies 202 used in the method include graphical 3D content components 203 (e.g., shapes and textures) and properties (e.g., coordinates and 204 indices) as well as high-level domain-specific components (e.g., 205 body parts) and properties (e.g., joint attributes, descriptors of 206 articulation levels, 3D animations of face and body, and behav207 ior controllers). The method does not use knowledge inference. 208 The method proposed in [29] permits 3D content creation at 209 the low abstraction level. The used ontology provides compo210 nents and properties that are equivalents of X3D nodes and at211 tributes, e.g., textures, dimensions, coordinates and LODs. The 212 method allows for inference on rule sets. For instance, if an 213 individual is of the atom class (body), generate a sphere to rep214 resent it in an ontology for chemical compounds (head). Thus, 215 final 3D content is based on both the explicit and implicit (in216 ferred) knowledge. However, the method does not enable map217 ping between high- and low-level concepts, so it is unsuitable 218 for modeling 3D content by domain experts. 219 Another method of creating 3D humanoids has been pro220 posed in [30, 31, 32]. After automatic segmentation of 3D mod221 els, the distinguished body parts are annotated. Two modes of 222 annotation have been developed. Automatic annotation is com223 pleted by software considering topological relations between 224 content elements (e.g., orientation, size, adjacency and over225 lapping). Manual annotation is completed by a user equipped 226 with a visual tool. Although the method benefits from inference 227 in annotating of 3D models, it is limited to virtual humans. 228 The method proposed in [33, 34] enables creation of non229 manifold 3D shapes using low-level properties. Once 3D shapes 230 are segmented, graphical properties are mapped to a shape on231 tology and form a low-level shape representation. The ontol232 ogy specifies diverse geometrical properties of shapes: non233 manifold singularities (e.g., isolated points and curves), one234 dimensional parts, connected elements, maximal connected ele235 ments, the number of vertices, the number of non-manifold ver236 tices, the number of edges, the number of non-manifold edges 237 and the number of connected elements. It permits representa238 tion of such objects as a spider-web, an umbrella with wires and 239 a cone touching a plane at a single point. 240 The tool described in [35] leverages widely known con241 cepts, services and hybrid automata to describe behavior of ob242 jects in 3D simulations. The tool has a client-server architec243 ture. The client is based on a 3D browser, e.g., for XML3D, 244 while the server is built of several services enabling 3D con245 tent creation. A graphical module maintains and renders 3D 246 scene graphs. A scene module manages global scene ontolo247 gies, which represent the created simulations. A verification 248 module checks spatial and temporal requirements against prop249 erties of content elements. An agent module manages intelli-

149

193

150

194

Table 1: Comparison of methods of ontology-based 3D content creation Approach Troyer et al. [22, 23, 24, 25, 26] Guti´errez et al. [27, 28] Kalogerakis et al. [29] Spagnuolo et al. [30, 31, 32] Floriani et al. [33, 34] Kapahnke et al. [35] Albrecht et al. [36] Latoschik et al. [37, 38, 39] Drap et al. [12] Trellet et al. [13, 14] Perez-Gallardo et al. [11]

Level of Abstraction Low (3D Graphics) High (Domain) ✓ general ✓ humanoids ✓ humanoids ✓ general interior design general archaeology molecules ✓ -

Inference ✓ ✓ ✓ ✓ ✓ ✓

The method proposed in [22, 23, 24, 25, 26] enables content creation at both the low and the high abstraction levels. At 188 the particular levels, different 3D content ontologies connected 189 by mapping are used. Low-level ontologies may be created by 190 graphic designers, while high-level ontologies may be created 191 by domain experts. Mapping of low- to high-level ontologies 192 adds interpretation to graphical components and properties. For

186

187

4

gent avatars, e.g., their perception of the scene. The user interface allows for communication with web-based and immersive 252 virtual reality platforms. Knowledge-based content representa253 tions are encoded in XML using the RDFa and OWL standards, 254 and linked to 3D content encoded in XML3D. 255 In [36], a method of 3D content creation based on point 256 clouds has been proposed. At the first stage of the method, an 257 input point cloud is analyzed to discover planar patches, their 258 properties (e.g., locations) and relations. Then an OWL rea259 soner processes a domain ontology including conceptual ele260 ments that potentially match the analyzed patches. Next, match261 ing elements are selected and configured to build a high-level 262 representation in the interior design domain. Created repre263 sentations are ontological equivalents to the input point clouds. 264 The method uses knowledge inference to analyze 3D scenes, 265 e.g., context of use and connections between 3D objects. 266 In [37, 38, 39], a general-purpose tool and method of 3D 267 content creation has been described. The method is based on 268 actors and entities, which represent 3D content at a high level, 269 are described by shared state variables and are subject to events. 270 In particular, the approach can be used in game design. 271 In [12], a method and software for representing underwa272 ter archaeological objects in 3D have been presented. In the 273 approach, a Java-based application generates an ontology rep274 resenting objects. Further, queries encoded in the SWRL lan275 guage [21] can be used to select objects to build a 3D visualiza276 tion. 277 In [13, 14], an approach to knowledge-based representation 278 of 3D molecular models has been proposed. The approach com279 bines different input (e.g., interaction using different haptic and 280 motion tracking devices) and output (e.g., presentation in 2D 281 and 3D) modalities to permit presentation and interaction suit282 able for particular types of content and tasks to be done. The 283 approach uses inference on ontologies describing the content 284 and SPARQL queries sent by users to visualize selected objects. 285 In [11], a system for 3D recognition of industrial spaces has 286 been presented. The method used in the system recognizes ob287 jects in point clouds presenting interiors of factories. The rec288 ognized objects, their properties and relations, which are spe289 cific to 3D graphics, are further represented using ontologies. 290 On this basis, topological relations between object are inferred. 291 In [40, 41, 42], an approach to describing X3D scenes with 292 embedded semantic metadata attributes and harvesting the meta293 data into separate knowledge bases have been presented.

[45], a model of virtual environments based on the MPEG-7 and MPEG-21 standards has been proposed to permit dynamic 307 scaling and adaptation of the geometry and functions of vir308 tual objects. In [46], an approach to 3D content adaptation to 309 different devices and contexts of use has been proposed. In 310 [47], a solution for streaming X3D content for remote visualiza311 tions with the adaptation of the level of detail based on several 312 parameters (e.g., network bandwidth, device capabilities and 313 user preferences) has been proposed. In [48], an approach to 314 multi-platform 3D content presentation based on MPEG-4 has 315 been presented. In [49], an approach to multi-platform visual316 ization of 2D and 3D tourist information has been discussed. 317 An approach to adaptation of 3D content complexity with re318 spect to the available resources has been proposed in [50]. In 319 [51, 52, 53, 54, 55], a method and a tool for building adaptable 320 animated VR applications have been proposed. The approach is 321 based on an ontology-based mapping between application- and 322 graphics-specific ontologies. 323 The aforementioned approaches are mainly devoted to con324 tent adaptation depending on the target hardware and software 325 platforms used. The solutions provide limited capabilities of 326 modification of particular content elements, and they do not ad327 dress on-demand customization of 3D content animations.

250

305

251

306

294

2.2. Adaptation of 3D Content

3D content adaptation may be considered as content customization performed implicitly with regards to the context of 297 use (e.g., user location or capabilities of the hardware and soft298 ware used, or past user’s activities). Approaches to adaptation 299 of 3D content have been designed mostly to permit content pre300 sentation on diverse hardware and software platforms. 301 In [43, 44], a rule-based framework using MPEG-7 has been 302 proposed to enable adaptation of content for different access 303 devices. Adaptation may include such operations as geome304 try and texture degradation as well as filtering of objects. In

328

2.3. Query Languages for Web Content

Several languages have been designed to allow for generic domain-independent queries to knowledge bases encoded in RDF, 331 e.g., the RDF Data Query Language (RDQL) [56], SPARQL 332 (the successor to RDQL) [57], SPARQL Update (SPARUL— 333 the extension of SPARQL with capabilities of updating seman334 tic graphs) [58] and the Second Generation RDF Query Lan335 guage (SeRQL) [59]. Also, SWRL can be used for querying 336 RDFS and OWL knowledge bases [60]. In contrast to the pre337 vious approaches, SWRL provides support for the syntax, struc338 ture and semantics of RDFS and OWL, which is an important 339 advantage in the context of applications on the semantic web. 340 Several languages have been designed to enable retrieval 341 of multimedia content (in particular images, audio and video), 342 e.g., the MPEG query format [61] and the Graphical Multime343 dia Query Language (GMQL) [62]. A number of works have 344 been devoted to the analysis of building information models 345 (BIM) using query languages, e.g., in [63], a query language 346 with specific spatial and temporal operators has been proposed. 347 The aforementioned languages have been intended for content 348 exploration, and they do not permit modeling and customiza349 tion of 3D content. 329 330

295

296

350

3. Creating Customized 3D Animations

The main contribution of this paper is the pipeline of customized animated 3D web content creation. The pipeline is an 353 extension of the approach proposed in [64] with a new archi354 tecture and visual tools for creating animation customization 355 queries. We provide an overview of the pipeline in this section, 356 followed by the description of our novel formal rule-based an357 imation representation in Section 4, as well as the presentation 351 352

5

of the new architecture of the web-based animation composition environment in Section 5. Further, we illustrate the con360 cepts with an example of query-based animation composition 361 in Section 6. Sections 7 and 8 provide results of evaluation and 362 a discussion. 363 The pipeline, which is presented in Fig. 1, consists of the 364 following stages. 358

411

359

412

365 366 367 368 369

370 371 372 373 374 375 376 377 378 379 380 381

382 383 384 385 386 387 388

• 3D objects creation provides 3D objects that will be animated in the 3D scene being modeled. This stage is based on the use of 3D scanners (e.g., Artec 3D) and modeling tools (e.g., Blender, 3ds Max) providing geometry and appearance of the objects, e.g., rigged characters with materials. • 3D scene modeling is based on the 3D objects created at the previous stage. The scene may be encoded using different 3D formats, programming languages and libraries (e.g., determined by the software development environment used, e.g., Unity and C#) as well as knowledge representation (e.g., Prolog) and semantic web technologies (e.g., OWL). The selection of scene representation technologies depends on whether the geometry, structure and appearance of the scene should be subject to ontology- or rule-based customization as well as exploration queries at the next stages. Examples of query-based customization of structural and presentational 3D scene elements have been presented in [65]. • Motion preparation is based on motion capture systems (e.g., OpenStage 2) and animation modeling tools (e.g., MotionBuilder), which provide activities that will compose animations of the 3D objects created at the first stage. The activities may be mutually independent, e.g., different physical exercises or various dance figures. This stage is independent of 3D scene modeling and may be completed in parallel.

413 414 415

objects finalize the previous and start the next activities in different poses, e.g., jumping followed by walking. Due to realistic body movements prepared using motion capture systems and interpolation used to combine them, the generated animations also look realistic.

417

The result of the presented pipeline is a 3D scene including 3D objects with customized animations.

418

4. The Rule-based Representation of Animations

416

In this section, a formalism used to represent 3D content in our approach is presented. Further, rule sets built upon this 421 formalism are explained. 419 420

4.1. Formalism 423 Query-based composition of animations is feasible due to 424 the use of the rule-based representation of animations, which 425 extends the approach proposed in [66, 67]. The representation 426 is a collection of rule sets, which consist of facts and rules built 427 upon predicates. A predicate, which is the basic concept of the 428 first-order logic (FOL), is a boolean-valued function that may 429 be specified on zero or multiple variables, which can be values 1 430 or other predicates : 422

431

predicate(X1 ,X2 ,...,Xn ) ⇒ {true, false}.

A fact is a predicate whose some variables may be set to particular values. A fact is unconditional – its fulfillment does 434 not depend on the fulfillment of other facts. The example facts 435 describe the category and duration of an activity: 432 433

436 437

category(activity, exercise), duration(activity, 10).

A rule is an implication, whose head is true if the body is true, thus in contrast to facts, rules express conditions. Both 440 head and body are logic statements on facts. A specific type of 441 rule is a Horn clause, in which the head consists of only one 442 fact, while the body is a conjunction of facts. The large advan443 tage of Horn clauses over other types of rules is the availability 444 of a polynomial-time inference procedure [68]. The example 445 Horn clause: 438

389 390 391 392 393 394 395 396 397 398 399

400 401 402 403 404 405 406 407 408 409 410

• Application ontology / rule set implementation provides a knowledge-based representation of activities prepared at the previous stage. The representation may be an ontology or a rule set designed with different standards, including the semantic web and logic programming. It covers application or domain knowledge about the activities, such as their conditions, context of execution, purposes, etc. An application ontology will be subject to queries at the next stage in order to create customized objects’ animations. Like motion preparation, this stage is independent of 3D scene modeling and may be completed in parallel. • Query-based animation composition adds complex animations to 3D objects in the scene. The animations are compositions of activities that have been selected by a query sent by a content consumer (user or application). The query-based animation composition is possible due to the use of an application ontology / rule set and the animation rule set (introduced in Section 4), which enable application-specific representation of activities. Once a composition of activities is determined, the animations of the activities are composed into the final animation using interpolation. Interpolation smooths the transitions between the subsequent activities, when the

439

next(Activity1 , Activity2 ) ← category(Activity1 , fitness), category(Activity2 , fitness), 448 duration(Activity1 , L1 ), duration(Activity2 , L2 ), L2 >=L1 .

446

447

means that two activities are arranged into a sequence if both are fitness activities and the second activity is longer than the 451 first one. In the proposed representation of animations, facts 452 and Horn clauses are used. 453 In the reasoning on the rule-based animation representation, 454 the closed world assumption is used, thus activity properties 455 that are not known to be true are considered false. Also, nega456 tion as failure is used to explicitly indicate the lack of proper457 ties. 449 450

1 Variables and constants are denoted with capital and small letters, respectively.

6

Figure 1: The 3D content creation pipeline

494 propriate facts to the rule set using the assertz predicate. The 4.2. Rule Sets 459 The rule-based animation representation consists of the an- 495 added facts have the following format: 460 imation rule set and an application rule set. The animation rule 496 instanceOf (E, event), 461 set enables temporal representation and reasoning on activities 497 time(E, T), 462 that are composed into animations. The rule set includes pred498 start(after(E), E), 463 icates of the event calculus [15, 69]. The event calculus uses 499 end(PreviousPeriod, E), 464 FOL to express events, moments in time, periods of time, as well 500 holds(executes(object, activity), after(E)). 465 as fluents, which are predicates varying over time. The rule sets 501 The facts determine that an event E that occurs at a point in time 466 are encoded in the Prolog logic programming language, which 502 T, starts the new period after(E) and finishes PreviousPeriod 467 is sufficiently expressive to implement the event calculus. First, 503 (started by another event). The last fact states that an object ex468 the language enables rules, which are necessary to implement 504 ecutes an activity within the period following E. 469 reasoning on events, moments, periods and fluents. Second, it 470 enables making statements on predicates, which is necessary to Listing 1: The animation rule set 471 designate activities lasting within periods or happening in mo- 505 start (after ( Event ) , Event ) . 506 end (before ( Event ) , Event ) . 472 ments in time. Third, it permits negation as failure, which is 507 473 necessary to ”close the world” and state that an activity did/- 508 unsaid ( Predicate ) ← Predicate , ! , fail . 509 unsaid ( Predicate ) . 474 does/will not occur. 510 475 Since the facts and rules in the animation rule set are appli- 511 in ( Point , Period ) ← start ( Period , Event1 ) , end ( Period , 512 Event2 ) , time ( Event1 , Point1 ) , time ( Event2 , Point2 ) , 476 cation- and domain-independent, the rule set is common to all 513 Point >= Point1 , Point < Point2 . 477 applications created with the proposed approach. The anima- 514 in ( Point , Period ) ← start ( Period , Event1 ) , unsaid (end ( 515 Period , Event2 ) ) , time ( Event1 , Point1 ) , Point >= 478 tion rule set is presented in Listing 1. In lines 1-2, the start 516 Point1 . 479 and the end predicates are associated with the after and before 517 518 holdsAt ( Fluent , Point ) ← holds ( Fluent , Period ) , in ( Point , 480 predicates, as an event starts the period following it and ends 519 Period ) . 481 the period preceding it. The unsaid predicate checks whether 520 notHoldsAt ( Fluent , Point ) ← holdsAt ( Fluent , Point ) , ! , 521 fail . 482 a predicate is true, if not—unsaid is satisfied (lines 4-5). In the 522 notHoldsAt ( Fluent , Point ) . 483 definition of unsaid, negation as failure has been used. A point 523 524 isExecutionSet ← executesAt (X , Activities , StartTime ) , 484 in time is in a period, if the point is between two events: start525 instanceOf ( Activities , ac ti vi ty Se qu en ce ) , eventSet ( 485 ing and ending the period, or after the event starting the period 526 Activities , StartTime , X , after ( start_evn ) ) . 527 486 if the period is not finished yet (lines 7-8). A fluent holds at 528 eventSet ( Sequence , T , Label , PrevPeriod ) ← instanceOf ( 487 a particular point in time if the point is in a period within which 529 Sequence , acti vi ty Se q ue nc e ) , first ( Sequence , 530 Activity ) , rest ( Sequence , Sequence2 ) , duration ( 488 the fluent holds (line 10). The complementing notHoldsAt 531 Activity , L ) , moment ( T ) , T2 is T +L , E is " evn_ " + 489 predicate has been defined using negation as failure (lines 11- 532 Label + " _ " + Sequence + " _ " + Activity , eventSet ( 533 Sequence2 , T2 , Label , after ( E ) ) , assertz (instanceOf (E , 490 12). The isExecutionSet predicate initiates the creation of 534 event ) ) , assertz (time (E , T ) ) , assertz (start (after ( E ) , E 491 events, points in time and periods for activities that are parts 535 ) ) , assertz (end ( PrevPeriod , E ) ) , assertz (holds ( 536 executes ( Label , Activity ) , after ( E ) ) ) . 492 of animations (line 14). It uses the eventSet predicate (lines 537 eventSet ( Sequence , T , Label , PrevPeriod ) ← Sequence == 493 16-17), which iterates over the activity sequence and adds ap- 538 null .

458

7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

17

The second part of the animation representation—an application rule set—is created for a particular use case at the ap541 plication ontology / rule set implementation stage within the 542 pipeline, and it semantically represents the activities prepared 543 at the motion preparation stage. 539

540

544

5. The Animation Composition Environment

The Animation Composition Environment (ACE) has been developed as a proof of concept implementation of the pre547 sented approach. ACE uses the rule-based animation represen548 tation and enables query-based animation composition within 549 the proposed pipeline of 3D content creation.

545

546

550

5.1. Architecture

The modules of ACE, interaction between users and the environment as well as the data flow are depicted in Fig. 2. The 553 environment consists of two main modules: the Visual Query 554 Builder (VQB) and the Animation Composer (AC). VQB is 555 a Java-based desktop application that allows a user to build 556 queries and send them to AC. It is based on Eclipse with the 557 Oxygen plugin. Oxygen is a well-established tool for creating 558 XML documents, which we use to allow users to visually build 559 queries encoded in XML. XML queries are converted to Pro560 log by the Query Transformer. Queries use concepts from the 561 application rule set to specify requirements that should be satis562 fied by the animations to be composed. Further, Prolog queries 563 are encapsulated into HTTP requests, and received by AC using 564 a RESTful Web Service installed on the Apache HTTP Server. 565 A query is conveyed to the Reasoning Engine, which is based 566 on the C#Prolog library [70]. The engine performs reasoning 567 on the sum of the query as well as the animation and applica568 tion rule sets. Whereas the animation rule set is application569 independent, the application rule set depends on the area of use 570 (cf. Section 6). 571 The result of the reasoning is a rule-based animation de572 scriptor, which specifies the composition of activities that meet 573 the user’s requirements and determine the final customized an574 imation. The descriptor is read by the Scene Manager to com575 pose the animation from the activities modeled or captured at 576 the motion preparation stage of the pipeline. The composed 577 animation is applied to appropriate 3D objects, which are in578 cluded in the 3D scene created at the 3D scene modeling stage. 579 Next, the animated 3D scene is delivered to the user through 580 the Apache HTTP server. Also, the animation descriptor is 581 published to the user as a WebGL application generated by the 582 Unity game engine. We have selected Unity as it is a widely583 used tool for developing VR/AR applications. To enable explo584 ration of the animation with queries (cf. Section 6) the anima585 tion rule set, application rule set, and animation descriptor are 586 also published to the user.

551 552

Figure 2: The architecture of the Animation Composition Environment (ACE)

customizing 3D content must be built using common or domainspecific concepts in an intuitive and user-friendly way. This is 593 enabled by the XML standard and available tools for creating 594 XML documents due to their characteristics: 591 592

595 596

597 598

599 600

601 602

1. XML documents may be visually designed, which is intelligible to non-IT-specialists. 2. XML enables encoding of any content using any set of concepts. 3. The structure of XML documents may be formally described using XML Schema. 4. XML documents may be efficiently transformed to other documents using XSL.

Because of the aforementioned advantages, XML is preferred over Prolog for query design. 605 Primary, VQB produces queries that are XML documents. 587 5.2. Visual Query Design 588 One of the primary goals of our approach is to allow on- 606 The GUI of VQB is presented in Fig. 3. In the presented ex607 ample, XML queries comply with the XML Schema for Fitness 589 demand customization of animations for average users and do590 main experts, who are not IT-specialists. Hence, the queries 603 604

8

Composition, which we have developed. The schema is depicted in Fig. 4. It specifies XML elements and attributes that 610 enable:

608

609

612

1) description of exercises and muscles: effects, difficulty, stimulation and parts of the body;

613

2) assignment of sequences of exercises to avatars;

611

614 615

3) specification of variables to be determined by the Reasoning Engine while processing queries.

While the XML Schema for Fitness Composition is specific to our domain and use cases, VQB can use various XML Schema 618 documents that are specific to other applications and domains. 619 Anyway, an XML Schema used by VQB must conform to the 620 application rule set used by ACE. 616 617

Figure 4: A fragment of the XML Schema for Fitness Composition used in the Visual Query Builder

or domain knowledge, encompasses properties specified in the application rule set and the XML Schema used in VQB. In the 641 example, users leverage VQB to prepare queries that specify de642 sirable properties of exercises to be demonstrated and send the 643 queries to AC. Query-based composition of fitness exercises en644 ables on-demand preparation of training plans for different peo645 ple in the form of customized instructional animated 3D scenes. 646 The example pipeline is presented with respect to its particular 647 stages (cf. Section 3). 639 640

648

Figure 3: The GUI of the Visual Query Builder enabling visual query design (the right panel) with the generated XML query (the middle panel) and prompt about permitted values of an attribute (the small window)

XML queries are further converted to Prolog queries, using the Java XML DOM library org.w3c.dom, and sent to the An623 imation Composer. Prolog queries include the content of the 624 prototype XML queries extended with assertion statements that 625 modify the target rule set by adding new statements creating 626 instances of classes (e.g., exercises) and specifying temporal 627 information using event calculus (cf. the animation rule set in 628 Section 4). This information is necessary to enable query-based 629 exploration of temporal properties of the composed animated 630 3D content. 631 The XML to Prolog transformation is performed on the client 632 side to preserve the independence of the AC from different types 633 of clients and formats used for query preparation. In our ap634 proach, the preparation of queries in the Prolog format, which 635 is processable by AC, is the task of query builders.

649 650 651

621

652

622

653 654 655 656

657 658 659 660 661 662 663 664

665 666

636

637 638

6. Example of Pipeline and Use Cases

667

An example of animation composition is illustrated with fitness exercises. The composition, which is based on application

• 3D objects creation. At this stage, three rigged 3D characters in the FBX format have been acquired from the Unity Asset Store2 . Materials of the clothes as well as characters’ skeletons have been modified in Unity (Fig. 5). • 3D scene modeling. At this stage, the characters have been embedded into a 3D scene presenting a park, which is a common place for doing fitness. The scene has been encoded in the Unity scene format, as it will not be subject to reasoning and queries – as opposed to the characters. • Motion Preparation. At this stage, a motion capture system (Organic Motion OpenStage 2) has been used to capture fitness exercises demonstrated by an instructor (Fig. 6). The system can simultaneously capture the motion of up to three people without using markers. The exercises have been stored in the FBX format independently of 3D characters, which enables application of the exercises to different characters. • Application ontology / rule set implementation. At this stage, an application rule set representing the fitness exercises has been implemented. Its fragment is presented in Listing 2. 2 https://www.assetstore.unity3d.com/en/#!/content/41056,

9

/40848, /50690

698 28 699 29 700 30

701 702 703 704 705 706 707 708 709 710 711 712 713

Figure 5: 3D characters and their skeletons in Unity

714 715 716 717 718 719 720 721 722 723 724 725 726 727

Figure 6: The markerless motion capture system Organic Motion OpenStage 2 with an actor being captured

669 670

The rule set specifies the classes of muscles (lines 1-4), as well as the properties of the exercises such as stimulated muscles (6-17), effects (19-24) and difficulty (26-30). Listing 2: A fragment of the application rule set used for query-based animation composition

6711 6722 6733 6744

partOf ( arm_triceps , arm ) . partOf ( shoulder , arm ) . partOf ( thigh_quadriceps , leg ) . partOf ( calf_triceps , leg ) .

6755 6766 6777 6788 6799 680 10 681 11 682 12 683 13 684 14 685 15 686 16 687 17

690 20 691 21 692 22 693 23 694 24

697 27

732 7333 734 735 7364 737 738 739 7405 741 7426 7448 745 7469 747 748 10 749 750 11 751 752 753 12 754 13

757 758 759 760 761 16 762 763 17

difficulty ( push_ups , hard ) . difficulty ( squat_jump , easy ) .

effect ( AE1 , endurance ) , stimulates ( AE1 , pectoral ) , effect ( AE2 , endurance ) , AE1 \== AE2 , assertz (instanceOf ( aEx1 , AE1 ) ) , assertz (instanceOf ( aEx2 , AE2 ) ) , assertz (instanceOf ( aFit1 , ac ti vi ty Se q uenc e ) ) , assertz ( instanceOf ( aFit2 , a ct iv it yS eq uen ce ) ) , assertz (first ( aFit1 , aEx1 ) ) , assertz (rest ( aFit1 , aFit2 ) ) , assertz (first ( aFit2 , aEx2 ) ) , assertz (rest ( aFit2 , null ) ) , assertz (duration ( aEx1 , 90) ) ,assertz (duration ( aEx2 , 60) ) , assertz (executesAt ( alice , aFit1 , 30) ) ,

755 14 756 15

effect ( push_ups , strength ) . effect ( squat_jump , endurance ) . effect ( squat_jump , flexibility ) . effect ( bear_crawl , strength ) . effect ( side_lunge , flexibility ) . effect ( burpees , endurance ) .

695 25 696 26

730 7312

effect ( JE1 , endurance ) , difficulty ( JE1 , easy ) , effect ( JE2 , strength ) , effect ( JE3 , flexibility ) , difficulty ( JE3 , medium ) , stimulates ( JE3 , th ig h_ qu ad ri ceps ) , assertz (instanceOf ( jEx1 , JE1 ) ) , assertz (instanceOf ( jEx2 , JE2 ) ) , assertz (instanceOf ( jEx3 , JE3 ) ) , assertz (instanceOf ( jFit1 , ac ti vi ty Se q ue nc e ) ) , assertz ( instanceOf ( jFit2 , a ct iv it yS eq ue nce ) ) , assertz ( instanceOf ( jFit3 , a ct iv it yS eq ue nce ) ) , assertz (first ( jFit1 , jEx1 ) ) , assertz (rest ( jFit1 , jFit2 ) ) , assertz (first ( jFit2 , jEx2 ) ) , assertz (rest ( jFit2 , jFit3 ) ) , assertz (first ( jFit3 , jEx3 ) ) , assertz (rest ( jFit3 , null ) ) , assertz (duration ( jEx1 , 60) ) , assertz (duration ( jEx2 , 30) ) , assertz (duration ( jEx3 , 90) ) , assertz (executesAt ( joe , jFit1 , 0) ) ,

7437

stimulates ( push_ups , arm_triceps ) . stimulates ( push_ups , pectoral ) . stimulates ( squat_jump , t h i g h _ q u a d r i c e p s ) . stimulates ( squat_jump , shoulder ) . stimulates ( bear_crawl , arm_triceps ) . stimulates ( bear_crawl , t h i g h _ q u a d r i c e p s ) . stimulates ( bear_crawl , calf_triceps ) . stimulates ( side_lunge , t h i g h _ q u a d r i c e p s ) . stimulates ( side_lunge , calf_triceps ) . stimulates ( burpees , arm_triceps ) . stimulates ( burpees , t h i g h _ q u a d r i c e p s ) . stimulates ( burpees , pectoral ) .

688 18 689 19

• Query-based animation composition. At this stage, a user forms a query using VQB and sends it to the AC. Listing 3 presents a Prolog query, which has been generated by VQB on the basis of an XML query. The query is a rule set that consists of three blocks, each specifying an example training plan, which is a sequence of exercises presented by an individual character in the scene. First, domain-specific requirements for the exercises are given (lines 1, 8, 15). In the example, Joe has to demonstrate a full range of exercises improving endurance, strength and flexibility, respectively. He has to start with an easy exercise improving endurance, followed by an exercise for strength, and finish with a medium exercise stimulating thigh muscles and improving their flexibility. Alice has to demonstrate two different exercises improving endurance. In addition, the first exercise should stimulate pectoral muscles. Katy has to demonstrate two different exercises stimulating muscles of legs, followed by an exercise stimulating triceps. Katy’s training should start with an easy exercise. For every exercise that matches the conditions, its instance is created (2, 9, 16). These statements are added by VQB during the XML-to-Prolog transformation. The instances will be the animations of individual exercises executed by the characters in the 3D scene. The following statements specify sequences of the created instances (3-4, 10-11, 17-18). Next, the duration of the exercises within the sequences is set (5, 12, 19). Finally, every sequence is assigned to a character and given a start time (6, 13, 20). Listing 3: A query specifying requirements for an animation composition

7281 729

668

difficulty ( bear_crawl , medium ) . difficulty ( side_lunge , medium ) . difficulty ( burpees , hard ) .

764 765

10

stimulates ( KE1 , LegMuscle1 ) , partOf ( LegMuscle1 , leg ) , difficulty ( KE1 , easy ) , stimulates ( KE2 , LegMuscle2 ) , partOf ( LegMuscle2 , leg ) , KE1 \== KE2 , difficulty ( KE2 , medium ) , stimulates ( KE3 , calf_triceps ) , difficulty ( KE3 , medium ) , assertz (instanceOf ( kEx1 , KE1 ) ) , assertz (instanceOf ( kEx2 , KE2 ) ) , assertz (instanceOf ( kEx3 , KE3 ) ) , assertz (instanceOf ( kFit1 , ac ti vi ty Se q uenc e ) ) , assertz ( instanceOf ( kFit2 , a ct iv it yS eq uen ce ) ) , assertz ( instanceOf ( kFit3 , a ct iv it yS eq uen ce ) ) ,

766 18 767 768 769 770 19 771 772 20

773 774 775 776 777 778 779 780 781 782 783 784 785 786 787

assertz (first ( kFit1 , kEx1 ) ) , assertz (rest ( kFit1 , kFit2 ) ) , assertz (first ( kFit2 , kEx2 ) ) , assertz (rest ( kFit2 , kFit3 ) ) , assertz (first ( kFit3 , kEx3 ) ) , assertz (rest ( kFit3 , null ) ) , assertz (duration ( kEx1 , 55) ) , assertz (duration ( kEx2 , 55) ) , assertz (duration ( kEx3 , 55) ) , assertz (executesAt ( katy , kFit1 , 15) ) .

To enable further query-based exploration of animations, animation descriptors are generated by the reasoning engine. A fragment of the descriptor for Joe’s exercises is presented in Listing 4. For the other characters, equivalent facts are generated. The animation descriptor is the result of reasoning on the sum of the animation rule set, the application rule set and the query. Lines 1-20, which specify the exercises, their sequences, duration and execution, are the results of the processing of lines 1-6 of the query (Listing 3). Lines 2240, which specify the events, their moments of occurrence, periods determined by the events, as well as fluents, are the results of processing of lines 14-17 of the animation rule set (Listing 1). The holds fluent links the exercises executed by Joe to corresponding periods, which enables temporal querybased exploration of the composed animation. Listing 4: A fragment of an animation descriptor

7881 7892 7903

instanceOf ( jEx1 , squat_jump ) . instanceOf ( jEx2 , push - ups ) . instanceOf ( jEx3 , side_lunge ) .

7914 7925 7936 7947

instanceOf ( jFit1 , ac t i v i t y S e q u e n c e ) . instanceOf ( jFit2 , ac t i v i t y S e q u e n c e ) . instanceOf ( jFit3 , ac t i v i t y S e q u e n c e ) .

7969 797 10 798 11 799 12 800 13 801 14

first ( jFit1 , jEx1 ) . rest ( jFit1 , jFit2 ) . first ( jFit2 , jEx2 ) . rest ( jFit2 , jFit3 ) . first ( jFit3 , jEx3 ) . rest ( jFit3 , null ) .

802 15 803 16 804 17 805 18 807 20

810 23 811 24 813 26 814 27 815 28

time ( evn_joe_jFit1_jEx1 , 0) . time ( evn_joe_jFit2_jEx2 , 60) . time ( evn_joe_jFit3_jEx3 , 90) .

816 29 817 30 818 31 819 32

start (after ( ev n _ j o e _ j F i t 1 _ j E x 1 ) , e v n _ j o e _ j F i t 1 _ j E x 1 ) . start (after ( ev n _ j o e _ j F i t 2 _ j E x 2 ) , e v n _ j o e _ j F i t 2 _ j E x 2 ) . start (after ( ev n _ j o e _ j F i t 3 _ j E x 3 ) , e v n _ j o e _ j F i t 3 _ j E x 3 ) .

820 33 821 34 822 35 823 36

end (after ( start_evn ) , e v n _ j o e _ j F i t 1 _ j E x 1 ) . end (after ( e v n _ j o e_ j F i t 1 _ j E x 1 ) , e v n _ j o e _ j F i t 2 _ j E x 2 ) . end (after ( e v n _ j o e_ j F i t 2 _ j E x 2 ) , e v n _ j o e _ j F i t 3 _ j E x 3 ) .

824 37 825 38 826 39 827 40

holds (executes ( joe , jEx1 ) , after ( e v n _ j o e _ j F i t 1 _ j E x 1 ) ) . holds (executes ( joe , jEx2 ) , after ( e v n _ j o e _ j F i t 2 _ j E x 2 ) ) . holds (executes ( joe , jEx3 ) , after ( e v n _ j o e _ j F i t 3 _ j E x 3 ) ) .

7.1. Users’ Effort in Animation Composition 840 As opposed to the ACE environment, the available approaches 841 to 3D modeling and game development lack capabilities of: 839

842

844 845

846 847

1) on-demand query-based animation composition performed in distributed web-based environments. Instead, they enable visual modeling or programming of animations with locally used tools; 2) animation composition based on application or domain knowledge. Instead, they require the use of technical concepts specific to computer graphics and animation.

Hence, the possible comparison of ACE to other solutions in terms of performance is limited to the process of animating 3D 851 content in the manner permitted by a particular tool, regard852 less of remote or local access to the scenes being customized, 853 and the abstraction level of 3D scene representation. We have 854 compared ACE to Unity in terms of the users’ effort in anima855 tion composition measured as the sum of the number of mouse 856 clicks and keystrokes. 857 To compare both environments, we have created several 858 3D scenes with animated avatars, each of which performs a 859 sequence of exercises. In ACE, we have used VQB to build 860 the scenes, while in Unity, we have used the visual anima861 tion editor. For both environments, we have counted clicks and 862 keystrokes necessary to build the scenes. The formulas below 863 are based on the number of avatars in the modeled scene and 864 the number of exercises that every avatar performs. In ACE, 865 the creation of a query composing a 3D animation requires: 849

The query-based exploration of the animation includes querying its descriptor, followed by reasoning on time-dependent prop- 866 830 erties of the animation. Examples of the exploration of the cre867 831 ated animation encompass the use cases presented in Table 2. 832 The processing of the example queries covers the facts and rules 868 833 specified in lines 1-12 of the animation rule set (Listing 1). 869

828

7. Evaluation

The ACE environment has been compared to the Unity game engine in terms of the effort required for composing animations 837 and evaluated regarding the performance of query transforma838 tion and processing.

850

812 25

d)

836

executesAt ( joe , jFit1 , 0) . instanceOf ( evn_joe_jFit1_jEx1 , event ) . instanceOf ( evn_joe_jFit2_jEx2 , event ) . instanceOf ( evn_joe_jFit3_jEx3 , event ) .

c)

835

848

808 21 809 22

834

duration ( jEx1 , 60) . duration ( jEx2 , 30) . duration ( jEx3 , 90) .

806 19

b)

Figure 7: An example 3D scene including characters with animations of fitness exercises composed on-demand

843

7958

a)

829

11

1) 9 initial clicks and keystrokes, 2) 9 clicks and keystrokes to add an avatar to the scene, 3) 6 and 17 clicks and keystrokes to add a variable and a requirement, respectively,

Table 2: An example of use cases of temporal query-based exploration of the animation Query explanation What time does Alice start the training? What are the exercises done by the characters at the moment equal to 57? How long is Joe training his arm triceps? Is Katy training her legs at the moment equal to 170?

870 871

872 873

874 875

Query instanceOf(E, event), time(E, T), holds(executes(alice, Ex), after(E)), unsaid(prevEx(T)), (prevEx(T) :- holds(executes(alice, SomeEx), after(E2)), time(E2, T2), T2
Response

Example 3D scene

T=30

Figure 7a

JE=squat jump, AE=burpees, KE=squat jump

Figure 7b

Time=30

Figure 7c

Yes

Figure 7d

4) 9 clicks and keystrokes to assign the exercise described by the variable and the requirement to the avatar. The overall formula for X avatars, each performing Y exercises, is the following: ACEclicks+keystrokes = 9 + 9X + (6 + 17 + 9)Y = 9(1 + X) + 32Y

In Unity, the composition of an animation of a single avatar requires 13 initial clicks and keystrokes plus 4 clicks and keystrokes 878 for every exercise assigned to the avatar. The overall formula 879 for X avatars, each performing Y exercises, is the following:

876

877

880

Unityclicks+keystrokes = (13 + 4Y)X

Fig. 8 presents gain calculated as the ratio of Unity to ACE clicks and keystrokes. The gain has been calculated for 3D 883 scenes comprising 1-100 avatars, each performing 2-100 exer884 cises. The gain slightly growths with the increase of the number 885 of avatars in the scene, and it rapidly growths with the increase 886 of the number of exercises assigned to a single avatar. ACE out887 performs Unity in the case of the vast majority of the evaluated 888 scenes except the scenes that have up to 7 avatars. The high889 est gain greater than 10 has occurred for the scene composed 890 of 100 avatars, each performing a sequence of 100 exercises, 891 while the lowest gain has occurred for the scene with 1 avatar 892 performing 100 exercises.

881

882

7.2. Query Transformation In this test, we have measured the time of transformation 895 from XML to Prolog for queries consisting of 100 to 1000 state896 ments (Fig. 9). For every size, we have randomly generated 897 and transformed 20 queries, and calculated the average trans898 formation time. The transformation time is proportional to the 899 square of the number of statements in the query. The time varies 900 from about 7 to 66 milliseconds, which is relatively short tak901 ing into account the large numbers of conditions specified in 902 the queries—each statement in a query expresses a single con903 dition.

Figure 8: Gain in the number of mouse clicks and keystrokes between the Unity and ACE environments depending on the number of avatars in the scene and the number of exercises assigned to a single avatar. Values higher then 1 indicate scenes with lower modeling effort when created using ACE.

893 894

7.3. Query Processing The performance of query processing has been measured as 906 the time of semantic reasoning on the sum of a rule set describ907 ing exercises and a query to the rule set describing conditions 904 905

Figure 9: Time of XML to Prolog query transformation depending on the number of statements in the query.

for an animation. We have completed two tests—modifying the size of queries used to compose the 3D scenes and modifying 910 the size of rule sets describing the exercises. 908 909

7.3.1. Time of query processing depending on the query size We have used a rule set consisting of 1000 statements (rules 913 and facts) and queries of different size in terms of the number 911 912

12

of variables and the number of statements. While the number of variables in a query has been modified within the range of 1 916 to 7, the number of statements has been between 20 and 100. 917 For every combination, 20 different queries have been randomly 918 generated. They have been used to calculate the average pro919 cessing times, which are presented as points in Fig. 10-23. 914 915

Altering the number of variables. Fig. 10-18 depict query processing time for queries including a constant number of state922 ments, but a variable number of variables. As presented in the 923 graphs, adding a new variable to a query is followed by about 924 a tenfold increase of the processing time. Since semantic rea925 soning requires exponential time, equations of exponential re2 926 gression with the coefficient of determination (R ) have been 2 927 determined and shown in the graphs. R is in the range 0.914928 0.986, which denotes a very good determination. The resulting 929 time of query processing for different queries varies from 0.01 930 to tens of thousands of seconds.

920

921

Figure 12: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 40 statements.

Figure 13: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 50 statements.

Figure 10: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 20 statements.

Figure 14: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 60 statements.

follows the increase of the number of statements in the majority of the graphs. However, the relation is much weaker than in the 939 case of the constant number of statements and the increasing 940 number of variables in a query. In two cases (Fig. 21-22), time 941 obtained for larger queries is shorter than for smaller queries, 931 Altering the number of statements. A similar analysis has been 942 which is accidental. 932 done with the focus on the changing number of statements and 933 a constant number of variables in a query (Fig. 19-23). The de943 7.3.2. Time of query processing depending on the rule set size 934 termined curves strongly oscillate around the regression lines, 944 In this test, we have used queries consisting of 10 statements 2 935 which are poorly matched (R in 0.020-0.694) and indicate only 945 with 5 variables and rule sets consisting of a variable number of 936 a general trend. The growth of the query processing time slightly 937

Figure 11: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 30 statements.

938

13

Figure 15: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 70 statements.

Figure 18: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 100 statements.

Figure 16: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 80 statements.

Figure 19: Time of query processing depending on the number of statements in the query. Determined for queries with 1 variable.

Figure 17: Time of query processing depending on the number of variables in the query. Determined for queries consisting of 90 statements.

Figure 20: Time of query processing depending on the number of statements in the query. Determined for queries with 2 variables.

statements within 200 to 1300 (Fig. 24). For every point n the graph, 20 rule sets have been randomly generated. The increase 948 of the number of statements is followed by the exponential in949 crease of the processing time, which is, however, less rapid than 950 in the case of the increasing number of query variables (Fig. 10951 18). A threefold increase of the number of statements causes a 952 tenfold increase of the query processing time. The query pro953 cessing time for different rule sets varies from 1 to 458 seconds. 946 947

954

8. Discussion

The evaluation results show that the proposed approach offers higher performance than Unity in terms of users’ effort in 957 composing animations. The gain increases with the growing 958 complexity of 3D scenes, which include more avatars that per959 form more exercises. Only for 3D scenes with a few avatars, 960 Unity outperforms ACE. The advantage of ACE over Unity for 961 larger scenes is caused by modeling animations at a high level 962 of abstraction (application- or domain-specific) using semantic 963 concepts and hiding technical details from content authors.

955

956

14

Figure 24: Time of query processing depending on the number of statements in the rule set. Determined for queries consisting of 10 statements with 5 variables.

Figure 21: Time of query processing depending on the number of statements in the query. Determined for queries with 3 variables.

used in different parts of the queries and selecting appropriate parts of the rule sets to be verified against the sub-queries. 977 For instance, some sub-queries can be devoted to stimulated 978 body parts, while others—to the difficulty of exercises. Sub979 sets would reduce the number of statements, while sub-queries 980 would reduce the number of statements and possibly the num981 ber of variables in the statements. Once obtained, the results of 982 sub-queries would be joined. Such a fragmentation could de983 crease the overall processing time, however, it depends on the 984 particular application as well as the use case. In some applica985 tions, fragments of rule sets could be distinguished in advance 986 on the basis of the disjoint sets of possible values of variables 987 used in statements based on different predicates. For instance, 988 statements describing effects of exercises are disjoint with state989 ments describing relations between body parts and muscles. In 990 other applications, sub-sets would need to be dynamically built 991 taking into account a particular query. Moreover, the results of 992 frequently issued queries could be cached and provided without 993 redetermining them upon every request. 994 In addition, reasoning on sub-queries and sub-sets could be 995 done in parallel. Such a solution would be sufficient for the an996 alyzed examples to provide customized animated 3D scenes in 997 reasonable time using synchronous client-server communica998 tion on the web. Composing animations for more complex 3D 999 scenes that need queries with over a dozen variables, which re1000 quire more processing time, would demand asynchronous com1001 munication with notifications about the generated 3D content. 1002 For instance, once an animated scene is generated, an e-mail 1003 with its URL is sent to the user. This would prevent 3D ap1004 plications in web browsers from timeouts when waiting for the 1005 response. 1006 In comparison to the query processing time, the time of 1007 query transformation from XML to Prolog and the time of gen1008 erating 3D animations are negligible, and they are no limitation 1009 in the entire process of composing customized 3D content ani1010 mations. 1011 An alternative implementation of the presented approach 1012 could be based on SPARQL, which is a semantic web standard 1013 primarily designed for querying semantic triple-stores. SPARQL 1014 also enables specification of rules and in the implementation 1015 could be combined with RDF, RDFS and OWL. Such imple975 976

Figure 22: Time of query processing depending on the number of statements in the query. Determined for queries with 4 variables.

Figure 23: Time of query processing depending on the number of statements in the query. Determined for queries with 5 variables.

The performance tests show that the approach is efficient enough to solve practical problems that demand rule sets with 966 hundreds of statements and queries with tens of statements. 967 The most important limitation of the approach is the number 968 of variables in queries. This significantly influences process969 ing time in comparison to the number of statements in queries 970 and rule sets. However, the burden of multiple variables in a 971 query could be reduced to some extent. Both the queries and 972 the rule sets can be divided into smaller sub-queries and sub973 sets, which could be further separately processed. Such a di974 vision could be done thematically, taking into account terms

964

965

15

mentation could benefit from some OWL concepts, including classes, properties and restrictions, as well as XSD data types, 1018 on which the standards rely. However, readability of a SPAR1019 QL/OWL implementation would be much lower than in case of 1020 Prolog, which enables uniform and concise expression of facts 1021 and rules as well as elegant methods of combining them. 1022 In terms of the computational properties, reasoning on rules 1023 is undecidable, regardless of the rule description language used 1024 – Prolog or SPARQL. To make the approach decidable, rules 1025 would need to be excluded in favor of decidable OWL profiles 1026 [71] (e.g., OWL 2 RL), which in turn, would prevent reason1027 ing on numerical properties. Hence, the use of event calculus, 1028 including events, fluents, moments and periods in time, would 1029 not be possible.

markers—at the low [75, 76] and high [77] levels of abstraction. This would provide a multi-layered 4-dimensional 3D 1073 content representation. Third, more efficient parallel algorithms 1074 could be developed to divide queries and rule sets, enhancing 1075 the overall process of query-based composition of 3D anima1076 tions.

1016

1071

1017

1072

1077

1078 1079 1080 1081 1082 1083 1084

1030

Unlike the available formats of humanoid animation (e.g., H1032 Anim [72]), the proposed approach addresses animation rep1033 resentation based on concepts specific to an arbitrary applica1034 tion or domain but not low-level representation of human body 1035 with its joints, segments and kinematics. In this regard, our ap1036 proach is orthogonal to the previous solutions. Such low-level 1037 formats can be used in our approach to enable the representa1038 tion of the final composed animation in a form processable by 1039 3D browsers. 1031

1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097

1040

9. Conclusions and Future Works

1098 1099 1100

The use of semantics for building 3D applications gains in- 1101 1042 creasing attention in the research community. It can be espe- 1102 1043 cially useful in web-based VR/AR environments in which ani- 1103 1104 1044 mated 3D content should be created by users with different re1105 1045 quirements based on reusable 3D object libraries stored in com1106 1046 mon repositories. The development of the current web towards 1107 1108 1047 the semantic web offers new technologies that can facilitate cre1109 1048 ation of animated 3D content by people without extensive skills 1110 1049 in 3D graphics and animation. Nonetheless, the available meth- 1111 1050 ods and tools do not enable on-demand composition of 3D con- 1112 1113 1051 tent animations based on application- or domain knowledge. 1114 1052 In this paper, a pipeline of animated 3D content creation 1115 1053 has been proposed. The pipeline is based on query-based se- 1116 1054 mantic composition of 3D content activities into more complex 1117 1118 1055 animations, which can be further explored with users’ queries. 1119 1056 It permits analysis of the past, current and future states of the 1120 1057 animated content. To represent content animations, the event 1121 1058 calculus is used. The representation can cover 3D content ani- 1122 1123 1059 mations at the application- or domain-specific level, and it en1124 1060 ables knowledge inference. This opens new opportunities for 1125 1061 developing tools intelligible to domain experts who are not IT- 1126 1062 specialists. Finally, due to the use of the Prolog declarative lan- 1127 1128 1063 guage, 3D animations created within the pipeline are suitable 1129 1064 for exploration, indexation and search. 1130 1065 The possible future research encompasses several directions. 1131 1132 1066 First, the approach can be combined with solutions for rep1133 1067 resenting interactions between 3D objects as well as between 1134 1068 objects and users [73, 74]. Second, the pipeline can be com- 1135 1069 bined with the knowledge-based representation of other VR/AR 1136 1070 content features, such as geometry, structure, appearance and 1041

16

References [1] Oracle . Java3D. 2019. URL http://www.oracle.com. [2] Microsoft . Direct3D 11 Graphics. 2019. URL https://docs.microsoft.com/. [3] W3C Consortium . VRML Virtual Reality Modeling Language. 1995. URL https://www.w3.org/MarkUp/VRML/. [4] Web3D Consortium . Getting Started with X3D. 2019. URL http://www.web3d.org/getting-started-x3d. [5] DFKI Computergraphics Lab of the Saarland University IVCI. XML3D. 2017. URL https://xml3d.org/. [6] Autodesk . MotionBuilder. 2019. URL https://www.autodesk.com/products/motionbuilder/. [7] Trimble . Sketchup. 2019. URL http://www.sketchup.com. [8] Blender Foundation . Blender. 2019. URL http://www.blender.org. [9] Autodesk . 3ds Max. 2019. URL https://www.autodesk.com.au/products/3ds-max/. [10] Sikos LF. 3D Model Indexing in Videos for Content-based Retrieval via X3D-based Semantic Enrichment and Automated Reasoning. In: Proceedings of the 22Nd International Conference on 3D Web Technology. Web3D ’17; New York, NY, USA: ACM. ISBN 978-14503-4955-0; 2017, p. 1–7. doi:10.1145/3055624.3075943. URL http://doi.acm.org/10.1145/3055624.3075943. ´ de Jes´us CG. GEODIM: A [11] Perez-Gallardo Y, Cuadrado JLL, Crespo AG, Semantic Model-Based System for 3D Recognition of Industrial Scenes. In: Current Trends on Knowledge-Based Systems. Springer; 2017, p. 137–59. [12] Drap P, Papini O, Sourisseau JC, Gambin T. Ontology-Based Photogrammetric Survey in Underwater Archaeology. In: European Semantic Web Conference. Springer; 2017, p. 3–6. [13] Trellet M, Ferey N, Baaden M, Bourdot P. Interactive visual analytics of molecular data in immersive environments via a semantic definition of the content and the context. In: Immersive Analytics (IA), 2016 Workshop on. IEEE; 2016, p. 48–53. [14] Trellet M, F´erey N, Floty´nski J, Baaden M, Bourdot P. Semantics for an Integrative and Immersive Pipeline Combining Visualization and Analysis of Molecular Data. Journal of Integrative Bioinformatics 2018;15 (2):1–19. [15] Kowalski R, Sergot M. A logic-based calculus of events. In: Foundations of knowledge base management. Springer; 1989, p. 23–55. [16] Floty´nski J, Walczak K. Ontology-Based Representation and Modelling of Synthetic 3D Content: A State-of-the-Art Review. Computer Graphics Forum 2017;35:329–353. doi:10.1111/cgf.13083. [17] W3C Consortium . RDF. https://www.w3.org/TR/rdf11-concepts/; 2014. URL https://www.w3.org/TR/rdf11-concepts/. [18] W3C Consortium . RDFS. https://www.w3.org/TR/rdf-schema/; 2014. URL https://www.w3.org/TR/rdf-schema/. [19] W3C Consortium . OWL. https://www.w3.org/TR/owl2-syntax/; 2012. URL https://www.w3.org/TR/owl2-syntax/. [20] Gruber T. Encyclopedia of database systems. http://tomgruber.org/writing/ontology-definition-2007.htm; 2009. URL http://tomgruber.org/writing/ontology-definition-2007.htm. [21] W3C Consortium . SWRL. 2004. URL http://www.w3.org/Submission/SWRL/. [22] Bille W, De Troyer O, Pellens B, Kleinermann F. Conceptual Modeling of Articulated Bodies in Virtual Environments. In: Thwaites H, editor. Proceedings of the 11th International Conference on Virtual Systems and Multimedia (VSMM). Ghent, Belgium: Archaeolingua; 2005, p. 17–26. [23] Kleinermann F, De Troyer O, Mansouri H, Romero R, Pellens B, Bille W. Designing Semantic Virtual Reality Applications. In: In Proceedings of the 2nd INTUITION International Workshop, Senlis. 2005, p. 5–10.

1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207

[24] Pellens B, De Troyer O, Bille W, Kleinermann F, Romero R. An ontology-driven approach for modeling behavior in virtual environments. In: Meersman R, Tari Z, Herrero P, editors. On the Move to Meaningful Internet Systems 2005: OTM 2005 Workshops. Berlin, Heidelberg: Springer Berlin Heidelberg; 2005, p. 1215–24. [25] De Troyer O, Kleinermann F, Pellens B, Bille W. Conceptual modeling for virtual reality. In: Grundy J, Hartmann S, Laender AHF, Maciaszek L, Roddick JF, editors. Tutorials, posters, panels and industrial contributions at the 26th Int. Conference on Conceptual Modeling - ER 2007; vol. 83 of CRPIT. Auckland, New Zealand: ACS; 2007, p. 3–18. [26] De Troyer O, Kleinermann F, Mansouri H, Pellens B, Bille W, Fomenko V. Developing semantic VR-shops for e-Commerce. Virtual Reality 2007;11(2-3):89–106. [27] Guti´errez M. Semantic virtual environments, EPFL 2005;. [28] Guti´errez M, Thalmann D, Vexo F. Semantic Virtual Environments with Adaptive Multimodal Interfaces. In: Chen YPP, editor. MMM. IEEE Computer Society. ISBN 0-7695-2164-9; 2005, p. 277–83. [29] Kalogerakis E, Christodoulakis S, Moumoutzis N. Coupling Ontologies with Graphics Content for Knowledge Driven Visualization. In: VR ’06 Proceedings of the IEEE conference on Virtual Reality. Alexandria, Virginia, USA; 2006, p. 43–50. [30] Attene M, Robbiano F, Spagnuolo M, Falcidieno B. Semantic Annotation of 3D Surface Meshes Based on Feature Characterization. In: Proceedings of the Semantic and Digital Media Technologies 2nd International Conference on Semantic Multimedia. SAMT’07; Berlin, Heidelberg: Springer-Verlag. ISBN 3-540-77033-X, 978-3-540-77033-6; 2007, p. 126–39. [31] Robbiano F, Attene M, Spagnuolo M, Falcidieno B. Part-Based Annotation of Virtual 3D Shapes. 2013 International Conference on Cyberworlds 2007;0:427–36. [32] Attene M, Robbiano F, Spagnuolo M, Falcidieno B. Characterization of 3D Shape Parts for Semantic Annotation. Comput Aided Des 2009;41(10):756–63. doi:10.1016/j.cad.2009.01.003. URL http://dx.doi.org/10.1016/j.cad.2009.01.003. [33] De Floriani L, Hui A, Papaleo L, Huang M, Hendler J. A semantic web environment for digital shapes understanding. In: Semantic Multimedia. Springer; 2007, p. 226–39. [34] Papaleo L, De Floriani L, Hendler J, Hui A. Towards a semantic web system for understanding real world representations. In: Proceedings of the Tenth International Conference on Computer Graphics and Artificial Intelligence; vol. 6. 2007, p. 10. [35] Kapahnke P, Liedtke P, Nesbigall S, Warwas S, Klusch M. ISReal: An Open Platform for Semantic-Based 3D Simulations in the 3D Internet. In: International Semantic Web Conference (2). 2010, p. 161–76. [36] Albrecht S, Wiemann T, G¨unther M, Hertzberg J. Matching CAD object models in semantic mapping. In: Proceedings ICRA 2011 Workshop: Semantic Perception, Mapping and Exploration, SPME. 2011,. [37] Fischbach M, Wiebusch D, Giebler-Schubert A, Latoschik ME, Rehfeld S, Tramberend H. SiXton’s curse - Simulator X demonstration. In: Hirose M, Lok B, Majumder A, Schmalstieg D, editors. Virtual Reality Conference (VR), 2011 IEEE. 2011, p. 255–6. URL http://dx.doi.org/10.1109/VR.2011.5759495. [38] Latoschik ME, Tramberend H. Simulator X: A scalable and concurrent architecture for intelligent realtime interactive systems. In: 2011 IEEE Virtual Reality Conference. IEEE; 2011, p. 171–4. [39] Wiebusch D, Latoschik ME. Enhanced decoupling of components in intelligent realtime interactive systems using ontologies. In: 2012 5th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS). IEEE; 2012, p. 43–51. [40] Floty´nski J, Walczak K. Attribute-based Semantic Descriptions of Interactive 3D Web Content. In: Kiełtyka L, editor. Information Technologies in Organizations - Management and Applications of Multimedia. Wydawnictwa Towarzystwa Naukowego Organizacji i Kierownictwa - Dom Organizatora. ISBN 978-83-7285-691-3; 2013, p. 111–38. [41] Floty´nski J, Walczak K. Microformat and Microdata Schemas for Interactive 3D Web Content. In: Ganzha M, Maciaszek L, Paprzycki M, editors. Proceedings of the 2013 FedCSIS Krak´ow, Poland, 8 - 11 September, 2013; vol. 1. Polskie Towarzystwo Informatyczne. ISBN 978-1-46734471-5; 2013, p. 549–56. [42] Floty´nski J. Harvesting of semantic metadata from distributed 3D web content. In: 2013 6th International Conference on Human System Inter-

1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278

17

actions (HSI). IEEE; 2013, p. 193–200. [43] Bilasco IM, Gensel J, Villanova-Oliver M, Martin H. 3DSEAM: a model for annotating 3D scenes using MPEG-7. In: ISM. IEEE Computer Society. ISBN 0-7695-2489-3; 2005, p. 310–9. [44] Bilasco IM, Villanova-Oliver M, Gensel J, Martin H. Semantic-based Rules for 3D Scene Adaptation. In: Proceedings of the 12th Int. Conference on 3D Web Technology. Web3D ’07; New York, NY, USA: ACM. ISBN 978-1-59593-652-3; 2007, p. 97–100. [45] Guti´errez M, Vexo F, Thalmann D. Semantics-based representation of virtual environments. IJCAT 2005;23(2-4):229–38. [46] Dachselt R, Hinz M, Pietschmann S. Using the AMACONT Architecture for Flexible Adaptation of 3D Web Applications. In: Proc. of the 11th International Conference on 3D Web Technology. Web3D ’06; New York, NY, USA: ACM. ISBN 1-59593-336-0; 2006, p. 75–84. doi: 10.1145/1122591.1122602. [47] Maglo A, Lee H, Lavou´e G, Mouton C, Hudelot C, Dupont F. Remote Scientific Visualization of Progressive 3D Meshes with X3D. In: Proceedings of the 15th International Conference on Web 3D Technology. Web3D ’10; New York, NY, USA: ACM. ISBN 978-14503-0209-8; 2010, p. 109–16. doi:10.1145/1836049.1836066. URL http://doi.acm.org/10.1145/1836049.1836066. [48] Celakovski S, Davcev D. Multiplatform Real-Time Rendering of MPEG4 3D Scenes with Microsoft XNA. In: ICT Innovations 2009. SpringerVerlag Berlin Heidelberg; 2010, p. 337–44. [49] Almer A, Schnabel T, Stelzl H, Stieg J, Luley P. A tourism information system for rural areas based on a multi platform concept. In: Proceedings of the 6th International Conference on Web and Wireless Geographical Information Systems. Springer-Verlag Berlin Heidelberg; 2006, p. 31–41. [50] Tack K, Lafruit G, Catthoor F, Lauwereins R. Platform independent optimisation of multi-resolution 3D content to enable universal media access. The Visual Computer 2006;22, Issue 8:577–90. [51] Walczak K, Rumi´nski D, Floty´nski J. Building contextual augmented reality environments with semantics. In: Proceedings of the 20th International Conference on Virtual Systems & Multimedia VSMM 2014, Hong Kong, 9-12 December 2014. IEEE; 2014, p. 353–61. doi: 10.1109/VSMM.2014.7136656. [52] Walczak K, Floty´nski J. On-Demand Generation of 3D Content Based on Semantic Meta-Scenes. In: Lecture Notes in Computer Science; Augmented and Virtual Reality; First International Conference, AVR 2014, Lecce, Italy, September 17-20, 2014. Springer International Publishing. ISBN 978-3-319-13968-5; 2014, p. 313–32. [53] Floty´nski J, Walczak K. Ontology-Based Creation of 3D Content in a Service-Oriented Environment. In: Abramowicz W, editor. Business Information Systems. Cham: Springer International Publishing; 2015, p. 77–89. [54] Walczak K, Floty´nski J. Semantic Query-based Generation of Customized 3D Scenes. In: Proceedings of the 20th International Conference on 3D Web Technology. Web3D ’15; New York, NY, USA: ACM. ISBN 978-1-4503-3647-5; 2015, p. 123–31. doi:10.1145/2775292.2775311. URL http://doi.acm.org/10.1145/2775292.2775311. [55] Floty´nski J, Krzyszkowski M, Walczak K. Semantic Composition of 3D Content Behavior for Explorable Virtual Reality Applications. In: Barbic J, D’Cruz M, Latoschik ME, Slater M, Bourdot P, editors. Virtual Reality and Augmented Reality. Cham: Springer International Publishing. ISBN 978-3-319-72323-5; 2017, p. 3–23. [56] W3C Consortium . RDQL - A Query Language for RDF. 2004. URL http://www.w3.org/Submission/2004/SUBM-RDQL-20040109/. [57] W3C Consortium . SPARQL. 2013. URL https://www.w3.org/TR/sparql11-query/. [58] W3C Consortium . SPARQL 1.1 Update. 2013. URL http://www.w3.org/TR/sparql11-update/. [59] Broekstra J, Kampman A. SeRQL: An RDF Query and Transformation Language 2004;URL http://gate.ac.uk/sale/dd/related-work/SeRQL.pdf. [60] O’Connor MJ, Das AK. SQWRL: A Query Language for OWL. In: Hoekstra R, Patel-Schneider PF, editors. OWLED; vol. 529. CEURWS.org; 2009,. [61] Smith JR, D¨oller M, Tous R, Gruhne M, Yoon K, Sano M, et al. The MPEG Query Format: Unifying Access to Multimedia Retrieval Systems. IEEE MultiMedia 2008;15(4):82–95. doi:10.1109/MMUL.2008.96. URL http://dx.doi.org/10.1109/MMUL.2008.96.

1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335

[62] Wu Z, Xu G, Zhang Y, Cao Z, Li G, Hu Z. GMQL: A graphical multimedia query language. Knowledge-Based Systems 2012;26:135–43. [63] Daum S, Borrmann A. Definition and implementation of temporal operators for a 4D query language. In: Computing in Civil Engineering (2013). 2013, p. 468–75. [64] Floty´nski J, Krzyszkowski M, Walczak K. Query-based Composition of Animations for 3D Web Applications. In: Proceedings of the 23rd International ACM Conference on 3D Web Technology. Web3D ’18; New York, NY, USA: ACM. ISBN 978-1-4503-5800-2; 2018,doi:10.1145/3208806.3208828. URL http://doi.acm.org/10.1145/3208806.3208828. [65] Floty´nski J, Walczak K. Customization of 3D content with semantic meta-scenes. Graphical Models 2016;88:23–39. doi: http://dx.doi.org/10.1016/j.gmod.2016.07.001. [66] Floty´nski J, Walczak K. Knowledge-based Representation of 3D Content Behavior in a Service-oriented Virtual Environment. In: Proceedings of the 22Nd International Conference on 3D Web Technology. Web3D ’17; New York, NY, USA: ACM. ISBN 978-1-45034955-0; 2017, p. 14:1–14:10. doi:10.1145/3055624.3075959. URL http://doi.acm.org/10.1145/3055624.3075959. [67] Walczak K, Floty´nski J. Inference-based creation of synthetic 3D content with ontologies. Multimedia Tools and Applications 2019;78(9):12607–38. doi:10.1007/s11042-018-6788-5. URL https://doi.org/10.1007/s11042-018-6788-5. [68] Russell S, Norvig P. Artificial Intelligence: A Modern Approach. Upper Saddle River, NJ, USA: Prentice Hall Press; 3rd ed.; 2009. ISBN 0136042597, 9780136042594. [69] Shanahan M. The event calculus explained. In: Artificial intelligence today. Springer; 1999, p. 409–30. [70] Pool J. C#Prolog - A Prolog interpreter written in managed C#. 2017. URL https://sourceforge.net/projects/cs-prolog/. [71] OWL 2 Web Ontology Language Profiles (Second Edition). 2012. URL https://www.w3.org/TR/owl2-profiles/. [72] Web3D Consortium . Humanoid animation (h-anim) component. 2008. URL https://www.web3d.org/working-groups/humanoid-animation -h-anim/. [73] Floty´nski J, Nowak A, Walczak K. Explorable Representation of Interaction in VR/AR Environments. In: Proceedings of AVR 2018, Lecture Notes in Computer Science. Springer International Publishing; 2018, p. 589–609. [74] Floty´nski J, Soboci´nski P. Logging Interactions in Explorable Immersive VR/AR Applications. In: 2018 International Conference on 3D Immersion (IC3D), Brussels, 5-6 Dec. 2018. IEEE. ISBN Electronic ISBN: 978-1-5386-7590-8; 2018, p. 1–8. doi:10.1109/IC3D.2018.8657830. [75] Floty´nski J, Walczak K. Semantic Representation of Multi-platform 3D Content. Computer Science and Information Systems 2014;11, No 4, October 2014:1555–80. [76] Rumi´nski D, Walczak K. Semantic Model for Distributed Augmented Reality Services. In: Proceedings of the 22Nd International Conference on 3D Web Technology. Web3D ’17; New York, NY, USA: ACM. ISBN 978-1-4503-4955-0; 2017, p. 1–9. doi:10.1145/3055624.3077121. URL http://doi.acm.org/10.1145/3055624.3077121. [77] Floty´nski J, Walczak K. Conceptual knowledge-based modeling of interactive 3D content. The Visual Computer 2015;31(10):1287–306. doi:10.1007/s00371-014-1011-9. URL https://doi.org/10.1007/s00371-014-1011-9.

18