Multimedia authoring framework with ILOG™ solver

Computer Communications 23 (2000) 1484–1497 www.elsevier.com/locate/comcom

Multimedia authoring framework with ILOG e solver J. Dospisil*, E. Kendall Monash University, McMahons Rd., Frankston 3199, Australia

Abstract Multimedia presentation delivery characteristics are impacted by the delivery environment including network bandwidth, latency, local disk storage, etc., and this may invalidate the presentation schedule. These varying characteristics are not addressed adequately in the authoring process. We propose a new approach to multimedia presentation scheduling that utilizes constraint technology and a flexible set of scheduling heuristics. The Constraint Satisfaction Authoring Framework aims at increasing the expressive power of a multimedia authoring paradigm with regard to resource capacity conflicts. The approach deals with domain uncertainty and incorporates it into the presentation schedule. q 2000 Elsevier Science B.V. All rights reserved. Keywords: Multimedia authoring; Resource scheduling; Constraint solver

1. Introduction The essence of multimedia authoring is the communication of a message to the reader by the author. In order to achieve this, the author uses an authoring tool to create the presentation schedule. Depending on the features of the authoring tool, the author may be able to vary some presentation parameters such as font and size of text objects, spatial layout of graphical objects, and temporal parameters of continuous media objects. Given the temporal specification of the playout of a temporarily ordered collection of multimedia objects, the quality of presentation depends on multiple components (resources) of multimedia delivery system: operating system, audio–visual subsystem, network bandwidth, and server database retrieval times. The presentation schedules created by any of the existing authoring tools are fixed predictive schedules with good performance on standalone multimedia workstations. They are not suitable for the varying performance of the multimedia delivery systems such as Internet. Although different types of reactive playout services have been suggested in several research projects (e.g. [12,19]) they seem to pay limited attention to the generation of optimized presentation schedules and their seamless integration with media objects. In particular, the model of resources and types of resource constraints is typically simplified or nonexistent. * Corresponding author. Tel.: 1 61-99044135; fax: 1 61-9904124. E-mail addresses: [email protected] (J. Dospisil), [email protected] (E. Kendall).

The overall goal of the Constraint Satisfaction Authoring Framework (CSAF) is to present a flexible, constraint based, authoring framework which can capture varying characteristics of delivery environments and deliver adjustable (elastic) multimedia presentation schedules. The heart of CSAF is the ILOG Constraint Solvere, which is used to generate flexible and optimal presentation schedules. 2. Motivation and background The objects involved in multimedia orchestrated presentations have typically very strict playout requirements. The violation of these requirements in networked multimedia results in subsequent violation of temporal relations (temporal and precedence constraints) between the media objects. The current authoring paradigm does not encourage resource scheduling which would predict and minimize these violations in a way that is highly independent of the underlying transport and highly transparent to the user. Furthermore, the resource allocation services also should prevent the suboptimal utilization of shared resources such as network bandwidth and should encourage users to review their quality needs with regard to the cost of delivery. 2.1. Synchronization and temporal models Within the authoring frameworks, temporal models are the main technique for manipulation with continuous media artifacts, in particular multiple continuous media presentation delivery. A number of approaches which stem out of the simple concept of time have emerged: reference points

0140-3664/00/$ - see front matter q 2000 Elsevier Science B.V. All rights reserved. PII: S01 40-3664(00)0019 2-4

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497 Table 1 Summary of properties of scheduling entities in selected case studies Project/property

HyTime

FireFly

AthenaMuse

Elastic activity duration Compile time scheduler Real-time scheduler Elastic end start variables Complexity of the search heuristics (none, simple, complex) Simple cost and benefit functions Resource usage conflict detection Global resource constraints, capacity conflicts Partial solutions supported

No Yes Limited No None

Yes Yes Yes Yes Simple

No No No No None

No

Yes

No

No

Yes

No

No

No

Not reported

Yes

Yes

Yes

[1,8,13], system time based event generation & Action [10,14], action points event generation [17], and object relationship definition (Zenith project [4]). These temporal models provide good semantics to allow well formed specifications of temporal relationships among media objects and the quality of service requirements. 2.2. Brief summary of current authoring frameworks and toolkits A multimedia presentation can be seen as two-dimensional architecture: content representation, and presentation realization. Content is mapped into the spatial and temporal layout of media objects. The collection of presentation devices is used to support the presentation realization. The examples of recent projects, which incorporate synchronization specification, scheduling, and resource allocation include the following: AthenaMuse [15], Firefly [5], HyTime/SGML [11], MHEG [18], MADE and CMIFed [11], AMOS with its d -sets [12]. Tables 1 and 2 summarize the features relevant to resource allocation and scheduling Table 2 Summary of properties of scheduling entities in the selected case studies MADE and d -sets CMIFed AMOS

MHEG

Elastic activity duration Compile time scheduler Real-time scheduler Elastic end and start variables Complexity of the search heuristics (none, simple, complex) Simple cost and benefit functions

Yes No Yes No None

Yes Yes Yes No Simple

Limited No Yes No None

No

Resource usage conflict detection Global resource constraints, capacity conflicts Partial solutions supported

No No

Other cost No scheme ? No Limited No

Yes

Yes

Project/property

Not reported

1485

processes for the subset of case studies. The summary is based on the available reports in the literature and may be incomplete. During the scheduling phase, these systems treat each resource separately (e.g. CPU, network bandwidth, etc.). Typically, only coarse-grained specifications are used to adjust the presentation parameters for each resource. We believe that the current authoring tools lack an integrated approach to optimal usage of resources. Further more, implementation of rich temporal specifications and strategies to stretch or shrink the activity playout duration with regard to its resource consumption requirements is scarce. In the Nsync toolkit [2], a distributed clock object is used to control synchronization in media streams to support finegrained synchronization, i.e. lip synchronization. The mapping between media time and real time is expressed by the following equation: Media 2 Time ˆ Speed p Real 2 Time 1 Offset The Offset is a scalar used to complete the mapping between media time and real time. The set of constraints is parsed and inserted into the stack for run-time evaluation. Temporal constraints specify the synchronization aspects of a multimedia presentation. The actual presentation realization relies on the set of resources and their internal capabilities to handle the quality of presentation. The GriNS’s playout engine [7] and its channel scheduler build a Petri-net based representation of a hypermedia document. The actual media playout is handled by a set of channel threads which directly controls the quality of playout through presentation devices. Only three of the surveyed systems [3,5,12] use a hybrid approach to resource management: a compile time scheduler to prepare presentation schedules in advance and then a runtime system which resolves fluctuations in media object behavior according to a fixed algorithm. We believe that the current authoring frameworks need richer semantic specification for resource allocation, which includes aspects of user perceived quality and optimization of resource variable capacities.

3. Constraint satisfaction authoring framework The architecture of our Constraint Satisfaction Authoring Framework (CSAF) relies on the following components: Presentation Composition Editor, Predictive Scheduler, and Sensory module. The Predictive Scheduler is further decomposed into two cooperating modules: Scheduling Engine, and Synchronization Enforcement module. An overview of the architecture is given in Fig. 1. The Predictive Scheduler uses its Scheduling Engine (Fig. 2) to create one or more consistent schedules which can be used for temporal management of the presentation and for the enforcement of synchronization among media objects. The dynamically varying internal constraints are represented by a set of quality classes (represented by a group of

1486

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Fig. 1. CSAF architecture.

Precision Objects) which contain information about system internal constraints and intercept via the sensory module possible changes in the multimedia system. The collection of estimation methods within the Precision Objects allows the Predictive Scheduler to gather and statistically process the long-term performance data and to apply appropriate statistical estimation methods to predict the behavior of the system. The Presentation Composition Editor is the subsystem, which interacts with the multimedia presentation author. The output is a coarse grained specification of the spatial and temporal presentation layout. The author’s quality and ordering requirements are communicated in the form of external constraints to the Composer. The Composer’s role is to create composite media objects, associate their handles with the temporal layout requirements, and communicate this information to the Predictive Scheduler. The Sensory module detects system internal constraints and provides feedback information regarding the needs of the presentation at run-time to the Precision Objects. The

Precision Objects maintain knowledge about the environment, as indicated in the diagram. The ILOGe Solver and Scheduler use C11 libraries to implement a standard propagation approach to enforce arcconsistency search algorithm. Constraints, activities, and resources are modeled as C11 classes. This approach proved to be very effective and easy to integrate with the current media control interfaces and composite media objects.

3.1. Resource reservation model and schedule flexibility threshold Let us assume a multimedia scenario j which presents i activities over n resources. All the components of the resource reservation time are variables selected from the admissible time domains: t1 (start time domain (ST)), t2 (end time domain (ET)), t3 (duration domain (d)), and t4 (unexpected delays domain (d )). An activity must reserve a resource for an extended duration Textij as suggested in

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

1487

Fig. 2. Scheduling engine details.

Eq. (1) and depicted in Fig. 3: RESOURCE_RESERVATION_EXTENDED_TIME Textij ˆ STijt1 …c† 1 ETijt2 …c† 1 dijt3 …c; dijt4 †

…1†

Each of the components of the activity duration is a function of the resource available capacity (denoted c). In addi-

tion, the activity duration depends on the size of unexpected delays …dijt4 †: Textij is an explicit entity that resides within the presentation schedule as a result of expected and unexpected events, which influence the actual duration of processing the activity. The upper and lower bounds of resource reservation time change dynamically whenever components of the resource reservation time change. This view is

1488

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Fig. 3. Resource reservation model.

different from that taken by the majority of authoring tools in which the activity duration is modeled as a constant that is independent of the resource. This implies that the resource reservation time is also an indivisible constant. The quality of a multimedia presentation is a function of the activity duration, which in turn depends on the resource presentation parameters and processing constraints (external and internal system constraints). We assume that for each activity Aij the average expected processing duration (allowance) dij is available and the required resource can initially be reserved for this duration. The resource can lack capacity or may have some leftover capacity, both of which will impact the presentation quality. In this context, we define a value—Quality slack time (QSTij) which represents the variations in presentation quality of the activity Aij as a function of its allowance, earliest start time (ESTij) and latest completion time (LCTij) (Eq. (2)) QSTij …c† ˆ LCTij 2 ESTij 2 dij

…2†

QSTij(c) is an explicit entity within the scenario presentation schedule. It is composed of two portions: (1) predictable delays impacting the activity allowance which we termed lower slack; and (2) unpredictable delays impacting the activity allowance which we termed upper slack. Note that as the values ESTij and LCTij are modified the value of QSTij(c) for the activity Aij and possibly other activities will also change. Resource capacity is a function of time as well as a function of Quality slack time. For simplicity, we assume that the resource capacity and the Quality slack time are directly proportional, and we compute the remaining capacity of the resource as: RRCij …t† ˆ RACij …t† 2 RCij …t†; where RRCij is the resource remaining capacity, RACij the resource available capacity, and RCij the resource’s required capacity. The schedule flexibility threshold is approximated as SFT ˆ P TI 2 1 QSTij …c†; where TI is the number of all non-overlap1 ping time intervals in the schedule. We have designed and implemented a scheduling approach that maintains flexibility in the form of a bounded quality threshold in units of time that is an explicit value for

each activity. The sum of QSTij(c) over all non-overlapping time units (intervals) represents the schedule flexibility threshold. This means that as long as the temporal dimensions of the activity remain within the expected bounds and the schedule makespan is within the schedule flexibility threshold there is no need to reinforce synchronization constraints and generate a new schedule.

3.2. Scheduling process and elasticity strategies The multimedia scheduling process is described as a Constraint Satisfaction Problem (CSP): a set of n variables A ˆ {A1 ; …; An } with values in the discrete finite domains Da1 ; …Dan : a set of m constraints C ˆ {c1 ; …cm } where ci : D…A1 † × … × D…Am † ! {true; false} for i ˆ 1; …; m: The problem is to find an assignment a [ D…A1 † × … × D…Am † such that ;m : ci …A† ˆ true: For each activity Ai [ A a variable is introduced. Furthermore, for each activity a domain of possible start times is specified D…Ai † ˆ ‰0; D 2 di Š where di is the activity allowance and at this point it is a constant integer value. The assignment …A1 ; …; AN † [ D…A1 † × … × D…AN † corresponds to a schedule s…Ai † ˆ Ai ; 1 # i # N where s…Ai † [ S and S is the domain of all possible schedules which satisfy the given set of constraints. A solution to the CSP assigns a value to each variable from its respective domain that satisfies all constraints. Three elasticity strategies have been implemented (Fig. 4): energy contention framework, activity elasticity framework, and flexible quality threshold. The activity elasticity {T,Tmax} is determined by three properties: the minimum expected allowance—T, the lower slack time and upper slack time. The allowance is specified during an authoring process. The lower and upper slack time represent a measure of the known and unknown quality fluctuation. The quality fluctuation depends on the performance of the resource used or synchronization policy applied and may be different for

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

1489

Fig. 4. Activity elasticity frameworks.

different resources or policies. The activity allowance bounds are computed as solver preconditions. The resource energy contention strategy is based on the amount of resource energy required to perform a given activity Ali in the time interval t 2 d il , t # t and to meet the quality threshold. Let the commulative resource energy

be E(Rk,t ), the resource energy required by an activity for the activity duration be Ei l …Rk † ˆ di l × Dil : For each time interval t 2 dil , t # t; we can set a constrained condition E il …Rk † ˆ dil; …t1 4 t2 † × Dil …d1 4 d2 †; where …t1 4 t2 † and …d1 4 d2 † are permissible domains for the activity allowance and the resource demand,

Fig. 5. Sample multimedia presentation.

1490

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

bounds and activity allowance to determine the actual resource presentation parameters.

Table 3 Resource classification and abbreviation scheme Resource type

Abbreviation

Audio Video Image or bitmap Text

A V I T

4. Quantitative results summary

respectively. The size of both domains can be established from the system internal constraints, resource presentation parameters, and constraints. The problem then consists of determining a suitable trade-off between the duration of the activity and the utilization of the resource. This is calculated as the cumulative resource energy E(Rk,t ) reduced by the sum of all remaining resource energy requirements Eil within this time interval (Eq. (3)) X Eil …3† Eil …Rk ; t† ˆ E…Rk ; t† 2 t 2 di l ,t#t

To manage quality degradation we introduce the quality threshold strategy. We use the utility function, which is a measure of the user satisfaction with the allocation of the resource capacity. The user specifies the resource requirement in terms of the perceived benefits (benefit function) with regard to the resource presentation parameters and energy requirement e. The expected benefits EB(e) are derived as the difference between the utility and the cost of the quality provided: EB…e† ˆ U…e† 2 P…e†; where U(e) is the expected user utility and P(e) the cost of the resource reservation. The constraint solver then uses resource energy Table 4 Activities and resources used in the example Activity name

Resource type used

Resource type

Activity number

Audio 1 Audio 2 Audio 3 Audio 4 Audio 5 Video 1 Video 2 Video 3 Video 4 Video 5 Video 6 Video 7 Video 8 Video 9 Video 10 Video 11 Video 12 Image 1 Image 2 Image 3 Image 4 Image 5 Text 1

Audio CD—A1 Audio phone A2 Audio CD—A1 Audio phone A2 Audio CD—A1 MPEG—V1 MPEG—V1 MPEG—V1 JPEG—V2 MPEG—V1 JPEG—V2 MPEG—V1 JPEG—V2 MPEG—V1 JPEG—V2 MPEG—V1 MPEG—V1 GIF—I1 GIF—I1 GIF—I1 GIF—I1 GIF—I1 ASCII—T1

Discrete Discrete Discrete Discrete Discrete Discrete Discrete Discrete Discrete Discrete Discrete Discrete Unitary Unitary Unitary Unitary Unitary Unitary Unitary Unitary Unitary Unitary State

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

This section provides results obtained by generating presentation schedules for three consecutive scenarios as shown in Fig. 5. The Scheduling Engine invokes pre-loading of those media objects that would otherwise lead to severe resource congestion. Our sample multimedia application is a complex presentation composed of three scenarios: Introduction, Course details, and Summary. The application is delivered over a network connection, which represents the shared resource of variable capacity. In the Introduction, we use two audio components, two video components, and a viewer with a list of courses (text-1). The Course details scenario is divided into four smaller presentation scenarios according to the course major content. The audio 3 component is used to play background music. The Summary scenario, composed of two video samples and two audio components, provides the summary of available courses. The resource classification and abbreviation scheme are given in Table 3. Activities and associated resources are summarized in Table 4. A discrete resource represents a shared resource with variable capacity. A unitary resource is a resource capacity that is always constant and its usage is limited to one activity at a time (in this case, the scheduler applies disjunctive constraints). The state resource availability for an activity depends on how its state is specified by constraints. The composition process pre-loads the data into the media resource buffers. The initial buffer loading (preload) does not interfere with the presentation process, and we expect that the activities consume only a small amount of energy and last over a short period of time. The actual competition for resources starts at the time t0, which is the actual start of the presentation. We use two video formats (MPEG-1, and M-JPEG), all images use the GIF format (24 bit color depth), and ASCII text is used in the viewer (text-1). Each media type is characterized by a set of presentation parameters P1 –P4, and a set of admissible tolerances within the class expressed as the constraints C1 –C3. Note that the presentation parameters as well as the constraints vary according to the resource type (Table 5). For example, there are no constraints for displaying text in a viewer. The presentation parameters express the assumed demand for resources. For example, we assume that to transfer over the network and display a 1 s video clip (with the frame size 352 × 240 pixels; 8 bit color depth, and presentation rate 30 fps) requires a 2475 kB/s bandwidth and the corresponding CPU time to display frames. If the presentation rate is reduced to 15 fps, the transfer time will be increased

Media object

Object name

P1

P2

P3

P4

C1 tr a

C2

C3 (cf b)

Video MPEG-1 Video JPEG Audio CD Audio phone Picture GIF ASCII text

V1 V2 A1 A2 I1 T1

30 fps 25 fps 16 bit samples 8 bit samples Resolution 300 × 200 Delay to display 1 s

352 × 240 pixel 160 × 120 pixel Playback speed ˆ 1.0 Playback speed ˆ 1.0 4 s delay to display –

8 bits color depth 8 bits color depth

QoS sync ˆ medium QoS sync medium QoS sync medium – – –

2,300,000 1,411,200 1,411,200 – – –

3 dropped frames 4 dropped frames 2 s preload buffer – – –

1:200 1:100 – – 1:2 –

a b

Transfer rate. Compression factor.

24 bits color depth –

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Table 5 Example of various resources presentation parameters and constraints

1491

1492

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Table 6 Critical time intervals Interval

Time

I1 I2 I3 I4 I5 I6 I7 I8 I9 I10

t0 –t8 t8 –t10 t11 –t12 t12 –t13 t14 –t16 t16 –t17 t17 –t19 t19 –t20 t20 –t22 t22 –t26

R

j EARiji , Rmax

230 , 250 120 , 250 235 , 250 180 , 250 160 , 250 180 , 250

R

j EARiji . Emax

280 . 250 270 . 250 275 . 250 275 . 250

accordingly and the required bandwidth can be reduced to 1238 kB/s. The composition server has to reserve sufficient resources (for instance the communication channel bandwidth) to ensure that all data arrive in time and that the communication channel bandwidth is optimally used at all times. Similarly, the presentation server has to plan the admission of other processes in such a way that there is still sufficient resource capacity to present this advertisement with the required quality. Let us assume that we use the shared resource Rj, EARiji is the resource Rj capacity demand by the Rj ˆ 250 is the maximum capacity of the activity Aij, and Emax resource Rj. As a result of the system’s internal constraints (e.g. communication channel bandwidth, and CPU-time available), we cannot assume that the actual resource capacity available at realization will match the demand to meet all the external constraints (user defined quality threshold, precedence constraints, etc.). Table 6 shows the critical time intervals (bold) in which our sample application may experience difficulty due to insufficient shared resource capacity. The fact that the application lacks the resource capacity in some intervals does not mean that the presentation activities Table 7 Abbreviations used in tables and text Abbreviation

Meaning

Rtype ACT EC QT EL PDF

Resource type Activity number Energy contention tradeoff can be applied for the activity Quality threshold values are available to control relaxation Activity elasticity can be applied Activity elasticity calculated as probability density function of known intra-activity gaps Activity allowance supplied by the Composition editor Actual activity duration Activity demand for a resource supplied by the Composition editor (in reservation units) Consumed energy Discrete resource Unitary resource, r denotes its number State resource, r denotes its number Estimated minimum capacity of a discrete resource in demand units Estimated maximum capacity of a discrete resource in demand units

AL Dur DE EN DR UR-r SR-r Cmin Cmax

for all media objects have to be affected. We may choose to reduce data rates for all activities or for only some activities. If there is sufficient disk space, we may pre-load some media and thus reduce the bandwidth requirements at realization. The interesting fact is that, due to the complexity of the problem, it is difficult to predict what will be the final outcome of changes in selected nominal presentation parameters. The possibilities are as follows: • Violated synchronization constraints. • Unexpected differences in presentation quality. For example, using two-color video with insufficient resource capacity may result in a gray scale video. • Frequent presentation interrupts and many dropped frames due to insufficient CPU-time. Our experiments show that all the above effects can occur. The following examples demonstrate the Scheduling engine approach to maintaining the flexible quality threshold and generating optimized valid schedules. 4.1. Examples of generated schedules To demonstrate three frameworks, activity elasticity, energy contention, and flexibility quality threshold management, we have selected the following examples: • estimated variable upper capacity bound of the shared resource; • use of the energy contention framework; and • use of the activity elasticity framework. The flexible quality threshold complements each of these frameworks by restricting or enabling the relaxation that would be otherwise imposed. We also assume that a minimum capacity is needed to establish the connection, preload some media assets, and maintain the requirement of producer–consumer paradigm (typically expressed as playback speed). The Scheduling engine determines this minimum capacity after negotiation with the Sensory module. The capacity demands are in large steps of n p 5 MB/s reservation units where n represents the number of reservation units (e.g. 4 reservation units ˆ 20 MB/s). 1 Abbreviations used in the result tables are summarized in Table 7. We have collected time complexity measurements for all examples (number of variables, number of constraints, number of fails, number of backtracks, and CPU time the Scheduling Engine needed to arrive with a valid optimized schedule). 4.2. Variable upper capacity bound of a shared resource In this example, we vary the estimated maximum and minimum capacity of the shared resource (discrete resource DR). We assume that the shared resource represents a 1 The size of the reservation units can be prescribed by the application author before the actual scheduling process is invoked.

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497 Table 8 Precedence constraints

1493

Table 9 Shared resource usage characteristics

Successor activity

Precedence constraint

Predecessor activity

Delta offset

18 14 19 15 2 2 4 3 1 1 4 3 6 8 7 15 9 9 10 11 12 10 11 11 12 12 16 16 17 20 21 22 23

StartsAfterStart StartsAfterStart StartsAfterStart StartsAfterEnd StartsAfterStart EndsAtStart EndsAtStart EndsAtStart StartsAfterStart StartsAfterStart StartsAfterStart StartsAfterStart StartsAfterStart StartsAfterStart StartsAfterStart EndsAtEnd StartsAfterStart EndsAtStart StartsAfterStart StartsAfterStart StartsAfterStart EndsAtStart StartsAfterStart EndsAtStart StartsAfterStart EndsAtStart StartsAfterEnd StartsAfterStart StartsAtEnd StartsAfterStart StartsAtEnd StartsAfterStart StartsAfterStart

13 13 18 14 15 4 6 5 15 2 2 4 4 6 8 7 1 4 3 5 7 5 5 8 7 7 15 7 16 16 20 21 19

20 0 0 10 0 10 10 10 20 5 10 0 0 10 0 0 0 5 0 0 0 10 0 5 0 5 10 0 0 10 0 0 10

communication channel or similar resource type and its maximum capacity can vary in accordance with the quality threshold as prescribed by the application author. Precedence constraints, which determine the ordering of activities for this example, are given in Table 8. The representation of precedence constraints assumes the following syntax: ksuccessor activityl; k precedence constraintl; k predecessor activityl; …kdelta offsetl†;

Fig. 6. Example of precedence constraint specification.

ACT

Rtype

AL

DE

EN

Dur Cmax ˆ 20

EC

1 2 3 4 5 6 7 8 9 10 11 12

DR DR DR DR DR DR DR DR DR DR DR DR

70 70 70 60 80 80 80 80 10 10 10 10

1 3 3 3 1 3 1 1 2 1 1 1

70 210 210 180 80 240 80 80 20 10 10 10

70 105 105 180 80 120 80 80 10 10 10 10

No Yes Yes Yes Yes Yes Yes Yes No No No No

QT

Yes

Yes

Yes

The graphical example of the above syntax is given in Fig. 6. The capacity bounds are estimated to be in the intervals {1,4} and {1,5} of reservation units including minimum capacity for the connection and setup. These are only estimations; it is the responsibility of the Scheduling Engine to optimize the upper capacity bound. The shared resource usage characteristics by activities are summarized in Tables 9 and 10. The bold values indicate actual activity duration enforced by the Scheduling Engine. Table 9 shows the activities to which the energy contention framework was applied (column heading EC). Table 10 summarizes the unitary resources usage by the remaining activities. The column heading EL indicates which activities will be tested on activity elasticity precondition and which computational method is to be applied (in this case, we have applied Probability Density Function—pdf—to estimate resource fluctuation [9]). The column AL contains the presentation author’s defined allowances and the column Dur shows the corresponding new values. The example deals with the resource configuration of one shared, four unitary and one state resource. The resource configuration is maintained across all examples in order to enable objective time complexity assessment of differences in the scheduling algorithm that is applied. Table 10 Unitary and state resources usage characteristics Activity

Rtype

AL

Dur

EN

EL

EC

13 14 15 16 17 18 19 20 21 22 23

UR-3 UR-3 UR-3 UR-3 UR-4 UR-2 UR-2 UR-1 UR-1 UR-5 SR-1

160 180 400 160 180 180 140 180 160 20 50

176 180 400 184 196 180 140 180 176 20 50

– – – – – – – – – – –

PDF

No No No No No No No No No No No

PDF PDF

PDF PDF

QT

Yes

Yes

Yes

1494

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Fig. 7. Example of variable capacity upper bound for the shared resource DR.

example and the activity elasticity is applied to the selected activities as indicated in column EL. The graphical representation of results for capacity bounds {1,20} is in Fig. 8. Since the maximum capacity is more than sufficient, the activity 5 could retain the prescribed allowance 80. The activities 10, 11, and 12 have the tight presentation constraints and the energy contention framework could not be applied due to authoring restrictions (Table 11, column EC). However, the activity 11 has variable quality threshold and its allowance could be stretched if necessary. There was no need to reorder the activities. The Scheduling Engine also records the amount of unused energy in the peak intervals. However, in the case with the maximum capacity bounds set to {1,12}, Scheduling Engine forces the stretching of the activity 5 to 320. This equalizes the capacity shortage over the interval 450–670 (Table 11).

As shown in Fig. 7, the satisfactory solution for the capacity bounds {1,4} was found after the relaxation of some precedence constraints for activity 2 (audio 2) and activity 3 (audio 3—background music). These constraints have been removed so both activities can be pre-loaded and the data can be stored locally. This example uses the energy contention framework to obtain the acceptable tradeoff between the shared resource available capacity and required quality threshold. Furthermore, it also demonstrates complex scheduling heuristics implemented within the constraint solver. 4.3. Energy contention framework This example shows the impact of the energy contention framework. Most of the activities are allowed to shrink or stretch according to the available resource capacity and the user’s expected benefit function (column EC). The allocation of activities and demands to the shared resources is given in Table 11. As indicated in Table 12, the assignment of the unitary and state resources remains the same as in the previous

4.4. Activity elasticity framework examples The elasticity framework was applied to the activities that require unitary or state resources with demand for a resource

Table 11 Shared resource usage characteristics ACT

Rtype

AL

DE

EN

Dur Cmax ˆ 20

Dur Cmax ˆ 12

EC

1 2 3 4 5 6 7 8 9 10 11 12

DR DR DR DR DR DR DR DR DR DR DR DR

70 70 70 60 80 80 80 80 10 10 10 10

2 2 4 4 4 3 3 3 2 1 1 1

140 140 280 240 320 240 240 240 20 10 10 10

70 70 70 60 80 80 240 240 10 10 10 10

70 70 70 60 320 80 240 240 10 10 10 10

Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No

QT

Yes

Yes

Yes

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497 Table 12 Unitary and state resource usage characteristics

1495

Table 13 Activity elasticity framework examples, unitary resources usage characteristics

ACT

Rtype

AL

DE

EN

EL

EC

13 14 15 16 17 18 19 20 21 22 23

UR-3 UR-3 UR-3 UR-3 UR-4 UR-2 UR-2 UR-1 UR-1 UR-5 SR-1

160 180 400 160 180 180 140 180 160 20 50

1 1 1 1 1 1 1 1 1 1 1

– – – – – – – – – – –

PDF

No No No No No No No No No No No

PDF PDF

PDF PDF

QT

Yes

Yes

Yes

that is always equal to 1. This means that the issue is resource availability rather than capacity. In this case the Scheduling engine cannot record the amount of unused capacity. Resource usage characteristics are summarized in Table 13. We preserved the precedence constraints and the energy contention constraints for the activities using the shared resource as given in Table 11. The impact of system internal constraints is computed using the probability density function (PDF). We assume that we can reconstruct upper and lower bounds of the activity from statistics gathered during playout and that a probability density function can be identified. The upper and lower bounds for a particular resource are held in Precision Objects and used to compute the estimated activity duration. In Fig. 9, only one activity is allowed to stretch (activity 18). All other activities are forced to preserve their allowance. The makespan was computed to 970 s. In Fig. 10, all activities adjusted their duration according to the system internal constraints and the prescribed method of elasticity computation (PDF). The makespan changed to 1059 s after the affected activities were stretched. All precedence constraints were satisfied. The minimum

ACT

Rtype

AL

13 14 15 16 17 18 19 20 21 22 23

UR-3 UR-3 UR-3 UR-3 UR-4 UR-2 UR-2 UR-1 UR-1 UR-5 SR-1

160 180 400 160 180 180 140 180 160 20 50

EL

Dur

PDF

160 180 400 160 180 196 140 180 160 20 50

QT

Yes

Yes

Yes

makespan has become dependent on the number of activities that are allowed to stretch and the expected presentation time changed to 1059 s. 4.5. Time complexity summary The time complexity results (Fig. 11) show the performance of the Scheduling Engine and the search process’ dependency on some indicators, in particular on the number of choice points, and the number of fails. In all examples, the statistics were collected at the time when no valid solution existed and the search was restarted with additional constraint relaxation applied. The chart in Fig. 11 shows that increasing number of choice points rapidly increases the number of fails. In order to assess time complexity performance, we kept the number of variables and the number of constraints constant. When the number of choice points was kept below the 200 mark, the number of fails remained constant too. There is a visible increase in fails corresponding to the increase in backtracks (choice points). The results show that the CPU time (for 486/60 MHz processor) varies slightly around the 20 s mark. The

Fig. 8. Shared resource DR, schedule for wide bandwidth.

1496

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

Fig. 9. Activity elasticity framework (restricted elasticity of the selected activities).

Scheduling Engine shows a great deal of independence of performance on the number of constraints (we have tested up to 120 constraints). Our scheduling algorithm shows the peak in CPU time consumption around 30 constraints and 1500 backtracks which we traced to the impact of the number of tight resource capacity constraints and the number of search restarts. The Scheduling Engine and scheduling algorithm implementation presented in this paper have been entirely implemented using Microsoft Visual C11 and ILOGe SOLVER and SCHEDULER libraries. 5. Conclusion CSAF incorporates uncertainty of the environment such as the varying quality of resources and it detects situations where it is not possible to maintain the requested quality, which is not possible to maintain due to the environmental changes. We have enhanced the coarse grained synchroni-

zation specification supplied by the presentation script with the following constructs: • Activity elasticity strategy incorporates the system internal constraints in the form of a variety of the resource constraints. The activity is a variable whose boundaries can be specified with regard to the system’s internal constraints and the available resource capacity. • Energy contention strategy is based on finding a suitable trade-off between activity duration and its demand for a resource. • Flexible quality threshold strategy is based on the observation that many multimedia applications are scaleable according to the utility function. Using this strategy, we can derive the expected benefit function and relate it to the cost of resource reservation and its presentation parameters. The CSFA framework is intended to support the enforcement of synchronization among multiple media objects with

Fig. 10. Activity elasticity framework with many elastic activities.

J. Dospisil, E. Kendall / Computer Communications 23 (2000) 1484–1497

1497

Fig. 11. Scheduling Engine performance chart.

regard to real-time constraints in networks by mapping network resource constraints into media playout time. Although, the CSAF framework showed good time complexity performance, the current implementation of search heuristics is not fast enough for generating corrected presentation schedules in real time. However, it can be used as a predictive scheduling mechanism for advance resource reservations.

[9]

[10]

[11]

References [1] F. Arbab, I. Herman, G.J. Reynold, An object oriented model for multimedia programming, Proceedings of Eurographics’93, 12(3), Blackwell, Oxford, 1993 (pp. C-101–C-113). [2] B. Bailey, J.A. Konstan, Nsync—A Constraint Based Toolkit for Multimedia. Department of Computer Science, University of Minnesota, 1997. http://www.cs.umn.edu/Research/GIMME. [3] G. Blakowski, J. Hubel, U. Langreher, Tool support for the synchronisation and presentation of distributed multimedia, Computer Communications 15 (10) (1992) 611–618. [4] G. Blair, D. Patrick, N. Davies, J. Mariani, C. Snape, Integrated support for complex objects in a distributed multimedia design environment, Journal of Object-Oriented Programming 6 (8) (1994) 30– 36. [5] C.M. Buchanan, P.T. Zellweger, Automatically generating consistent schedules for multimedia documents, Multimedia Systems 2 (1993) 55–67. [7] D.C.A. Bulterman, L. Hardman, J. Jansen, K.S. Mullender, L. Rutledge, GriNS: A Graphical Interface for creating and playing SMIL documents. Proceedings of the Seventh International World Wide Web Conference (WWW7), April 1998, http://decweb.ethz.ch/ WWW7/1939/com1939.html. [8] N. Correia, N.M. Guimaraes, Time and Synchronisation Objects for

[12]

[13]

[14]

[15] [17]

[18]

[19]

Multimedia Application Construction. Position paper, Fourth Eurographics Workshop on Object Oriented Graphics, 1994. J. Dospisil, Constraint Satisfaction Authoring Framework: Resource Management Service Specification for Distributed Multimedia Playout, PhD thesis, Department of Computer Systems Engineering, Royal Melbourne Institute of Technology, Melbourne, 1998. S. Gibbs, L. Dami, D. Tsichritzis, An Object Oriented Framework for Multimedia Composition and Synchronisation, MULTIMEDIA Systems, Interaction and Applications, in: L. Kjelldahl (Ed.), First Eurographics Workshop, Stokholm, 1991, 18/19. L. Hardman, D.C.A. Bulterman, G. van Rossum, The Amsterdam hypermedia model: adding time and context, Communications of the ACM 37 (2) (1994) 50–63. H. Thimm, W. Klass, d -sets for Optimized Reactive Adaptive Playout Management in Distributed Multimedia Database Systems. Proceedings of the IEEE International Conference on Data Engineering, New Orleans, 1996, pp. 584–592. I. Herman, G.J. Reynolds, MADE: A Multimedia Application Development Environment, Proceedings of the IEEE International Conference on Multimedia Computing and Systems, Boston, May 1994. R.G. Herrtwich, Time Capsules: An Abstraction for Access to Continuous-Media Data. Proceedings of 11th Real-Time Systems Symposium, Lake Buena, Vista, FL, 1990, pp. 11–20. M.E. Hodges, R.M. Sasnett, Multimedia Computing: Case studies from MIT Athena Project, Addison-Wesley, Reading, MA, 1993. T.D.C. Little, A. Ghafoor, Interval-based conceptual models for timedependent multimedia data, IEEE Transactions on Knowledge and Data Engineering 5 (4) (1993) 551–563. B. Markey, Emerging Hypermedia Standards—hypermedia market place prepares for HyTime and MHEG, Proceedings of Summer USENIX Conference, Nashville, TN, 1991, pp. 59–74. M. Papathomas, G.S. Blair, G. Coulson, P. Robin, Addressing the real-time synchronisation requirements of multimedia in an objectoriented framework, IS&T/SPIE Proceedings, vol. 2417, 1995.