Adaptive holistic scheduling for query processing in sensor networks

J. Parallel Distrib. Comput. 70 (2010) 657–670

Contents lists available at ScienceDirect

J. Parallel Distrib. Comput. journal homepage: www.elsevier.com/locate/jpdc

Adaptive holistic scheduling for query processing in sensor networks Hejun Wu a,∗ , Qiong Luo b a

Department of Computer Science, Sun Yat-sen University, China

b

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, China

article

info

Article history: Received 13 April 2009 Received in revised form 20 January 2010 Accepted 6 March 2010 Available online 12 March 2010 Keywords: Wireless sensor networks Query processing Scheduling Performance

abstract We observe two deficiencies of current query processing and scheduling techniques for sensor networks: (1) A query execution plan does not adapt to the hardware characteristics of sensing devices; and (2) the data communication schedule of each node is not adapted to the query runtime workload. Both cause time and energy waste in query processing in sensor networks. To address this problem, we propose an adaptive holistic scheduler, AHS, to run on each node in a wireless sensor network. AHS schedules both the query evaluation and the wireless communication operations, and is able to adapt the schedule to the runtime dynamics of these operations on each node. We have implemented AHS and tested it on real motes as well as in simulation. Our results show that AHS improves the performance of query processing in various dynamic settings. © 2010 Elsevier Inc. All rights reserved.

1. Introduction Query processing in Wireless Sensor Networks (WSNs) is promising for various applications due to its communication efficiency [1,20,21,28,38]. For instance, a habitat monitoring application may want to know when and where the light intensity is below 200. This task can be done by injecting a SQL query ‘‘select light, location from sensors where light < 200, sample period 1 minute’’ into the monitoring WSN. In this WSN, a node samples the light every minute and reports query results only when the predicate (light < 200) is satisfied. Scheduling is indispensable for such query processing for the following three reasons. First, scheduling sensor nodes to sleep is the most effective means to save energy; otherwise nodes will keep idle listening between query execution and communication. Idle listening consumes nearly the same amount of energy as receiving whereas sleeping only consumes about 1/1000 of the energy of idle listening for a given length of time [40]. Second, the multi-hop communication in a WSN requires careful scheduling among nodes. Without communication scheduling, there will be severe packet loss due to collisions or due to transmission to sleeping nodes. Collision is especially severe in in-network query processing, because nodes are time synchronized nodes and they send their results simultaneously epoch by epoch [14,19,30,35]. Third, scheduling is needed to coordinate sensing, computing, and

∗

Corresponding author. E-mail addresses: [email protected], [email protected] (H. Wu), [email protected] (Q. Luo). 0743-7315/$ – see front matter © 2010 Elsevier Inc. All rights reserved. doi:10.1016/j.jpdc.2010.03.004

receiving to ensure a correct order for query result generation and to minimize the time and energy spent in waiting between these operations. Existing scheduling protocols mainly focus on scheduling of wireless communication for reliability and efficiency [4,14,30,33]. As a result, these protocols lack a consideration for sensor query processing on the timing coordination of query evaluation and communication. This deficiency often causes transmission to start before the query results on some nodes and results in data inaccuracy. In our previous work [36], we made an initial attempt on scheduling for in-network query processing in WSNs. We proposed a set of communication rules and proved that they ensure reliable receiving for sensor query processing. Based on these rules, we designed a scheduling protocol, called DCS, that made neighboring nodes negotiate their communication timings and enabled each node to coordinate the query evaluation time in accordance with its communication timing. However, with these existing scheduling schemes, including DCS, we found in our experiments that there remained significant time and energy wastage in query processing in sensor networks [37]. The two major sources of time and energy wastage are: (1) the unnecessary long active time in the schedules of sensing and query evaluation operations on a node and (2) the idling time in the communication schedule of a node. These two problems occur because the actual sensing time, query evaluation time and communication time depend on the runtime workload of a query, but the schedules are produced a priori to support the maximum workload of the query. In this paper we address the time and energy wastage problem in sensor query processing: by making the schedules match

658

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

the actual workload on a node so as to minimize the energy consumption and to reduce the delay of the node. Specifically, we propose an Adaptive, Holistic Scheduling (AHS) scheme that dynamically schedules both the evaluation and the communication of a query to adapt to the query workload on each node. The goal of AHS is to make the active time and idling time as short as possible so as to minimize the energy consumption as well as to reduce the delay. Our contribution lies in the following three aspects. (1) To the best of our knowledge, this is the first work that investigates workload estimation and workload adaptive scheduling for query processing in WSNs. (2) This work demonstrates how to reduce the delay and energy consumption by scheduling query operators and communications together. (3) This work evaluates the costs of operators and communication and scheduling performance on Telos B motes [25] and gives practical suggestions for WSN query processing applications. The remainder of this paper is organized as follows. Section 2 describes the problem scenario and the assumptions, and formulates the problem of AHS. Section 3 shows the overview of AHS and its heuristics. Sections 4–6 detail the design and implementation of the adaptive scheduling mechanisms for communication and operators in AHS. We present our evaluation results in Section 7 and review related work in Section 8. Finally, we conclude this paper and outline future research directions in Section 9. 2. Problem scenario, assumptions and definitions 2.1. Problem scenario Selectively reporting online data is the major type of applications for sensor query processing, since applications that collect all sensory data to a central server for offline processing seldom need in-network processing [16,21,28,39]. Fig. 1 illustrates an example of selectively reporting sensory data. In the WSN in this figure, each node is equipped with two AA batteries, a microcontroller, a wireless module and a few sensing modules. The batteries on a sensor node are energy constrained and it is usually tedious or infeasible to replace the batteries in WSNs. In the WSN shown in Fig. 1, the sink injects a query that requires all nodes to check their temperature readings and those that have a temperature reading larger than 900 to return the query results to the sink every epoch. An epoch, or sample interval, is a fixed time interval for a node to process a query and return its result [20,21,39]. The epoch of the query in Fig. 1 is 60 s. Such a query is called a continuous query since it requires the node to keep sending its query results, if any, every epoch [21]. This kind of continuous query is common in current WSN applications [14,36]. Therefore, our AHS focuses on scheduling these continuous queries in WSNs. Compared with query evaluation in traditional databases, sensor query evaluation has unique characteristics in the following three aspects. (1) Tuple fetching. In traditional databases, all fields in a base tuple are readily available no matter whether they are in a rowbased or column-based storage model. In contrast, a sensor data tuple is a conceptual (virtual) one, and the values of the fields are acquired on-the-fly: a field in a sensor data tuple does not have an actual value until it is sampled by a sensor device [21]. (2) Sampling operator. As each field is independently sampled, sampling is defined as an operator on single attributes. Sampling operators on different attributes in a sensor node can be executed in parallel. A sampling operator can also run in parallel with a nonsampling operator, if the non-sampling operator does not process the result of the sampling operator. (3) Cost distribution. In a sensor node, the major cost of query processing not only includes the cost of active operations, such as computation, receiving and transmission, but also includes the cost of idle listening in waiting for query results from other nodes [36,40].

2.2. Assumptions We list our assumptions on the problem scenario, before defining the scheduling problem. (1) Time synchronization: We assume that all nodes are time synchronized so that each node knows the communication timings of its neighbors. Currently, many mechanisms implement microsecond level time synchronization in WSNs [10,11,23,31,34]. (2) Tree network: Existing sensor query processing systems assume a tree network structure due to its simplicity, duplicatefreeness and communication efficiency [21,26,39]. In a tree network, the sink node is the root. All other nodes in the network report their query results, if any, to the root node every epoch. The parent of a node serves as the router of the node. A neighbor of a node is another node that is within the node’s transmission and receiving range. By this definition, a parent of a node is also a neighbor of the node. Two nodes of the same parent are siblings. (3) Fixed schedule length: We assume that the length of the schedule of each query is fixed on a node, and the schedule length is the same as the length of the sample interval of the query. In sensor query processing, as nodes periodically report query results, if any, in each epoch, this epoch is also the duration of an active/sleep cycle. Hence, a schedule only needs to specify the timings for the operations within one epoch. (4) Fixed slot length. A slot is the time unit in a schedule. As in previous work [14,19,30], we fix the length of a slot to be the time length for transmitting a data packet. Longer slot lengths may cause energy wastage, since sometimes a node may not be able to transmit continuously due to collisions with its neighbors. 2.3. Problem definition We first describe performance metrics such as the energy consumption and the response time of a query. Then, we formally define the scheduling problem based on these metrics. Definition 1 (Node Epoch Energy Consumption (E)). E is the energy consumption of a node within an epoch of a query. E consists of three components: query execution energy consumption (EQ ), communication energy consumption (EC ), and sleeping energy consumption (ES ), denoted in (1). EQ is the energy consumption in executing the query operators; and EC is the energy consumption in transmitting and receiving the data packets. E = EQ + EC + ES .

(1)

Each component of E in is computed by: Ex = UIx Tx

(2)

where U is the node voltage, Ix is the electric current and Tx is the time in performing the operation in that component. Definition 2 (Node Epoch Response Time (T )). T is the interval on a node from the starting time of an epoch to the time that the query result of the node arrives at the sink. T is calculated in (3), where TQ is the query operator execution time, TQR is the needed time for receiving the data from other nodes to generate the query result on this node, since in-network query processing on a node may require the data from other nodes. TOVL is the overlap between TQ and TQR on this node. D(i, 0) is the communication delay for transmitting the query result from this node, denoted as node i, to the sink node, denoted as node 0. T = TQ + TQR − TOVL + D(i, 0).

(3)

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

659

Fig. 1. An application scenario.

TN = max(T1 , T2 . . . , TN ).

(4)

Definition 4 (Energy Efficient Scheduling for Sensor Query Processing in WSNs (EES)). Given a WSN and a query injected on each node of the WSN, a set of slot allocation functions {f 0 } for the WSN, in which the node energy consumption is Ef 0 , find the slot allocation f from {f 0 }, which has the minimum epoch energy consumption: f = argminf 0 {Ef 0 }.

900 800 700 600 0

10

20

30

40

50

60

Time(minues) Fig. 2. Samplings of light during sundown in a campus.

3. Overview

(5) 3.1. Approach

Definition 5 (Delay Efficient Scheduling for Sensor Query Processing in WSNs (DES)). Given the same input as that in Definition 4, and the network response time of the WSN scheduled by f 0 is TNf 0 , find a slot allocation function f , from {f 0 }, which has the minimum network response time: f = argminf 0 {TNf 0 }.

1000

Light

Definition 3 (Network Epoch Response Time (TN )). In a WSN rooted at the sink, node 0, N is the total number of sensor nodes excluding the sink, node i (0 < i ≤ N ) is a sensor node in the network. Ti is the node response time of node i and the network response time

(6)

DES can be reduced to the NP-hard problem defined and proved in the work of Lu et al. [19], which schedules the transmission of one packet for each node per epoch to get the minimal communication delay. Usually, the objective slot allocation function f for DES is different from that for EES, since the most energy efficient schedule on a node is a local optimal one for energy consumption, but DES requires the global optimal schedule to get the fastest result generation and reporting. In processing of continuous queries, energy consumption is much more important than response time, due to the limited energy supply from batteries. Therefore, AHS mainly addresses the EES problem. Meanwhile, AHS attempts to heuristically schedule the nodes to return the results as soon as possible to reduce the response time.

Considering that the energy consumption of sleeping is constant in unit time, the EES problem can be converted to minimize EQ + EC or to maximize ES in Eq. (1), since the epoch length of a query is fixed. Our approach is to assign the slots to a node in such a way that the slots exactly match the amount of workload on the node. This way, the time spent in query execution and communication is minimized on the node. Additionally, we need a mechanism that is able to adapt the schedule of each node to the runtime dynamics. The workload of a node is directly related to the query selectivity on this node. The query selectivity is the ratio between the amount of sensory data that satisfy the query predicate and the amount of sensory data totally sampled. Fig. 2 gives an example of the sensory data from real measurement of a light sensor node in a WSN deployed in our campus during a sundown period. The values are the MICASB sensor readings and can be converted to the standard unit of light [8]. We observe that the sensor readings usually keep constant or have a low variation for certain periods. For instance, in Fig. 2, all of the light readings of the sensor node are lower than 1000 and most of them are higher than 750. Since the sensory values are relatively stable, the query selectivity on a range predicate, e.g., light > 750,

660

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

time, and the average number of retransmission times per epoch. With these statistics, the query plan manager rearranges the order of the operators or the slot manager reallocates slots if needed. 3.3. Workflow overview

Fig. 3. AHS architecture.

will change little over time. This phenomenon enables us to design a slot allocation function to satisfy most of the usual workloads and change occasionally to cater for unexpected workloads. In consideration of the above factors in query processing, AHS takes the following steps in scheduling of a query: First, it estimates the query selectivity; second, it allocates communication slots based on the query selectivity; third, it allocates the slots for query operators in accordance with the communication slots; fourth, it executes and adjusts the schedule for runtime query processing. 3.2. Architecture The architecture of AHS is designed to take the selectivity based adaptive scheduling approach. Fig. 3 illustrates the architecture design of AHS. As shown in this figure, the AHS architecture on each node includes a buffer manager, a query plan manager, an operator executer, a slot manager and a runtime status monitor. The buffer manager stores intermediate results from the query operators on the node and from the children. These intermediate tuples will be sent to the operators for further processing if needed, or will be sent to the parent if the final processing is completed. The query plan manager constructs a query evaluation plan for each query and optimizes the operator execution order to reduce the query execution energy. In a query plan, each operator processes individual attributes instead of materialized tuples. Specifically, the selection operator evaluates a predicate that compares an attribute with a constant or compares two attributes. Such a selection predicate is called a term and is not necessarily an execution unit in traditional databases [27]. The Boolean operations of ‘‘AND’’ and ‘‘OR’’ between the selection operators are called connectors. With this definition, a query plan in AHS differs from that in traditional databases. The slot manager allocates the time slots for query execution and communication. The slot allocation algorithm is similar to the one in our previous scheme, DCS [36], except that the slots are allocated based on estimated selectivities and a query evaluation plan. Note that in AHS, multiple operators share a slot due to their short execution time. Additionally, the slot manager runs a schedule maintenance routine on each node. The interval between two consecutive runs of the maintenance routine is called a maintenance cycle. The length of the maintenance cycle is fixed so that the nodes can communicate in each maintenance cycle. The maintenance routine performs the following four tasks: (1) it broadcasts the information of the node’s transmission slots if they changed; (2) it updates the information of the transmission slots about the neighbors; (3) it reallocates or releases transmission slots due to changes in selectivities; and (4) it re-estimates selectivities, if there are route changes. We will describe the purpose of these tasks in the later sections. The runtime status monitor logs the runtime statistics of the following factors: the historical sensory data, the estimated selectivity of each selection operator, the length of idle listening

The scheduling process of AHS is performed in two phases, first, the initial schedule is set up after query injection; second, the schedule is dynamically adjusted upon query workload changes. When query processing starts, each node keeps a log of a few most recent sensory data and traffic on this node. Then, using this log, the node estimates the selectivity of each query. With the estimated selectivities, the node arranges the communication time and chooses the best execution order for the query operators that will use the shortest execution time. The shortest execution time will also result in the lowest energy consumption in query execution. Then, the node turns back to schedule the start timing of each operator so that the query results, if any, will be ready before the transmission starts. During query processing, each node keeps monitoring the selectivity. When the absolute value of the selectivity change exceeds the system defined threshold, AHS either allocates more slots or releases some slots to adjust to the workload on the node, depending on the sign (+/−) of the change. AHS may also change the execution order of the query operators if such changes reduce the total execution time. In the following, we present the detailed scheduling process of communication and operators in AHS. 4. Workload adaptive wireless communication scheduling Scheduling of wireless communication is performed before that of query operators in AHS. The reason for this arrangement is that neighboring nodes need to coordinate their wireless communication timing first and then they arrange a query evaluation plan based on the communication. This coordination helps in removing or reducing the time gaps between query operator and communication. The time gaps cause frequent switching of running modes, which leads to higher energy consumption and a longer delay on a node. 4.1. Wireless communication scheduling requirements There are three major requirements for wireless communication scheduling in AHS for energy efficiency and reliability: (1) to ensure reliable transmission and receiving; (2) to match and adapt to the workload; and (3) to guarantee that the results of each query are returned within the query epoch. AHS follows the four communication rules in scheduling of wireless communication proposed in our previous work [36]. These four communication rules are listed below. Among them, the first three were proved to be able to avoid collisions. The fourth is to ensure the query results are ready before the transmission on the parent nodes. i. Neighboring nodes should not transmit at the same slot. ii. Siblings should not transmit at the same slot. iii. For any two neighbor nodes A and B, all children of node A should not transmit at the same slot as node B. iv. The transmission slot of a parent node should be later than those of its children if it should aggregate the results of its children. In addition to the rules, we design a set of scheduling strategies and mechanisms for wireless communication in AHS as follows.

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

661

4.2. Communication slot allocation As discussed in the previous sections, selectivity indicates the workload of a query. Hence, to make the communication slots match the workload in the initial schedule, AHS allocates the slots of a node in accordance with the query selectivity. Algorithm 1 shows this allocation process of the Transmission Slots (TS) and Receiving Slots (RS) for a query on each node following the four communication rules. In Algorithm 1, the node starts the slot request when it is ready for TS allocation, as shown in Line 3. Here the ready status is only for the queries that need the in-network aggregation. In this kind of query, the TS of each child should be allocated before the TS allocation on this node. In the other kinds of query, ready is always true. Algorithm 1 Adaptive Query Scheduling Input Query selectivity S, Epoch Ep; Output Transmission slots TS, Receiving slots RS. 1: i = 0, RS = 0, TS = 0, TSAS = 0, TSBK = 0; (slot) 2: while (allocation not finished) do 3: if ((not received TS) and (ready == true)) then 4: /*ready for slot allocation*/ 5: TSRQ .Nr = S × N 6: TSRQ .nonAppSlots = nonAppSlots + RS 7: send TSRQ to the parent 8: end if 9: switch(event) case received TSAS/TSBK from the parent: 10: TS = TS + TSAS /TSBK 11: broadcast TS case received TSRQ from a child: 12: TSAS = TSRQ .num 13: RS = RS + TSAS 14: send TSAS to the child 15: i=i+1 16: if i = = children.num and S < 1 17: TSBK = the first δ slots in the appslots set 18: RS = RS + TSBK 19: broadcast TSBK to children 20: end if 21: break case received TS assignment from a neighbor: 22: mark the slots in this assignment as nonAppSlots 23: break 24: end switch 25: end while The variables in this algorithm are as follows. TSRQ is a request for TS that will be sent to the parent. A TSRQ request includes (1) num: the number of TS required and (2) nonAppSlots: the set of receiving slots on the node and the slots that have been taken up by its neighbors and children. Nr is computed by S × N, where S is the selectivity of the query on this node, and N is the total number of query result packets on this node under the condition that each node has one result. TSAS is the TS set assigned by the parent node to a child. The size of TSAS is the same as that required by the child, i.e., TSRQ .num in Line 4. TSBK the backup TS set, whose size is δ (Line 17). As shown in Algorithm 1, TSBK is shared by all of the children of this node. A child node can use these backup slots for result transmission, when S × N slots are not enough. The determination of this parameter δ and the estimation of S will be described next. When a node receives TSAS or TSBK from its parent, it should broadcast the information of this TS set so that its neighbors will not use them to avoid communication collisions.

Fig. 4. Selectivity computation in the first level.

4.3. Selectivity estimation AHS uses a two-level estimation framework for the selectivity estimation. The motivation is to reduce the estimation overhead while maintaining the estimation accuracy: the second level will only be started if the first level suggests that the difference between the estimated selectivity and the current one exceeds a predefined selectivity threshold; otherwise, it is skipped. In the following, we first discuss the selectivity estimation of a single predicate in this framework. In the first level of our selectivity estimation framework, a linear regression model is used due to its simplicity and its capability to predict the trend of data changes [24]. We found that a linear model is accurate enough in our prototype test on the temperature, light and humidity data in our campus. The reason is that the changes of these sensory data are close to linear when the sample interval is at the minute level, which is the usual time unit in most monitoring applications. In this selectivity estimation framework of AHS, a user may apply different models and thresholds that match his/her data distribution. The data estimation model gives a range of estimated sensory data value [Yˆm , YˆM ]. With this range, AHS uses (7) for the first level estimation, given a range (Pm , PM ) specified in the query predicate. If the query predicate is an open range, we use the value range in the hardware for the open end. The function map([Yˆm , YˆM ], (Pm , PM )) gives the length of the fraction of [Yˆm , YˆM ] that falls within (Pm , PM ), as shown in Fig. 4. SE =

map([Yˆm , YˆM ], (Pm , PM )) YˆM − Yˆm

.

(7)

Yˆm is the estimated minimum value and YˆM is the estimated maximum sensory data value in the next epoch. They are calculated from Yˆm = Yˆ + Em and YˆM = Yˆ + EM , where Yˆ is the estimated sensory data value of the next epoch while Em and EM are the minimum and maximum estimation errors calculated using the training dataset in the execution log on each node under the linear regression model. The training dataset is stored in a queue and is updated using pairs of sampled data values and estimated data values periodically. To avoid the outdated values to be used, AHS sets a timeout to each sensory dataset and the data items of a dataset will be updated in the later epochs when timeout occurs on the dataset. The size of the training dataset is determined by the memory size on each node. In AHS for TelosB motes, we set the training dataset size for each sensor to contain 40 attribute values. In the second level selectivity estimation, Eq. (8) is used. It calculates the average selectivity of the next n epochs, given the predicate (Pm , PM ), Ymˆ(t ) and YMˆ(t ), where Ymˆ(t ) and YMˆ(t ) give the minimum and maximum estimated sensory data values at time t, (t ∈ [ts , tn ]). There are K intersection points between the four curves Ymˆ(t ), YMˆ(t ), Pm , and PM . The time value of each intersection point is {xi }, i ∈ [l, k]. k−1

P R xi+1

SEI =

i=1

xi

map([Ymˆ(t ), YMˆ(t )], (pm , pM ))dt

R tn t1

(YMˆ(t ) − Ymˆ(t ))dt

.

(8)

662

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

WSN:

0 1

3

Query plan of Q-y on node 1:

Query plan of Q-x on node 1:

2 Device p

Device r

Device s

Transceiver

Fig. 6. A example WSN and its query plans.

Fig. 5. Selectivity computation in the second level. Table 1 Time costs of the operators (ms). Temperature

Light

Humidity

Selection

Aggregate

68

19

77

2

5

Fig. 5 illustrates the computation of Eq. (8). The selectivity is the ratio of the shaded area to the area between Ymˆ(t ) and YMˆ(t ) (t ∈ [ts , tn ]). The computation of the integral on a node is as follows. First, it finds the intersection points. In this figure, the number of intersection points, K , is 3. In each segment between two consecutive intersection points, we use the antiderivative of min(Pm , Ymˆ(t )) and max(PM , YMˆ(t )), to compute the shaded area. Since the functions are known in advance, a sensor node can use their antiderivatives directly without computing the antiderivatives due to the CPU capacity limitation. For instance, the function max(PM , YMˆ(t )) between (x1 , x2 ) is YMˆ(t ) and that between (x2 , x3 ) is PM . In query processing, AHS records the selectivity estimation errors so that the scheduling algorithm will use the estimation error to determine the size of backup slots for later queries. The overhead of this selectivity estimation will be discussed together with the scheduling overhead. 5. Scheduling of query operator We first measured the electric current and execution time of each query operator on TelosB motes, as shown in Table 1. The electric current of each operator is 21.8 mA. The delays reported in the data sheet of the sensor chip on a TelosB [25] are slightly shorter than those we measured (e.g., as specified in the data sheet, humidity sensing costs 55 ms instead of 77 ms in Table 1.). The reason is that we included the delay of starting the sampling operator and the chip, whereas the data sheet only specifies the delay from the chip receiving the command to the data being ready. The energy consumption of an operator on a sensor node is give by E = UIT , where U is the voltage of the node, I is the electric current and T is the execution time of the operator. The electric current is measured when the radio is on, since the parent nodes need to listen to the channel during query execution as their receiving slots overlap with the query execution. We pay more attention to the electric current of parent nodes than on leaf nodes in this paper, as leaf nodes consume much less energy than parent nodes. The table shows that the electric current of each operator is the same but the execution times of them are much different on a Telos B mote. Additionally, we found that the electric current of two sampling operators that execute simultaneously is still about 23 mA. The reason may be that the sensor circuit consumes constant energy no matter if it samples or not on Telos B motes. This energy consumption result on a Telos B mote, together with that on a

Mica-2 mote in previous studies [21], indicate that the operator scheduling strategy should consider the sensor type and mote board hardware design, since the energy consumption of sampling varies greatly on different types of hardware. Considering these characteristics of different hardware platforms, we propose a hardware oriented operator scheduling strategy for sensor query processing in AHS: The operators should be executed in parallel if possible and an operator should be terminated whenever its result will be of no use. The result of an operator becomes of no use, when the query can be evaluated without this result. For example, suppose a WSN with AHS is processing Query 1 (Note: the value 1800 is the output value of the temperature sensor and it can be converted to the temperature units by: 0.04 × 1800 − 39.6 = 32.4 °C [29].). A node in this WSN can start all sampling operators simultaneously and terminate all sensing when it finds that its light does not exceed 500. This hardware oriented scheduling strategy is a practical extension to the previous selectivity based query operator scheduling strategy [22]. Query 1 SELECT nodeid from sensors WHERE temperature> 1800 and light > 500 SAMPLE INTERVAL 60 seconds The termination of an operator at runtime is realized by using the timer interrupt on a sensor node running TinyOS 1.x, which does not support preemptive execution. The following illustrates this operator termination procedure. When there is an update of any attribute value, the operator that updates the attribute sets a predefined short time interval to the timer. The time interval is set to the shortest possible time for this timer to interrupt and jump to the timer interrupt routine immediately. In the timer interrupt routine, AHS checks whether this update makes the results of the current running operators to be of no use. Suppose that AHS finds out that the result of an operator will be of no use, it then stops the devices that are being used by the operator and jumps to the next operators to be executed. This runtime operator termination saves energy consumption since it needs shorter execution time than its normal execution. 6. Overall scheduling of query processing 6.1. Example We now illustrate the overall scheduling process, including both the scheduling process of communication and that of operators in a network, using an example network shown in Fig. 6. Suppose in the WSN there are two running queries, an acquisition query Q-x and an aggregation query Q-y, whose epoch lengths are the same. The query plans of these two queries on node 1 are shown in the figure, where the query plan of Q-y illustrates that the aggregate operator c needs to aggregate the result of the children of node 1. Suppose that the sensory data sampling operators, p, r, and s are on different attributes and operator p has longer time than r and s, which have the same execution time. Moreover, suppose the selectivity of operator a is much larger than other operators. Fig. 7 shows two schedules by AHS and DCS [35], respectively. Other existing scheduling protocols would arrange similar slots to

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

663

Fig. 7. Scheduling of query operators.

DCS for this WSN. In the figure, schedule-i is by DCS and schedule-ii is by AHS. Both are for node 1 in the WSN of Fig. 6. In the two schedules shown in Fig. 7, p, r , s are the sampling operators that use different sensing devices on node 1, and they can run simultaneously or run in parallel with another non-sampling operator on this node. The starting times of p, r , s are slightly different, since the CPU of node 1 needs to send a command to start p, r, and s sequentially. Operators a, b, c are the computation operators to generate the query results. In this figure, we use parallel time lines to illustrate that the operators are executed concurrently. In schedule-i, the transmission slot, t1 , is used for forwarding the result of Q-x from the child of node 1, slots t2 and t3 are for transmitting the result of Q-x and Q-y, respectively. In schedule-ii, the transmission slots x1 and x2 are for transmitting the result of Q-x and Q-y on node 1. As can be seen from the comparison between the two schedules in Fig. 7, the two differences are as follows. First, the sleeping time in AHS is much longer than that of DCS in each epoch resulting from selectivity-based dynamic scheduling in AHS. Additionally, since operator b has lower selectivity than a, it is executed earlier than a. When operator b outputs a binary false, operator p will be terminated and operator a will not be executed. This reduces the runtime energy consumption.

the communication time of one packet; TA is the computation time for allocating the slots to the operations such as query operators and communication; TI is the idle listening time between communications; and NC is the number of children. Among these, NB and NC are maintained in the route table; IS , IC , II , TS , TC , TA can be measured in advance, TI is the remaining part of schedule setup time. EO = UIS TS + (2NC + NB + 4)UIC TC + UIC TA + UII TI .

To evaluate the overhead more accurately, TA in Eq. (9) can be further decomposed to a number of time segments, as shown in Eq. (10). In this equation, d is the depth of the query evaluation tree; m(i) is the number of operators at the ith level in the query evaluation tree; t is the average time to schedule an operator; diffculty(i, j) is the difficulty factor in arranging the jth operator at the ith level in the evaluation tree (denoted as operator (i, j)). diffculty(i, j) is one plus the maximum number of other operators whose results operator (i, j) needs to check before its execution. For instance, an operator that is connected to another single operator by ‘‘and’’ needs to check at most the result of one operator before its execution and thus its difficulty factor is 2. Apparently, this difficulty factor helps to include the time in runtime operator scheduling, as presented in the previous section.

6.2. Overhead and tradeoff The scheduling overhead is an important factor in the performance of sensor query processing. In AHS, the scheduling overhead is considered in both schedule setup and update, and its tradeoffs are also taken into account, so that the benefit from scheduling would always be positive. On each node, the schedule setup includes the following seven steps (NB and NC are the number of neighbors and children): 1. Computing of needed number of transmission slots (mainly selectivity estimation); 2. Transmitting a TSRQ packet to its parent (communication times: 1); 3. Receiving a TSRQ packet from its children (communication times: NC ); 4. Transmitting a TSAS packet to each of the children (communication times: NC ); 5. Broadcasting a TSBK packet to all children (communication times: 1 + NB ); 6. Broadcasting its TS and TSBK (communication times: 2 + NB ); 7. Allocating the slots and arranging the order of the query operators. The overhead is accumulated using (9), where U , IS , and IC are the voltage, electric current in selectivity estimation, and average electric current of a node in transmission and receiving; TC is

(9)

TA = t

m(i) d X X

diffculty(i, j).

(10)

i=1 j=1

Suppose a query that needs to run m epochs is injected in the WSN that AHS is running. The benefit of scheduling on each node would be EB = m(UIactiv e Tepoch − UIactiv e Tactiv e − UIsleep Tsleep ) when the schedule makes the node be active for Tactiv e and asleep for Tsleep time. Apparently, as long as (EB − EO ) > 0, the query schedule can be set up on each node. Usually, Tepoch is much larger (about 100 times for a query with an epoch of 60 s on a single node) than Tactiv e . The time overhead in updating a schedule is calculated in Eq. (11). For simplicity, the process of the deduction of Eq. (11) is moved to the Appendix of this paper. TOU =

3h(h + 1) 2

U (IC − II )TC d(1 − )N e.

(11)

With Eq. (19), we can estimate the overhead in a network as follows. Suppose that the network has 10 hops and each hop with 10 nodes (sink not included), also suppose that TC = 60 ms = 0.06 s (60 ms is long enough to transmit a packet on existing sensor nodes.),  is 0.1, (IC − II ) = 2 mA = 0.002 A, the total additional overhead in this worst case is about 1.782 J. On average, each node only needs 0.01782 J, which is close to the energy consumption of one epoch.

664

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

Table 2 Network description.

schedulers with these query scenarios of different predicates and selectivities.

Size

Topology description

Type

100 300 500 6

4-hop, 80 m ∗ 80 m rectangular area 6-hop, 120 m ∗ 120 m rectangular area 8-hop, 160 m ∗ 160 m rectangular area Single-hop

Simulation Simulation Simulation Simulation / TelosB motes

7. Evaluation 7.1. Experiment setup The experiments were performed both on TelosB motes [25] and on a simulator we built. The TelosB motes were used to investigate whether AHS is runable on real WSNs and to measure the parameters that would be used in simulation. The simulator was used to compare the detailed performance of AHS with the other representative schemes. In real mote experiments, we implemented AHS using the nesC language [12] on TinyOS [17] so as to run it on real sensor motes. The code size is 40.4 KB and it needs 5.8 KB RAM during runtime. We installed the code to 6 TelosB motes, deployed them to form the 6-node WSN (Table 2), and then measured the detailed performance of each component on each node. The reason for choosing TelosB motes is that a TelosB mote is equipped with more memory (10KB RAM) than motes such as MICA2 [8] (4KB RAM). With this larger memory, we were able to pre-install a sensory dataset on each node. After each node sampled, it dropped the sensory data and took one item from the pre-installed dataset as the sensory data input for a query. This is to control the selectivity, similar to the selectivity control in the simulator. In simulation experiments, we built a WSN simulator using C++. In this simulator, each node is an instance of the node class and different nodes communicate via the simulated wireless communication interface. This simulator uses the circle communication model as follows: the sensor nodes have the same communication range and a node cannot receive messages when there is more than one of its neighboring nodes transmitting simultaneously. This model is widely used by the WSN community. The transmission range of all nodes is 25 meters, similar to that in existing work [30]. The sensory data are synthesized from the real sensory data using the linear regression model. The selectivity are thus controllable in this simulator by injecting different segments of data to the nodes. The scheduling schemes were implemented as different methods to schedule the queries using the communication interface. The network deployments in the experiments are listed in Table 2. The length of each slot is set to be 60 ms, as we measured the longest packet transmission time of a TelosB mote was about 57 ms. The large networks (100 and 500-node) were used to test the performance such as response time and adaptation of the network. The small network (6-node) was used to study the detailed performance such as schedule adaptation in accordance with the input sensory data. In simulation, the source sensory data for the input of the 6node simulated network are collected from a garden in our campus. The source data for the 100 and 500-node simulated networks are synthesized from the Intel Lab dataset [15] using the linear regression model to interpolate the sensory data to the locations that are not covered in the Intel Lab dataset, since the number of nodes in the original dataset is only 54. Both of the two datasets are for the input of the simulated light and the temperature sensor. The queries used in the simulation and the TelosB mote experiments are Query 2–4 as shown below. Among these queries, Query 2 has two predicates on two different types of sensory data, Query 3 on the same type of sensory data, and Query 4 is a single predicate query. They were to evaluate the performance of the

Query 2 SELECT nodeid from sensors WHERE temp > τ0 and light > ι0 SAMPLE INTERVAL 60 seconds Query 3 SELECT light, nodeid from sensors WHERE light < ι1 and light > ι2 SAMPLE INTERVAL 60 seconds Query 4 SELECT temperature, nodeid from sensors WHERE temp > τ1 SAMPLE INTERVAL 60 seconds With the above input and tools, we compared a few current representative distributed scheduling schemes such as FPS [14], SS [30] and DCS [36]. FPS makes each node choose the unused slots of its parent for transmission. These unused slots are randomly chosen and advertised by the parent node. The problem with this protocol is that a transmission slot chosen by a node may be in conflict with the neighbors of its parent. As a result, the parent node will experience packet loss. SS allows each node to send route and slot requests to other nodes, which may forward the requests to the sink. Once the sink receives the requests it sends back an acknowledgement packet, and the nodes can fix their transmission and receiving slots once they receive the acknowledgement packets to them. The problem with SS is that a slot request goes on too long and may be lost in flooding and this causes many useless communication slots for forwarding on the internal nodes, when the farther nodes did not receive the acknowledgement packets. In addition to these distributed scheduling schemes, a centralized scheme was used in the simulation study. The centralized scheme runs on the sink of a WSN and arranges the schedules of all nodes in the network, assuming that the sink knows the network topology and the routing paths of the WSN. This centralized scheme schedules a randomly chosen node at each time until all nodes in a WSN are scheduled, and the composition of these schedules of the nodes is called a network schedule. This centralized scheme tests a number of such network schedules and chooses the one with the shortest response time. It is also able to adapt to the selectivity of the queries and dynamically updates the schedules of nodes to minimize the energy consumption. Due to its unrealistic assumptions, this centralized scheme is not realistic, but it is able to find a network schedule that has relatively low average energy consumption and short response time, although the network schedule may not be optimal. We also evaluated the performance of query operator scheduling of AHS by comparing it with a static database and a static WSN operator scheduler and the Eddy [2] scheduler. Among these schedulers, the static schedulers did not consider the selectivity changes on the node. The static database scheduler constructed a tuple with all attributes of a node before query execution, and thus wasted a great amount of time in sampling useless attributes. Both Eddy and AHS were designed to adaptively reorder the operators in accordance with the query selectivity to reduce the total query execution delay. In order to focus on the comparison of adaptivity to the selectivity, the Eddy scheduler was extended to only sample the useful attributes. 7.2. Experiments on TelosB motes In experiments on TelosB motes, we first measured the energy consumption of each major component in AHS. Some parts of these results were also used to calculate the energy consumption of

Table 3 Max energy cost of AHS components (micro-Joules). Second-level

Transmission

Receiving

Monitoring

0.441

3.600

3.960

0.070

Energy Consumption (Joules)

First-level 0.189

0.025

AHS

DCS

0.02 0.015

Energy Consumption (Joules)

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

0.24

Central

665 FPS

SS

DCS

AHS

0.2 0.16 0.12 0.08 0.04 0 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0.01

Selectivity

0.005

Fig. 9. Average node energy consumption per epoch.

0 10%

30%

50%

70%

90%

100%

Selectivity of Each Node Fig. 8. Average node energy consumption per epoch.

each node in later simulation, since the other schemes also used the communication and computation operations as in this table. Table 3 shows our measurements of the different components, including the first/second level selectivity estimation, one packet transmission/receiving and status monitoring in running Query 2. We then ran the single-predicate query, Query 4, in the 6-node network to measure the average energy consumption of DCS and AHS. The reason for choosing a single-predicate is to enable us to focus on the relation between the energy consumption and the selectivity of a single predicate so that we can find out the performance trend of the schedulers. Fig. 8 shows the average energy consumption of a node running DCS and AHS within one minute. In comparison with DCS, a node running AHS only consumed about half of the energy consumed by a node running DCS on average, due to the selectivity based scheduling strategies in AHS. The reason is that AHS promptly switched the node to the sleep mode when it found that the query predicate was not satisfied, whereas DCS did not. The figure also shows that the energy consumed on each node was positively correlated to the selectivity on the node. This demonstrates the adaptivity of AHS — the higher a node’s selectivity became, the more transmission slots the node was allocated with. The internal supporting mechanism in AHS for this adaptivity is the dynamic schedule adjustment. As a result of this adaptivity, a node running AHS consumed about 60% of the energy consumption in running DCS, when the query selectivity was 100%. Meanwhile, as the query selectivity was 100% on the node and the query workload would be the same for the node, no matter whether it was running AHS or DCS, it would get the same number of transmission slots in AHS and DCS. However, since AHS could sleep a node in the query layer and AHS enabled the operator and communication pipelining, it reduced the total idling time in query processing. In contrast, DCS fixed the query execution time, which involved long idle listening among query evaluation, receiving and transmission. 7.3. Simulation 7.3.1. Energy consumption In this experiment, we used the energy consumption parameters in Table 3 for the simulated networks in evaluating the energy cost of different schemes. We made the schemes schedule Query 2 and then we calculated the average energy consumption of each node within one minute. During simulation, we also changed the predicates to get 10 levels of average selectivity (0.1–1).

The energy consumption of the scheduling schemes under different predicates are shown in Fig. 9. Among the schemes, the WSNs running AHS had the lowest energy consumption and the energy consumed on each node is positively correlated to the selectivity on each node. This demonstrates the adaptivity of AHS. As shown in the real mote experiments, even when the selectivity was 100%, AHS still made the network consume less energy than other schemes. The reason is that AHS used the hardware-oriented query operator scheduling strategy, which enabled the operators of a query to be executed within a shorter period of time in total than others. Although this saving is not significant (about 0.005 J/min), the accumulation of this saving is high enough to lengthen the network life by about 10%. Moreover, the centralized scheme (marked as Central in Fig. 9) made the networks consume the highest energy among all schemes. The reason is that the centralized scheme keeps each node in the listening mode for a fixed period of time after query processing to receive a new schedule if any. This caused much higher additional energy consumption in the centralized scheme. 7.3.2. Response time We measured the response time of the five scheduling schemes in the 100-, 300-, and 500-node networks under different selectivities of Query 2. The comparisons of the schemes are shown in Fig. 10. In this figure, the response time is denoted using number of slots, since the unit of the schedules is a slot and the number of slots indicates virtual time (the time on a node). The result of the 100-node network shows that, since the schedule on each node was optimized dynamically based on the selectivity, AHS improved the average response time by more than 60% compared with those running other distributed schemes. The schedules of nodes running AHS were updated in accordance with the selectivity whereas the other distributed schemes assigned fixed transmission slots to each node. Furthermore, the response time of AHS and that of the centralized scheme are similar. The reason is that, in addition to the adaptive scheduling, AHS arranged communication and query execution pipelines. This cross-layer scheduling of operator and wireless communication reduced the delay in result reporting and made schedules close to the centralized scheme. In comparison with the larger networks of 300 and 500 nodes with the 100-node network, the differences between the results of AHS and the centralized scheme increased in the larger networks. The differences show the benefit of using global information in the centralized scheme, especially in large networks. In contrast, AHS lacked such information in scheduling, which might cause longer waiting time between the schedules of nodes in communication and query execution. Nonetheless, except for this centralized scheme, AHS significantly outperformed the other schemes due to its workload adaptive scheduling.

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670 FPS

SS

DCS

AHS

300 200 100 0 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

1000

Central

FPS

SS

DCS

AHS

800 600 400 200 0 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

1200 Response Time (slots)

Central

400

Response Time (slots)

Response time (slots)

666

Central

FPS

SS

DCS

AHS

1000 800 600 400 200 0 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

Selectivity

Selectivity

Selectivity

100-node

300-node

500-node

Fig. 10. Response time of the networks.

a

b

Fig. 11. Sensory data and slot serial number on each node.

7.3.3. Adaptation To investigate the detailed runtime adaptation of AHS, we tested the 6-node network with Query 2, in which the predicate is set to be ‘‘light > 850 and light < 900’’, so that the selectivity of the query on each node changes with our source dataset, as shown in Fig. 11a. The dataset were collected from our campus using real sensor motes. This dynamic setting allowed us to observe the changes in the network response time resulting from the schedule optimization in AHS. Fig. 11b shows that AHS adapted to the query workload according to the data dynamics. For instance, the sensory data value of node 2 initially satisfied the predicate. At this stage, the delay of node 2 is 3. When the data value of node 2 changed to be larger than 900, i.e., the predicate was not satisfied, it changed its schedule after a few epochs and its delay was reduced to 2. Similarly such a process can be seen on other nodes. 7.3.4. Packet Loss Since nodes in real world applications might experience signal collapse due to radio noise or low power batteries, we tested the schemes with different packet transmission error rates (Pt ), as defined in Eq. (12). In Eq. (12), ne is the number of packets that experienced transmission error and nt is the total number of transmitted packets. We used the resulting packet loss rate (Pr ), as defined in Eq. (13), to describe the robustness of the schemes, as some schemes were able to retransmit the packets when there were transmission errors. In Eq. (13), nl is the number of packets that are lost. Pt = Pr =

ne nt nl nt

(12)

.

(13)

During the test, we simulated five packet transmission error rates: 10%–50%, since error rates higher than 50% are not realistic. The average resulting packet loss rates per epoch of the schemes on the 100-node network are presented in Fig. 12. AHS significantly outperformed other schemes in terms of reliability. The reason is that AHS reserved slots for receiving on each parent node and the length of the receiving slots was proportional to the transmission error rate. These reserved slots could be used to retransmit the

Fig. 12. Packet loss rates of the schemes under different transmission error rates.

packets that previously failed to be transmitted. Since DCS has fewer slots in conflict with the neighbors on each node and DCS also reserves backup slots [36], it performed better than other schemes. 7.3.5. Query execution delay We finally evaluated the query execution delay in simulation. We ran the AHS and Eddy schedulers, and two static schedulers, to schedule the operators of Query 2 and measure their query execution delay. Fig. 13 shows the delay of the schedulers. In the experiments, we tested the schedulers under different selectivities of s1 (temp > τ0 ) or s2(light > ι0 ) in Query 2. As shown in each graph, only one selectivity was varied and the other one was fixed between s1 and s2 in each experiment [2]. The costs of the s1 and s2 are different, due to the different sampling cost for temperature and light, as shown in Table 1. The graphs show that the static-DB scheduler consumed the longest time in query execution, since it sampled all of the attributes on the node. Moreover, the graphs show that on average the query execution delays of AHS and Eddy were much shorter than the two static schedulers, due to their adaptivity to the query selectivity. Among the four graphs, AHS is found to have the shortest query execution delay, no matter which selectivity is changing.

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

667

Fig. 13. Query execution delay.

The reason is as follows. The Eddy scheduler used a lottery scheme to decide the order of sampling and selection, whereas the AHS scheduler used a deterministic scheme to decide the order. Since the sensory data in WSNs are relatively stable, the deterministic scheme is able to give the order of query operator execution with the lowest cost using the accurately estimated selectivity. In contrast, the lottery scheme sometimes decides an order that is not of the lowest cost due to its randomized nature. Nonetheless, for sensory data such as noises that were not so stable, the Eddy scheduler would outperform AHS. This is because AHS would not be able to accurately estimate the selectivity in such a circumstance, whereas the lottery scheme in the Eddy scheduler could ‘‘compensate’’ this inaccuracy. At this stage, we focus on the stable sensory data such as temperature, light, and humidity. We are considering extending AHS with such a lottery scheme in our future research prototype. 8. Related work We review a number of representative scheduling techniques both in query processing and in networking. In addition, we discuss the current selectivity estimation techniques, since AHS relies on the estimated selectivity. There have been noticeable efforts in studying scheduling for data streams to reduce the response time, memory usage, and slowdown or to maximize output rates. Among these scheduling schemes for streams, the dynamic scheduling and QoS aware scheduling strategies were proposed to reduce the response time and to maximize the output rates in Aurora [5]. The Chain scheduling algorithm mainly addressed the QoS issues in scheduling [3]. Approaches to reduce the slowdown are also studied for heterogonous query processing workloads in streams [28]. Scheduling for query processing in traditional databases usually focuses on result sharing, pipelining, and caching [9,13]. The objective is to avoid redundant processing, to improve parallelism and to reduce intermediate result caching. Moreover, there are two novel scheduling schemes that change query plans at runtime to adapt to the dynamics of data and environments for adaptive query processing [2,33] for traditional databases. Of these two, Eddies [2] focuses on the adaptation to the dynamics

of data while the other one [33] attempts to maximize the output rate through rate-based pipeline scheduling. These studies demonstrate that fixed query plans are inefficient in dynamic environments and that adaptive scheduling is crucial for a robust performance. Due to their high complexity and the different application environments from sensor nodes, the scheduling or optimization techniques in traditional databases are mostly inapplicable for WSNs. Nevertheless, our scheduler shares the same focus on adaptivity with the previous work. Current scheduling schemes in existing in-network sensor query processing, systems such as Cougar [38] and TinyDB [22], fix both the query execution order and the active time on each node. For instance, to save the energy the scheduler in TinyDB puts a node into sleep when the node has been active for four seconds in each epoch. FPS [14], SS [30] and DCS [36], were for general data collection applications without query predicates. FPS is a distributed ondemand scheduling protocol for tree networks. In FPS, a parent assigns slots, randomly chosen from its unused slots, to its children upon request. This helps reduce the collision between the siblings, but does not reduce the collisions among non-sibling neighbors. SS is a cross-layer scheduling protocol that arranges transmission slots while setting up routes. The schedule and route construction time may be extraordinarily long due to the flooding transmission of the setup packets in SS. In DCS, we proposed four rules to avoid collisions occurring on parent nodes so that the children can successfully transmit their results. AHS follows these rules to produce communication schedules, but adds more mechanisms for adapting the schedules to the runtime dynamics. These three schemes can be adopted in query processing, since they are able to allocate each node with slots for transmitting its results. There are also three representative scheduling protocols for special cases of data collection. The work by Lu et al. starts scheduling for nodes that only transmit one packet per epoch [19]. The scheduling protocol by Cao et al. is for event detection applications [4]. This protocol sets a random wake-up time to each node and uses multiple iterations on each node to adjust the wakeup time to reduce the event detection delay. Trigoni et al. proposed

668

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

a wave scheduling protocol that schedules nodes in groups [32]. However, these scheduling schemes are not applicable to query processing, due to their specific design for their target applications: They schedule at most one slot per epoch for each node either for event or short packet reporting. This way of scheduling is not useful for the tasks of query processing, which needs various numbers of slots for query result reporting on the nodes in a WSN. The S-MAC protocol is a Medium Access Control (MAC) protocol that uses synchronized periodic sleep/wakeup scheduling [32]. In S-MAC, a node is time synchronized with its neighbors so that after sleeping for a fixed period, it can wake up simultaneously with its neighbors for communication. In comparison with the Carrier Sense Multiple Access (CSMA) protocols [35], S-MAC reduces the energy wastage from idle listening. Nonetheless, as pointed out in previous work, S-MAC increases the packet delivery latency during sensory data collection [19]. Moreover, if used for sensor query processing, there would still be much energy and time wastage in the periodic wakeup, as the wakeup is not adapted to the pace of query processing. We next study the selectivity estimation techniques in traditional databases and data streams, which are related to the selectivity estimation approach in AHS. In read-dominant traditional databases, the datasets are stable and data update is usually rare. Under this circumstance, the hybrid approach that uses the online sampled data and the previously collected data achieves a high accuracy in selectivity estimation [18]; in comparison, data at each sensor node has strong temporal locality. Hence AHS uses the most recently sampled data as the training dataset for the selectivity estimation. Since the selectivity is computed based on a fraction of data in a database, the estimated selectivity may have errors. Similar to the selectivity estimation in WSNs, in which the current sampled data is used to predict the future, it may also have errors. Chen et al. proposed an adaptive selectivity estimation approach that adaptively adjusts the estimated selectivity using the query feedback and curve-fitting techniques [6]. Our selectivity estimation framework in AHS uses the first level estimation to decide whether to re-estimate the selectivity at runtime. Choi et al. proposed an effective approach for the selectivity estimation for a stream of moving objects [7]. In their approach, the possible future position range of a moving object is estimated as the selectivity for a query of moving objects. This selectivity estimation is similar to that in WSNs in that the sensory data keeps changing although not so frequently. In AHS, we also predict the ranges of the sensory data values and use the ranges to estimate the selectivity in the second level estimation. 9. Conclusion and future work We have presented AHS, our adaptive, holistic scheduler that schedules both query operators and communication. This scheduler bases its schedule decision on the selectivity of each operator and query so as to allocate an efficient schedule to each node, instead of allocating all nodes with the same number of slots, as previous work does. AHS also gives the hardware oriented scheduling strategy that adapts the schedule of each node to the sensing operations. Since AHS allows the nodes to share the unused communication slots, the selectivity estimation errors are well contained. Consequently, AHS is able to reduce both the energy consumption and the response time. Moreover, AHS adapts both the query schedules and the communication schedules to the runtime dynamics. Specifically, the query operators may be reordered using cost-based heuristics and the communication slots can be dynamically shared, allocated or released. This operator and communication slot adaptation occurs on a per query epoch basis so that the query processing

performance can be adjusted quickly to the runtime environment. Our two-level selectivity estimation framework is light-weight and flexible. The experimental results show that AHS outperforms an existing scheduling scheme DCS in both the energy consumption and response time of a WSN. We are further exploring more scheduling policies and query optimization techniques in sensor networks running multiple queries with correlations. Acknowledgments This work was supported by grant 617307 from the Hong Kong Research Grants Council and grant 60903215 from the National Natural Science Foundation of China (NSFC). Appendix Detailed Analysis of Energy Consumption in Section 2.3 EQ and EC can be further broken into smaller parts, as shown in (14) and (15). In (14), M is the number of operators having been executed, POj is the power (Watt) and TOj the execution time of the jth operator. The power of an operation on a node is calculated by U × I, where U is the voltage of the node and I is the electric current of the node running that operation. PIj , TIj are the power and time in idling during executing the jth operator, respectively. In (15), S is the number of packets transmitted, PT and TT are the power and time for transmitting one packet; R is the number of packets received, PR and TR are the power and time for receiving one packet; PI and TI are the energy and time spent in idle listening during communications. EQ =

M X (POj TOj + PIj TIj )

(14)

j =1

EC = S (PT TT ) + R(PR TR ) + PI TI .

(15)

Adaptation and Tradeoff in Scheduling in Section 4.2 The backup parameter δ allows a certain extent of schedule adaptation to different workloads. It is apparent that the larger δ makes the schedules of nodes applicable to a larger range of selectivity. However, the larger δ also causes longer delay on each parent node. This requires a tradeoff between the delay and the size of backup TS set, δ . Considering this, Eq. (16) is designed to determine δ on each parent node. In (16), Erra is the absolute value of the average error of the selectivity estimation; S the query selectivity on this node; NRC the total number of result packets from children under the condition that each child has one result; TRQ the required response time (deadline) of the query and TE the estimated response time of the network. TE is computed using the network delay model of our scheduling algorithm as shown in (17). TE is calculated before query injection and then injected to a WSN together with each query by the sink of the WSN. (TRQ − TE )/TRQ shows the margin in the deadline for the longer delay. This is to ensure that the additional backup slots do not exceed the query deadline.

  TRQ − TE δ = (1 − S )NRC TRQ

TE = TQ + (1 − p)

h X

f (i)NRi + T0 .

(16)

(17)

i =1

In this delay model of a WSN, TQ is the estimated query evaluation time for one query result; p is the ratio of parallel

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

transmission among all of the transmissions of a query in the network; h is the number of hops in the network; f (i) is the forwarding function, which is defined as follows. For a query with in-network aggregation on each parent, f (i) = 1, since no result forwarding is needed; otherwise, f (i) = i, since a result will be forwarded without aggregation; NRi is the number of result packets at hop i under the condition that each node has one query result for the query; T0 is the current used time by other queries in the epoch. Among them, TQ is the query evaluation time of one query result on a leaf node and can be estimated from the query evaluation plan. This is because the query evaluation can be arranged during the receiving time on each internal node and the delay is only caused by the generation of the first result in the WSN. p can be initially fixed on the sink through scheduling the communication of a query and measuring the response time T , and then calculating Ph p by p = i=1 f (i)NRi /T , where h, T0 and NRi are known in the sink. The energy efficiency would become worse if a node kept idle listening in these backup slots. To avoid this energy waste, AHS specifies that each node should add a completion flag in the last packet of query result transmission. When a node did not receive all of the completion flags of its children, it would start listening in the first backup slot and keep listening until timeout, or receive the packets and the completion flags of the children who did not report completion. The timeout value is statistically fixed to the time period after which the probability of receiving new packets is very low (<0.05) in a WSN. In AHS, the value is fixed as 1.5 slots. One problem is that the backup slots may not be enough for all children when the workload increases significantly. To address this, we design the TS reusing and reallocation mechanisms for AHS to address this problem as follows. TS reusing is a mechanism that enables a node to use the unused slots of its sibling for transmission to the parent. In this mechanism, the node monitors the communication of its neighboring siblings, since it knows the transmission slots of them, as demonstrated in Algorithm 1. Recall from Section 2 that a neighboring sibling is both a sibling and a neighbor of the node. If a node detects that one of its neighboring siblings sent the completion flag before its TS ends, it will try to transmit its data during these unused TS of this neighbor sibling. TS reallocation is a mechanism similar to the memory reallocation in computers. In this mechanism, if a node finds out that its TS and the backup slots are not enough for its query result reporting, the node estimates the size of the additional TS required for the packets. With this estimated number of additional TS, the node starts the heuristic scheduling algorithm to send a new TSRQ. Upon receiving this request, the parent will assign more TS to the node in the same way as that in initial schedule setup. On the other hand, if the size of TS of a node is too many for its transmission, it will send a request to its parent for releasing its TS. The released slots will be reallocated by the parent to the other children or by the neighbors. To avoid frequent changing of slots, AHS sets a threshold that specifies the shortest interval between a slot request and a slot release on each node. The value of the threshold is set long enough to make the benefit of slot reallocation to be much larger than its overhead. We will describe the scheduling overhead of both communication and query evaluation in the later sections.

669

Table 4 Symbols used in problem definition (Section 2.3). Symbol

Definition

E EQ EC M POj TOj U I PIj TIj S PT TT R PR TR PI TI TQ TQR TOVL

Energy consumption of a node within an epoch Energy consumption in executing a query in an epoch Energy consumption in communicating the data packets in an epoch Number of operators having been executed Power of the jth operator Execution time of the jth operator Voltage of a node Electric current in running an operation of a node Power in idling during the execution of the jth operator Idling time during the execution of the jth operator Number of packets transmitted Power for transmitting a packet Time for transmitting a packet Number of packets received Power for receiving a packet Time for receiving one packet Energy in idle listening during communication Time in idle listening during communication Query operator execution time Needed time for receiving the data from other nodes Overlap time between TQ and TQR Number of received packets from other nodes Communication delay of a packet from node i to node 0 Network epoch response time Energy Efficient Scheduling Problem in sensory query processing Slot allocation function Energy consumption of a node scheduled by a function f ’ Delay Efficient Scheduling Problem Network epoch response time of a WSN

λ

D(i, 0) TN EES f Ef 0 DES TNf 0

update occurs at the maintenance slots, in which a node keeps idle listening, the overhead of each packet should be more energy in communication than idle listening within the period TC . EU = 3h0 U (IC − Iidle )TC .

(18)

As mentioned in the introduction section, the electric current of idle listening is similar to that of communication (20 mA in idle listening and 22 mA in communication on a Telos B mote [25]), the additional overhead of schedule update is usually negligible. Moreover, as specified in AHS, the schedule update is performed only when the current communication slots of a node in addition to the backup slots are not enough to support its workload. The worst case for AHS schedule update is as follows: Suppose there is a network with N nodes and h hops, the number of nodes at the ith hop (0 < i ≤ h) is Ni . Initially, the nodes of a WSN are only allocated with d(1 − )NC e backup slots, where  is the selectivity estimation error and NC is the average number of children, as their selectivity is estimated as 0. In each epoch, first, there is a node at hop h which finds that it needs to do schedule update and it initiates a schedule update. After that, suppose the nodes at hop h finds the need for schedule update and initiates a schedule update one by one. When all nodes at hop h finishes schedule update, their parents also go through this process one by one. Finally, the nodes at hop 1 finish schedule update of themselves. In total, the total schedule update overhead is accumulated in (19), in which N is the total number of nodes in a network and the other variables are defined. TOU = 3

h X (id(1 − )Ni e)U (IC − II )TC i=1

Overhead Analysis of Schedule Update in Section 6.2

= The overhead of one time schedule update initiated by a node at hop h0 is given by (18) in which the variables are the same as those in (9). There are total 3h times of communication: each node among the path from h to the sink needs to send a TSRQ and to receive a TSAS, and then broadcast its new TS . Since schedule

3h(h + 1) 2

U (IC − II )TC d(1 − )N e.

(19)

Symbols and Acronyms in Section 2.3, Section 4.2, and Section 6.2 See Tables 4–6.

670

H. Wu, Q. Luo / J. Parallel Distrib. Comput. 70 (2010) 657–670

Table 5 Symbols and Acronyms in Algorithm 1 (Section 4.2). TS RS TSRQ TSAS TSBK

δ 

TRQ Te NRC TQ p h f (i) NRi T0

Transmission Slot Receiving Slot Packet of request for TS TS set assigned by a node to its child the backup TS set the size of TSBK Average error of the selectivity estimation Required query response time (deadline) Estimated response time of the network Total number of result packets from children Estimated query evaluation time the ratio of parallel transmissions among all transmissions Number of hops in the network Forwarding function the number of result packets at hop i Start time of the query

Table 6 Symbols used in the overhead analysis (Section 6.2). EO NB NC U IS TC IC TI II TA



EB m Tepoch Tactiv e Iactiv e Isleep Tsleep TOU N

Overhead in terms of energy consumption Number of neighbors Number of children Node voltage Electric current in selectivity estimation Communication time of one packet Average electric current of a node in communication Idle listening time between communications Electric current in idle listening Computation time for allocating the slots Average error of the selectivity estimation Energy saved due to scheduling Query lifetime in terms of epochs Duration of an epoch Active time in an epoch Electric current in active mode Electric current in sleeping Sleeping time in an epoch Time overhead in updating a schedule Total number of nodes in a network

References [1] D.J. Abadi, S. Madden, W. Lindner, Reed: Robust, efficient filtering and event detection in sensor networks, in: VLDB, 2005. [2] R. Avnur, J.M. Hellerstein, Eddies: Continuously adaptive query processing, in: SIGMOD Conference, 2000. [3] B. Babcock, S. Babu, M. Datar, R. Motwani, Chain: operator scheduling for memory minimization in data stream systems, in: SIGMOD, 2003. [4] Q. Cao, T.F. Abdelzaher, T. He, J.A. Stankovic, Towards optimal sleep scheduling in sensor networks for rare-event detection, in: IPSN, 2005. [5] D. Carney, U. Çetintemel, A. Rasin, S.B. Zdonik, M. Cherniack, M. Stonebraker, Operator scheduling in a data stream manager, in: VLDB, 2003. [6] C.-M. Chen, N. Roussopoulos, Adaptive selectivity estimation using query feedback, in: SIGMOD, 1994. [7] Y.-J. Choi, C.-W. Chung, Selectivity estimation for spatio-temporal queries to moving objects, in: SIGMOD, 2002. [8] Crossbow, http://www.xbow.com. [9] N.N. Dalvi, S.K. Sanghai, P. Roy, S. Sudarshan, Pipelining in multi-query optimization, in: PODS, 2001. [10] J. Elson, L. Girod, D. Estrin, Fine-grained network time synchronization using reference broadcasts, in: OSDI, 2002. [11] S. Ganeriwal, R. Kumar, M.B. Srivastava, Timing-sync protocol for sensor networks, in: SenSys, 2003. [12] D. Gay, P. Levis, J.R. von Behren, M. Welsh, E.A. Brewer, D.E. Culler, The nesc language: a holistic approach to networked embedded systems, in: PLDI, 2003. [13] A. Gupta, S. Sudarshan, S. Viswanathan, Query scheduling in multi query optimization, in: IDEAS, 2001. [14] B. Hohlt, L. Doherty, E.A. Brewer, Flexible power scheduling for sensor networks, in: IPSN, 2004.

[15] Intel, Lab data http://berkeley.intel-research.net/labdata. [16] A. Kamra, V. Misra, J. Feldman, D. Rubenstein, Growth codes: maximizing sensor network data persistence, in: SIGCOMM, 2006. [17] P. Levis, S. Madden, D. Gay, J. Polastre, R. Szewczyk, A. Woo, E.A. Brewer, D.E. Culler, The emergence of networking abstractions and techniques in tinyos, in: NSDI, 2004. [18] Y. Ling, W. Sun, N. Rishe, X. Xiang, A hybrid estimator for selectivity estimation, IEEE Trans. Knowl. Data Eng. 11 (2) (1999) 338–354. [19] G. Lu, N. Sadagopan, B. Krishnamachari, A. Goel, Delay efficient sleep scheduling in wireless sensor networks, in: INFOCOM, 2005. [20] S. Madden, M.J. Franklin, J.M. Hellerstein, W. Hong, Tag: a tiny aggregation service for ad-hoc sensor networks, in: OSDI, 2002. [21] S. Madden, M.J. Franklin, J.M. Hellerstein, W. Hong, The design of an acquisitional query processor for sensor networks, in: SIGMOD, 2003. [22] S. Madden, M.J. Franklin, J.M. Hellerstein, W. Hong, Tinydb: an acquisitional query processing system for sensor networks, ACM Trans. Database Syst. 30 (1) (2005) 122–173. [23] M. Maróti, B. Kusy, G. Simon, A. Lédeczi, The flooding time synchronization protocol, in: SenSys, ACM, New York, NY, USA, 2004, pp. 39–49. [24] D.C. Montgomery, E.A. Peck, Introduction to Linear Regression Analysis, second ed., Wiley-Interscience, 1992. [25] C.T. Motes, http://www.xbow.com/products/product_pdf_files/wireless_pdf/ telosb_datasheet.pdf. [26] R. Muller, G. Alonso, Efficient sharing of sensor networks, in: MASS, 2006. [27] R. Ramakrishnan, J. Gehrke, Database Management Systems, third ed., McGraw-Hill, 2003. [28] M.A. Sharaf, J. Beaver, A. Labrinidis, P.K. Chrysanthis, Tina: a scheme for temporal coherency-aware in-network aggregation, in: MobiDE, 2003. [29] SHT11, http://www.sensirion.com/images/getfile?id=25. [30] M.L. Sichitiu, Cross-layer scheduling for power efficiency in wireless sensor networks, in: INFOCOM, 2004. [31] B. Sundararaman, U. Buy, A.D. Kshemkalyani, Clock synchronization for wireless sensor networks: a survey, Ad Hoc Networks 3 (3) (2005) 281–323. [32] N. Trigoni, Y. Yao, A.J. Demers, J. Gehrke, R. Rajaraman, Wave scheduling and routing in sensor networks, TOSN 3 (1) (2007) 2. [33] T. Urhan, M.J. Franklin, Dynamic pipeline scheduling for improving interactive query performance, in: VLDB, 2001. [34] J. van Greunen, J.M. Rabaey, Lightweight time synchronization for sensor networks, in: Wireless Sensor Networks and Applications, 2003. [35] A. Woo, D.E. Culler, A transmission control scheme for media access in sensor networks, in: MOBICOM, 2001. [36] H. Wu, Q. Luo, W. Xue, Distributed cross-layer scheduling for in-network sensor query processing, in: PerCom, 2006. [37] H. Wu, Q. Luo, P. Zheng, L.M. Ni, Vmnet: realistic emulation of wireless sensor networks, IEEE Trans. Parallel Distrib. Syst. 18 (2) (2007) 277–288. [38] Y. Yao, J. Gehrke, The cougar approach to in-network query processing in Sensor Networks, SIGMOD Record 31 (3) (2002) 9–18. [39] Y. Yao, J. Gehrke, Query processing in sensor networks, in: CIDR, 2003. [40] W. Ye, J.S. Heidemann, D. Estrin, An energy-efficient mac protocol for wireless sensor networks, in: INFOCOM, 2002.

Hejun Wu received the Ph.D. degree in Computer Science from The Hong Kong University of Science and Technology in 2008. He is now an assistant professor at Department of Computer Science, Sun Yat-sen University. His research interests are in wireless sensor networks, distributed systems, and embedded systems.

Qiong Luo received the Ph.D. degree in Computer Science from the University of Wisconsin-Madison. She is an associate professor of computer science at the Hong Kong University of Science and Technology. Her research interests are in database systems, distributed systems, pervasive computing, and sensor networks.