Novel clock synchronization algorithm of parametric difference for parallel and distributed simulations

Computer Networks 57 (2013) 1474–1487

Contents lists available at SciVerse ScienceDirect

Computer Networks journal homepage: www.elsevier.com/locate/comnet

Novel clock synchronization algorithm of parametric difference for parallel and distributed simulations Linjun Fan a,⇑, Yunxiang Ling a, Tao Wang b, Xiaomin Zhu a, Xiaoyong Tang c a

Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha 410073, PR China Faculty of Mathematics and Computer Science, Universitat of Bremen, Bremen 28359, Germany c College of Information Science and Technology, Hunan Agricultural University, Changsha 410128, PR China b

a r t i c l e

i n f o

Article history: Received 20 December 2011 Received in revised form 31 December 2012 Accepted 4 February 2013 Available online 10 February 2013 Keywords: Distributed simulation applications Time synchronization Parametric difference clock Synchronization algorithm of parametric difference Local area networks

a b s t r a c t Clock synchronization is crucial in distributed simulation applications (DSAs). Traditional synchronization methods are mainly based on computers’ physical clocks and have network delays. Consequently, the clock jumps easily, and the stability of the algorithm is less than excellent. Furthermore, a new class of DSAs with flexible and dynamically assembled simulation components to optimize the performance of clock clusters is emerging. To remedy the problems, we propose a novel logic parametric difference clock (PDC), which is recursively constructed by an alterable parametric difference frequency (PDF) and its counter. We also develop a simple and efficient synchronization algorithm of parametric difference (SAPD), in which the indeterminate network delay is considered in the PDF, and the PDC counter is used as the synchronous symbol. Compared with existing typical algorithms, the SAPD has the predominant advantages of convergence, stability, and precision for DSAs in local area networks. Ó 2013 Elsevier B.V. All rights reserved.

1. Introduction We consider the clock synchronization issues of parallel and distributed simulation systems with dynamically assembled components in an Ethernet network. In recent years, large-scale distributed applications (e.g., grid computing, cloud computing, pervasive computing, and Internet of Things) have become a major trend in distributed systems [1–5] mainly because of the enormous development of information technologies, power technologies, sensor networks, and systems on a chip. Nowadays, more distributed application systems are increasingly characterized by novel work mechanisms, brand-new physical architecture, and dynamic topology of nodes [6]. Fig. 1 shows this kind of heterogeneous system and its novel applications. The next gen-

⇑ Corresponding author. Tel.: 86 15173117936. E-mail addresses: [email protected] (L. Fan), yxling@tom. com (Y. Ling), [email protected] (T. Wang), [email protected] (X. Zhu), [email protected] (X. Tang). 1389-1286/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.comnet.2013.02.004

eration’s distributed or simulation systems are predicted to exceed traditional ones in local area networks (LANs) and wide area networks (WANs) because of the former’s predicted distribution, generality, gridding, and serviceoriented traits. Especially in simulation fields, serviceoriented composable distributed simulation applications (DSAs) are becoming a popular trend [7]. In general, some of the major features of emerging distributed application systems or DSAs are as follows [6]:  Generality. The novel system is more suitable for constructing a common distributed platform such as Grid and Cloud environment to meet all kinds of requirements than the systems in other application fields. That is, the system, supported by first-class hardware and software, is capable of giving a package solution to many applications.  Dynamic assemblage. The topology of system nodes, which is dynamically made up of unknown distributed nodes, is transformable, unpredictable, and therefore stochastic.

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

1475

Fig. 1. The heterogeneous system and its novel applications.

 Redundancy. Various redundant measures are utilized to strengthen the robustness of the system because of the unstable nodes in new distributed systems.  Staged synchronization. The dynamic changes in topology indicate that many nodes are often synchronized at the beginning and end of some tasks. Clock synchronization is one of the most fundamental issues of each distributed application system or DSA, an age-old problem [8,9] that many researchers have studied. To enable clock consistency in such simulation environments, a scalable and common clock synchronization service is required. In prior studies, many effective methods, such as network time protocol (NTP) [10] in traditional Ethernet, precision time protocol (PTP) [11] in industry control, and Global Positioning System (GPS)-based synchronization [12], are presented to solve this problem. However, these methods have many flaws, such as network delays, missing packages, and high cost. For instance, WSNs have many protocols and algorithms, such as reference broadcast synchronization (RBS) [13], flooding time synchronization protocol (FTSP) [14], and timing synch protocol for sensor networks (TPSNs) [15]. Considering the specific requirements of low battery, durative stability, high security, and narrow bandwidth, many scholars have proposed extended methods to improve the efficiency, robustness, and security of their algorithms [16–23]. However, these algorithms [24] are based on the characteristics of WSN, and the synchronization mechanisms of these algorithms are not well applied to DSAs in LANs because several indefinite factors exist for cluster synchronization in traditional Ethernet and multitask operation systems, such as all kinds of delay in network transport, operation system schedules, packing and unpacking, and function calls [25]. With the popularity of this new DSA class, time synchronization exhibits some new problems (e.g., decreased stability, lower accuracy, and high spending of synchronization) with regard to novel distributed application systems. Moreover, traditional synchronization protocols and algorithms have difficulty keeping clocks in phase with the general application platform of distributed simulation, especially when the entry and exit of simulation nodes are

frequent. To implement steady time synchronicity in such an extended distributed simulation environment, a simple, flexible, and efficient algorithm is required. Considering the features of novel distributed application systems and investigating the advantages and disadvantages of classical synchronization algorithms for traditional networks and WSNs, we propose a novel logic clock model and synchronous scheme that does not modify the physical clock values of local computers. The goal of our algorithm is to explore and resolve the clock synchronization problem of new DSAs (e.g., grid-based and service-oriented DSAs) in the Internet from a new perspective. The main contributions of this paper are as follows:  We introduce a new logic parametric difference clock (PDC) model of heterogeneous clock clusters in our self-designed distributed simulation middleware (runtime infrastructure [RTI] based on service-oriented architecture, which is similar to high-level architecture/RTI).  We mainly propose a novel clock synchronization algorithm of parametric difference (SAPD) that uses PDC in distributed simulation systems but can be used in real distributed systems in LANs. The SAPD sufficiently considers the dynamic changes in systems, the essence of time synchronization, and the staged synchronization requirements of fluctuant nodes. Moreover, by adopting the broadcasting and parametric difference of RBS [13], the interactive synchronicity of the reachback firefly algorithm (RFA) [26], and the multi-broadcast estimation of cooperative time synchronization (CTS) [27], the SAPD provides higher stability within microsecond accuracy and a brand-new method for time synchronization by calibrating the parametric difference frequency (PDF) stringently according to the frequency wave.  We analyze the PDC model and the SAPD theoretically and experimentally and validate their feasibility and efficiency through simulation measurements. The remainder of the paper is organized as follows. In Section 2, we expand the related research on time synchronization in distributed systems, including WSNs and grid systems. In Section 3, we propose a new clock model,

1476

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

PDC, and a novel time-calibrated algorithm, SAPD, which is completely different from existing algorithms. In Section 4, we validate the performance of the proposed algorithm by analyzing the results of the simulation experiments. In Section 5, we provide the conclusion and the recommendations for future work. 2. Related work Related research on time synchronization in distributed systems is presented in the following subsections. Although the focus of this study is on time synchronization, we also review the literature on this domain relevant to our proposed clock model and performance-stable algorithm. 2.1. Concepts of synchronization Generally, synchronization is a constraint relation of timekeeping requiring the coordination of events to operate a system in unison [28]. According to the definition, synchronization is a time-related ordinal relation, that is, synchronized objects should keep a relation of sequence consistency in the entire synchronization process. For some parallel and distributed applications in personal computers and the Internet, the essential feature of synchronization for some events is that systemic logistic actions are well balanced and errorless when serial events occur simultaneously at the system’s running state [29]. Thus, all concurrent systems (e.g., parallel and distributed simulation systems) are principally concerned about how to realize their time synchronization, which is the premise of steady system operations. The solution to event synchronization is the adoption of semaphores, locking, and critical sections in centralized systems [30–32]. However, distributed simulation systems primarily consider time-correlated synchronization solutions because of the heterogeneous and distributed characteristics of such systems. Therefore, the primary synchronizing task in a distributed simulation system is the adjustment of the time of each dispersive node, that is, the logical time maintained by all nodes or by the given ones and the physical time maintained by the nodes. To meet its different requirements, time synchronization is classified into three categories: event sort, relative synchronization, and absolute synchronization [8,13]. Event sort is the simplest synchronous style in which the ordinal relation of events is judged according to their type. Based on the non-jumping clock at each local node, the aim of relative synchronization, with emphasis on not modifying local times directly, is to realize the switching and calibration of local clock values for different nodes by obtaining information, such as clock offsets and drifts, from other nodes. Absolute synchronization implies that all non-referenced nodes synchronize with a fiducially referenced clock node. Thus, the local time may jump because of the probable alteration of time protocol. 2.2. Time synchronization Over the years, many different clock synchronization protocols and algorithms for distributed application sys-

tems have been proposed to maintain the time consistency of global clocks. Moreover, many scholars have presented significant schemes, such as hardware, software, and hybrid schemes, for time synchronization [33]. GPS-based clock synchronization [12,34] is a hardware scheme with very high precision of a submicrosecond and smaller systemic spending, which fits all distributed systems in bare circumstances. However, the signal of GPS-based clock synchronization is very difficult to drill through the barriers, an attribute that makes such synchronization not suitable for given environments (e.g., underwater, indoors, and underground). GPS receivers also require relatively higher energy and cost [35], which are impossible for large-scale deployment. NTP [10,36], proposed by Professor Mills of Delaware University in 1985, is an excellent mechanism because of its robustness, scalability, flexibility, and ubiquitous deployment. NTP is a sender–receiver time synchronization mechanism popular in the traditional Internet and is regarded as the standard Internet time protocol. In particular, the combination of GPS and NTP can reach higher precision of the order of a few microseconds [37]. However, NTP-based schemes have too many weaknesses [38–40] to be applied: indefinite network delay heavy weight implementation, high frequency interchanges, and great expedition. As to the clock skew and network delay, scholars have proposed some algorithms [40–42] to estimate and remove them, but these issues have not been radically resolved. The standard IEEE-1588 (PTP) [11] applied by industrial control fields is a typical protocol of the hybrid synchronization scheme, in which a hardware scheme, such as GPS, assists to attain high synchronization accuracy of the order of nanoseconds while software solutions are not implemented because of some particular requirements. In recent years, the wide application of WSNs has led to the emergence of excellent synchronization algorithms: RBS [13], TPSN [15], FTSP [14], RFA [26], and CTS [27]. The RBS algorithm [13], based on the receiver–receiver mechanism, maintains microsecond-level synchronization by removing the sender’s indeterminacy from the critical path, with the reference beacons sent by nodes to their neighbors at the physical layer. This algorithm needs a broadcast message sent by the third-party node as a reference beacon with no explicit time stamp. The basic process of RBS is as follows. First, the broadcast messages are received by almost all receiving nodes in the identical broadcast area at one time when the time stamp of each receiver is recorded. Then, the time stamps of all receiving nodes are interchanged by sending broadcasts. Finally, by comparing and calculating the receivers’ time difference matrix and clock skew information, global clocks are synchronized with high precision. This algorithm can reach synchronization accuracy of 11 ls on Berkeley motes and 6.29 ± 6.45 ls on Compaq iPAQs running the Linux 2.4 kernel. The greatest advantage of RBS is that it eliminates the largest sources of nondeterministic latency by using the multi-broadcast parametric difference channel to synchronize receivers with each other [33], making it better than NTP. However, the protocol is inapplicable to pointto-point networks and is computationally expensive in

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

large-scale networks requiring Oðn2 Þ message exchanges, which cause long convergence time. Ganeriwal et al. propose a network-wide clock synchronization protocol based on TPSN [15] (which relies on the traditional approach of sender–receiver synchronization [33]) and that is similar to NTP, providing more scalability and steadier precision than RBS because of the two-way message exchange scheme for acquiring the synchronization of two nodes. A hierarchical structure is first established in the WSN, and then the two-way message exchange scheme is used to achieve the time synchronization of two nodes by taking the root node as the reference node. The two phases in TPSN are level discovery and synchronization. The accuracy of TPSN is much higher than that of RBS [33]. However, TPSN is not effective in energy conservation and not suitable for highly mobile applications and multi-hop communications. Another scheme is called FTSP [14], a sender–receiver time synchronization protocol based on the time stamping at the media access control (MAC) layer. By using low communication bandwidth and multi-hop synchronization, FTSP aims to reach network-wide synchronization of the local clocks of the participating nodes [33]. The FTSP achieves its robustness against node and link failures by utilizing the periodic flooding of synchronization messages and implicit dynamic topology updates [14]. The accuracy of this protocol in a single-hop network can attain 1.48 ls, the highest among existing representative time synchronization protocols. This scheme results in the fault tolerance of message losses and network-wide equilibrium time. However, in FTSP, larger bandwidth and weakness of generality are observed because of the use of time stamps at the MAC layer; the clocks of the local nodes may also work backwards when a clock value is modified suddenly to a lower value. As a result, FTSP is used only in WSNs. Geoffrey et al. [26] and Hu and Servetto [27] have presented novel RFA and CTS, respectively, for WSN synchronization. RFA is a biologically inspired intersynchronization algorithm based on a mathematical model originally proposed by Mirollo et al. to interpret the spontaneous synchronization of fireflies’ neurons [43], which has an obvious convergent efficiency. The core of RFA [11,26] is to design a model of pulse-coupled oscillators in which each node is an oscillator periodically emitting a self-generated pulse, and each oscillator adjusts its phase slightly by observing the pulses of other oscillators. The brief feedback step results in the nodes’ tight synchronization by calibrating the phases. Although this algorithm is not yet mature, many scholars have found that the modeling and simulation model is relatively outstanding and that its convergence is proved by TinyOS-based motes for MICAz nodes using TOSSIM [26]. The basic idea of CTS [27] is as follows. First, the reference node sends w-pulses to other neighboring receiving nodes N k (k = 1, 2, . . . , m) at T s (s = 1, 2, . . . , w), while the adjacent nodes receive the synchronous pulse. For a moment, the sending times T s of this pulse are saved in each receiver N k . Then, according to the T s of the recent transmission of m pulses, while the sender’s m þ 1 pulse is transmitting, the sending time T sþ1 of the reference node is estimated by the neighboring nodes, which send synchronous pulses aligned with those of the reference node. The

1477

composite in-phase pulses can reach farther areas because of the overlap of pulse signals. Finally, all nodes in the whole network are synchronized by executing the first two steps repeatedly. Thus, the CTS algorithm has few errors and is suitable for time synchronization in large-scale networks. In sum, the effective novel RFA and CTS algorithms may be improved and prevalently adopted in future WSNs. The main principle of our SAPD is similar to that of RFA but is also partially based on RBS and CTS. Moreover, our algorithm adopts these ideas: the broadcasting and parametric difference of RBS, the multi-broadcast estimation of CTS, and the interactive synchronicity of RFA. In contrast to previous algorithms, the SAPD is based on a virtual clock (i.e., PDC), which does not alter any local physical clock. The SAPD is also common and pervasive for logic time synchronization in novel DSAs grounded on complicated distributed environments (e.g., WSNs, grid systems and traditional LAN and WAN). 3. PDC model In this section, we propose a PDC model based on the timing model of physical clocks. Next, we introduce the synchronization schemes (i.e., single-hop and multi-hop) for PDC. Lastly, we analyze the characteristics of PDC. The notations in this paper are summarized in Table 1. 3.1. PDC construction Before the construction of the PDC, three essential conditions are established: Hypothesis 1. Each network node’s clock is steady in the range of the allowable degree of precision.

Table 1 Definitions of the notation throughout this paper. Notation

Definition

PDC PDF F N f t T (t) R D C (t) P Pn DT P k;n F k;n fk0 C k;n Pn

Parametric difference clock The frequency of PDC The abbreviation of PDF The number of distributed nodes Actual frequency of a compter’s physical clock The current physical time for each node Clock offset from t0 to t The frequency offset of f The excursion rate of f The timing value of physical clock The counting value of PDC The counting value of PDC for the node’s n-step The time interval between adjacent steps The counting value of PDC for the k-node’s n-step Actual PDF for the k-node’s n-step Labeled frequency of a computer’s physical clock The counting value of physical clock for the k-node’s nstep Averaging value for all nodes’ counting values of PDC

vk;n

Parametric-difference factor

FT k;n FBk;n F 0k;n

Peck value of F k;n Lowest value of F k;n Calibrated F k;n

TDI

Tiny deviation interval from P n

1478

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

Hypothesis 2. The network environment is statistically stable. Hypothesis 3. Some noise factors of the Ethernet are not considered in ideal situations. Generally, the model of a computer’s physical clock is as follows:

1 TðtÞ ¼ Tðt0 Þ þ Rðt  t 0 Þ þ Dðt  t 0 Þ2 2

ð1Þ

where t is the current physical time, T (t) is the clock offset from t 0 to t; R is the frequency offset of the clock, and D is the excursion rate of clock frequency due to ageing crystal oscillators. Generally, the frequency offset R is affected by high temperature, whose error scope covers several ppms to hundreds. The error arising from the third item of the right part of Eq. (1) is decided by the equation’s first two terms. According to the analysis, the primary factor in clock deviation is the frequency excursion of crystal oscillators. We assume that computer clocks are stable, that is, the frequency offset R is a fixed value. Therefore, the timing model of physical computer clocks is denoted as follows:

CðtÞ ¼ at þ b

ð2Þ

where a is the ratio of a computer clock’s labeled frequency to the actual one, b is the initial timing offset of computer clocks, and C(t) is the counter of a computer clock. Usually, the actual frequency of a physical computer clock cannot be modified. Therefore, the synchronized physical clock will be out of step after a few moments, and the timing value will jump owing to the direct calibration C(t), resulting in a loss of continuity of the local clocks. The labeled frequencies of computer clocks are actually inaccurate. Consequently, many studies on synchronization have focused on the measurement ways of searching the actual frequency values, and clock time is equal to the value that is the neighboring counting differences divided by the actual frequency value. These methods that measure and calibrate the accuracy of clock frequency significantly influence the counting value, such as tiny changes in a clock’s frequency may result in the acute jumping and backtracking of the counter. As the frequency of physical clocks is not adjusted and the time is also unaltered, avoiding jumping changes, we attempt to construct a special clock with modifiable frequency and timing value. We can establish a virtual clock similar to a computer’s physical clock. Definition 1. P n and Pnþ1 are the recursive counting values of PDC when the synchronous proceedings are in n-step and n þ 1, DT is the time interval between adjacent steps, and F is the frequency of the PDC (i.e., the PDF). The timing model of a fictitious clock is recursively constructed as follows:

Pnþ1 ¼ Pn þ F DT We call this equation the PDC timing model.

3.2. Proposed SAPD In this section, we present the schemes for time synchronization in a single-hop and a multi-hop network based on SAPD. 3.2.1. SAPD in a single-hop network To the best of our knowledge, the single-hop network is characterized by low delay and a simple planar topology structure and is thus suitable for the time synchronization of distributed clock clusters based on the PDC. However, the PDC is implemented based on the multicast/broadcast parametric difference. Hence, we must constrain the size of the network to prevent a broadcast storm. In this section, the definitions and equations for SAPD construction are presented. Definition 2. Pk;n is the counting value of the PDC, F k;n is the actual frequency of the PDC, fk0 is the marking frequency of a computer’s physical clock, and C k;n is the counting value of a physical clock. The counting value of the PDC is recursively defined as follows:

Pk;n  Pk;n1 ¼ F k;n1 

C k;n  C k;n1 fk0

ð4Þ

The average of the total counting values of all nodes in the parametric difference adjustment is as follows:

Pn ¼

m 1X Pk;n m k¼1

ð5Þ

In Eq. (5), k is the serial number of the nodes, n is the step number, and m is the amount of the nodes. Definition 3. In PDF calibration, the parametric difference factor is kk;n when the k-node’s n-step is in process. PDF is equal to the ratio of the average counting value of the PDC for all nodes P n to that of the current node P k;n :

kk;n ¼

Pn P k;n

ð6Þ

To converge as soon as possible, when kk;n > 1, the node’s PDF should be increased. When kk;n < 1, the PDF should be diminished. When kk;n  1, the PDF reaches a stable state. Based on kk;n , PDF is adjusted by the following recursive formula:

F k;n ¼ kk;n1  F k;n1

ð7Þ

Definition 4. The peak value of the PDF for the k-node’s nstep is FT k;n , and the lowest value of PDF is FBk;n . F 0k;n is the convergent frequency of the PDC:

F 0k;n ¼

FT k;n þ FBk;n 2

ð8Þ

In a single-hop network, the algorithm consists mainly of two parts: zero-point time calibration and PDC synchronization (Fig. 2).

ð3Þ  Zero-point time calibration

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

1479

Fig. 2. The processes of time synchronization for SAPD.

Fig. 3. The temporal logic of zero-point time calibration.

The first step is the synchronization scheme, which provides a credible baseline for the next step and whose calibration process resembles that of CTS. The time calibration processes are as follows. First, the sponsor node synchronously sends w pulses to all receiving nodes. Second, each receiving node calculates the time T s when the w + 1 pulse is sent according to the previous received status. Finally,

the PDC parameters of all nodes are initialized at time T s . Thus, we have the following equation:

Pk;0 ¼ 0; F k;0 ¼ F R ; k ¼ 1; 2; . . . ; m: Fig. 3 presents the temporal logic of zero-point time calibration.

1480

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

Algorithm 1. Pseudocode of SAPD Algorithm 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20:

0; F j;0 F R ; /⁄ Parameters Initialization of Pk;0 PDC ⁄/ 0; FBj;i 0; /⁄ FT j;i ⁄ and FBj;i are Set FT j;i initialized ⁄/ for each calibrating step i½1; n do while jP j;i  Pj;i j! ¼ 0 do /⁄ i is synchronizing steps ⁄/ Collect C j;i at the real time t by Eq. (2); Get P j;i for each node by Eq. (4); /⁄ The counting value of PDC for each node ⁄/ Calculate Pn by Eq. (5); /⁄ The average value of Pn for all nodes ⁄/ Figure kj;i by Eq. (6); Adjust F j;i by Eq. (7); /⁄ Calibrate the PDF ⁄/ for each synchronous node j½1; k do Get FT j;i based on kj;i ;/⁄ Compute PDF’s peak value ⁄/ Get FBj;i by kj;i ; /⁄ Compute PDF’s trough value ⁄/ if FT j;i ! ¼ 0 and FBj;i ! ¼ 0 then F 0j;i is calculated by Eq. (8); /⁄ Calculate the new PDF ⁄/ end if end for Update the PDF: F j;i F 0j;i ; end while end for

 PDC synchronization This step is the core of time synchronization, in which a third-party’s node or several synchronized nodes in turn act as a sender for referenced beaconing (Fig. 4). First, the counting values of the local PDC are obtained when all nodes receive the reference beacon. Second, parametric difference data are interchanged between all receiving nodes. Finally, the frequency-calibrated procedure of the local PDC is executed based on the clock data received in the second step.

Algorithm 2. Pseudocode of calibrating PDF

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26:

while P j;i ! ¼ P j;i do for each synchronizing step i½1; n do for each node j½1; k do flag 1; /⁄Initialization, flag is a tag for pole value of F ⁄/ if flag ¼¼ 1 then if kj;i > 1 then flag 1; /⁄ Peak tag ⁄/ else flag 0; /⁄ Trough tag ⁄/ end if end if if flag ¼¼ 1 and kj;i < 1 then F jþ1;i ; /⁄ Find trough value ⁄/ FT j;i flag 0; /⁄ Trough tag ⁄/ end if if flag ¼¼ 0 and kj;i > 1 then F jþ1;i ; /⁄ Find peak value ⁄/ FBj;i flag 1; /⁄ Peak tag ⁄/ end if if FT j;i ! ¼ 0 and FBj;i ! ¼ 0 then F 0j;i is renewedly calculated by Eq. (8); /⁄ Convergence Operation ⁄/ FT j;i 0; FBj;i 0; /⁄ Parameters are reset ⁄/ end if end for end for end while

The detailed process of SAPD is described in Algorithm1, and the accelerated convergence steps for the calibrating frequency of PDC are described in Algorithm2, which is a subset of Algorithm1. The two algorithms are based on the broadcast parametric difference in RBS so their complexity is Oðn2 Þ in single-step cluster synchronization.

Fig. 4. The logic steps of synchronizing PDC.

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

3.2.2. SAPD in multi-hop network In a multi-hop network scheme, to resolve the synchronization problem in large-scale distributed systems, a hierarchical tree structure based on supernodes is introduced to achieve clock synchronization step by step. Generally, the basic synchronization process has the following two facets. First, the sponsor in the top-level network sends the command for PDC synchronization to all supernodes in the top single-hop network. Second, the supernodes at all subnet levels (each subnet has only one supernode) stimulate the orderly synchronization of their corresponding subnodes. After a few times, all nodes in the whole network are synchronized based on the PDC. Conversely, with the extension of network size, the influence of a single node on PDC stability becomes smaller, ensuring the steadiness of time synchronization. 3.3. Characteristics of PDC The new clock synchronization model is called the PDC because of the iterative adjustment made to the frequency through the exchange with the local counting value for each node and because of the working principle’s similarity to the reference broadcast method of RBS. Through the PDC, the synchronization object is transferred to the counting value P; the intention is to maintain the precise global ordinal relation of the logistic clocks of all nodes in any stepping period. The model of the PDC reflects its essence: instantaneous time stamps. That is, the counting value of PDC is built at the very moment of parametric difference timing. The PDC has the characteristics of a logistic clock because of the ‘‘logic’’ execution based on the alteration of PDF and counting values. The calibrated stable PDC can achieve an action similar to that of a physical clock through a precise function conversion relation with a physical clock. In other words, PDC has some of the traits of a physical clock. Thus, PDC can be regarded as a mixed clock. A working PDC does not alter any information in a computer’s local physical/system clock, can avoid problems related to operating system applications or the problem of the clock being at the maximum because of the continual calibration of the physical clock, and enables multiple distributed applications with different time resolutions to perform at a single node in parallel. Based on these characteristics, the PDC can achieve the time synchronization required by new DSAs. 4. Performance evaluation In this section, we analyze the goals and conditions of the simulation and present the performance evaluation. 4.1. SAPD performance analysis Absolute clock synchronization in the real world is impossible to achieve, that is, the collection of clock information for all tested nodes is barely carried out at the same physical time. Thus, we adopt the simulation method to evaluate and analyze the PDC model and the SAPD. Gener-

1481

ally, aiming to assess the time synchronization methods, scholars have mainly focused on the degree of synchronization precision, synchronization spending, and stability. In terms of precision, SAPD accuracy is a unit resolution of the current counting value P referring to the mechanism of the PDC, which is theoretically 1/F according to the PDC model (Eq. (3)). The expression 1/F P 1/f is based on the SAPD theory, where f is the real cycle of a computer’s physical clock. The expression implies microsecond precision because the oscillator frequency of a computer’s mainboard is usually 107 Hz. For example, the actual oscillator frequency of a computer is usually 14.318 MHz so the accuracy of our algorithm is approximately not less than 1/14.318, which is equal to 0.0698 ls. However, the precision of the SAPD in real situations is affected by network performance and the nodes’ running status. Therefore, the precision should be evaluated through experimental measurements. In terms of spending, the core of our algorithm is based on the broadcast parametric difference in RBS. Thus, the SAPD has a period of Oðn2 Þ complexity approximately in single-step clustering synchronization. The particular synchronizing cost will be measured by experiments. Finally, as we propose a completely novel clock synchronization mechanism for DSAs, in which the counter P of the PDC is the basis for the quick and steady synchronization in the whole network, we introduce the simulation method in an ideal status to verify its attributes of convergence and feasibility for clock synchronization on the Internet. Accordingly, each node’s PDF and its counter’s deviation from the average for all nodes are considered. 4.2. Experiment In this section, all simulation experiments performed on MATLAB in Windows XP are described. The computer hardware has a dual-core processor with 2.70 GHz and 2 GB of memory. In the contrast analysis, previous research results are cited, and the synchronization of the five nodes is illustrated. 4.2.1. Convergence and stability of SAPD In our experiment, we assume that the physical clock model of a computer is C ¼ ft, where C is the counting value of the clock, f is the real frequency, and t is the physical time. Nodes are divided into five groups: G1 ; G2 ; G3 ; G4 , and G5 . The number of nodes in each group is 2, 5, 10, 20, and 40, respectively. The step length of the simulation is set to 0.1 s, and the step numbers are 1000. In the initial deployment of the PDC, we set the parametric difference frequencies of all nodes to 1000 Hz and the counting values to 0. Considering the reality of each node’s physical clock and the veracity of simulation evaluation, the selected node frequencies of each group are close to reality. Some detailed parameters and settings of the experiment are shown in Table 2. To select the appropriate version, we evaluate two kinds of SAPD: plain and convergent. To validate the theoretical stability of the proposed SAPD, two sets of simulation experiments are implemented. The absolute consistency of clocks of all nodes in the real world is certainly impossible to achieve. Following

1482

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

Table 2 Configuration of experiment. Groups

f (HZ)

Nodes (N)

Step (s)

FR (HZ)

P k;0

G1 G2 G3 G4 G5

980, 1020 980, 1020, 990, 1005, 1000 980, 1020, 990, 1005, 1000, 995, 1012, 998, 1010, 1002 980, 1020, 990, 1005, 1000, 995, 1012, 998, 1010, 1002, 1010, 996, 979, 1008, 1000, 992, 1014, 1003, 988, 1002 980, 1020, 990, 1005, 1000, 995, 1012, 998, 1010, 1002, 1010, 996, 979, 1008, 1000, 992, 1014, 1003, 988, 1002, 989, 1015, 996, 1005, 1007, 997, 1014, 999, 1019, 1002, 1018, 999, 977, 1002, 1000, 990, 1010, 1013, 984, 1012

2 5 10 20 40

0.1 0.1 0.1 0.1 0.1

1000 1000 1000 1000 1000

0 0 0 0 0

Fig. 5. Performance of original SAPD on stability for 5 nodes.

Fig. 6. Performance of original SAPD on stability for 10 nodes.

a previous analysis of SAPD performance (Section 4.1), we primarily measure the convergent attribute of the SAPD and its counter’s deviation from the medial one. For the first set of simulation studies, we mainly examine the convergent performance of the original SAPD. We select two groups with 5 and 10 nodes, respectively (Figs. 5 and 6, respectively). The convergent trend of the original PDF is not satisfying, which is maintained at a vibrated wave line, even considering the ideal experiment settings. Moreover, based on the comparison of the counting values of all the nodes of the SAPD with the average values, the deviation becomes increasingly larger with the accumulation of time and the augmentation of node numbers (Figs. 5b and 6b). The changes in PDF make its cycle

increase gradually. Apparently, the original SAPD is far from the one we imagine, whose counting value P does not converge to a stable value V at the real time t. That is, the current P cannot be regarded as the reference standard of clock synchronization. However, for the imagined SAPD, the value P of all nodes is identical and changeless at the same physical time t; conversely, when all nodes have the same value P, they occur at the same time t. According to Eq. (3), the counting value of the PDC should not be adjusted directly for the sake of continuity. Moreover, for each node, the frequency curve shows a vibrant trend of its cycle’s gradual increase (Figs. 5 and 6). Thus, the adoption of the average of the parametric difference frequency’s peak value and the lowest one to shrink

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

PDF is more feasible for clock synchronization. Using Eq. (8), we repeat the simulation experiments (Figs. 7–10). For the second experimental set, the modified convergent SAPD is validated by choosing four groups with 5, 10, 20, and 40 nodes. The convergent performance of SAPD in an ideal state, compared with the original one (Figs. 5 and 6), is significantly better by calculating the average of the PDF poles (Figs. 7–10). The PDC for each node also

1483

reaches a stable status in the range of some error when the simulation procedure is in 56 steps (5.6 s) (Fig. 7b). Two tiny deviation intervals labeled TDI, in which each P is close to the value, appear when the simulation steps are approximately 56–65 and 70–90. We refer to them as TDI-I and TDI-II. Ninety steps (9 s) are needed to achieve the higher timing degree of precision for synchronized nodes when the node number is 5 (Fig. 7). The lengths of

Fig. 7. Performance of convergent SAPD on stability for 5 nodes.

Fig. 8. Performance of convergent SAPD on stability for 10 nodes.

Fig. 9. Performance of convergent SAPD on stability for 20 nodes.

1484

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

Fig. 10. Performance of convergent SAPD on stability for 40 nodes.

TDI-I and TDI-II are broadened when the node number is 10 (Fig. 8b). The trend of TDI-I is combined with that of TDI-II when the node number is 20 (Fig. 9b). Finally, when the node number reaches 40 (Fig. 10), the two TDIs are assembled to the only interval (56–90). Compared with the original SAPD, the logical clocks of all nodes in their revised version are calibrated further, obtaining a higher degree of precision after the simulation step goes through 100. The latter simulation results show that as the node number increases, the time spending for the synchronization of all nodes in the whole network extend, and the efficiency of this algorithm decreases. To achieve the same deviation gap ðj P k;n  P n j 0:001Þ, the spending of 10 nodes is approximately 9.54 s, that of 20 nodes is around 10 s, and that of 40 nodes is around 9.3 s (Fig. 11b). These results are summarized in the bar (Fig. 11a). The synchronous accuracy of our convergent SAPD becomes higher with the increase in node numbers in the same error scope (Fig. 11). The deviation has a smaller value when the simulation steps exceed 100 than when the steps are between 56 and 100, indicating that the more the node number increases, the longer the time synchronization is wasted, the higher is the synchronized degree of

precision, and the better is the stability of synchronization (Figs. 8b–10b). However, whether the number of nodes is increasing or decreasing, once all nodes are synchronized, the stability of SAPD becomes excellent, and all synchronized nodes can hardly be out of synchronization for future applications. From the latter experiment, we also observe that the convergent effectiveness of the revised SAPD when N = 5 and N = 40 is better than that of SAPD when N = 10 and N = 20 (Fig. 12a). That is, for small-scale nodes (i.e., N = 10, 20), the synchronization performance is less outstanding than that for large-scale nodes (i.e., N = 40); the rapid synchronization of several nodes, such as N = 5, can easily be understood. The modified SAPD has a predominant stability and convergence when N equals 40 (Fig. 12b). Thus, the more the distributed nodes (N P 40) are added, the better the stability performance of the improved SAPD. Based on the previous simulation analysis, the accelerated convergent SAPD has some of the excellent characteristics of convergence and back-stepping, which are effective steps for quick node synchronization. Thus, in reality, we should choose the convergent version of SAPD.

Fig. 11. Synchronization time in deviation gap 0.001 and deviation degree of P k;n for Steps 87–100.

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

1485

Fig. 12. Deviation degree of P k;n for Steps 100–1000.

Fig. 13. Performance contrast with previous algorithms. (a) Synchronization Precision comparison with different methods. (b) Analysis of the convergence and stability of different algorithms.

Although the initial synchronized timing is longer, once all nodes are synchronized, the stability of SAPD becomes satisfactory, which fits the nodes’ time synchronization in a large-scale network environment and avoids the repetitious jumping and calibration of local clocks caused by traditional synchronization methods. 4.2.2. Comparison with previous algorithms In our experiments, some assumptions and settings are confirmed to assure the rationality of the simulation measurements: (1) Network delay is determinate, and the jumps of local clocks are period variations. (2) We consider the clock jitter accumulation with the normal distribution. Moreover, because of the loss of synchronism in contrastive algorithms, the required amended offset of each correction is the same. (3) A steady state occurs when the precision of the algorithms is in due scope. For example, RBS and TPSN no longer execute their convergence operations when the accuracy is only at a microsecond level. The convergent step based on NTP should also be stopped when its precision has obtained a millisecond error.

The setup of experimental parameters is as follows: (1) The number of synchronizing nodes is 5, and the number of times of the iterative experiments is 20. (2) The total simulation time is 300 s, and the step propulsion at a time is 0.1 s. (3) In terms of the clock offsets, we set the calibrating value ±0.5 ls. Particularly, the NTP value is set to ±0.5 ms, considering its precision rank. The simulation models of the synchronization algorithms are based on their respective theories and mechanisms and have been discussed reasonably. The simulation model of SAPD is uniform to the design, and its error accuracy may be affected by real network environments and the number of nodes. By analyzing the PDC model and SAPD theoretically, we consider the stochastic influences on the error estimate as a Gaussian distribution. In terms of stringency and stability, we choose the verified RBS on Berkeley motes platform, TPSN, and NTP instead of SAPD. In terms of precision, in view of the microsecond error of the other algorithms, we replace NTP with precise NTP + GPS to facilitate comparison in the graphs. Segmental data in contrasting experiments originate from previous research results. To determine the differences intuitively in the convergence and stability

1486

L. Fan et al. / Computer Networks 57 (2013) 1474–1487

of these algorithms, we fit the graphs of the simulation results into one single integrated graph (Fig. 13b). Iterative experiments indicate that the accuracy scope of SAPD is from 0.63 ls to 7.28 ls and that its average value is approximately 3.14 ls. In the experiments repeated 20 times, the average precision of SAPD outperforms RBS and TPSN but is worse than that of NTP + GPS, which integrates a hardware scheme (Fig. 13a). This precision mainly profits from the fact that the PDC counter is taken as a synchronizing symbol as well as the convergence operation of PDF. Other algorithms may have some instability because of the fluctuant oscillator frequency of physical clocks that must be corrected repetitively (Fig. 13b). However, SAPD has the best stability among the time synchronization algorithms, indicating that SAPD does not jump once the clocks of all the nodes are in phase. This finding can be attributed to the fact that the clock model we propose is logical. The convergence of SAPD is similar to that of RBS but worse than that of TPSN and unexpectedly better than that of NTP (Fig. 13b). The reason is that a fast constringency can be achieved by calibrating the PDF, according to the SAPD algorithm. 5. Conclusions and future work In this work, we attempt to incorporate the ideas of parametric difference in RBS, cooperating operation in CTS, and firefly algorithm into the novel SAPD algorithm we propose based on PDC in new DSAs. New models or algorithms should be designed and implemented to improve the performance of time synchronization algorithms, especially stability and precision, which are their key attributes. The new time synchronization problems are first formulated in such simulation environment related to novel distributed applications (e.g., WSNs, distributed grid systems). To solve this problem efficiently, the PDC clock model and its SAPD are proposed in contrast to traditional ones. The performance of SAPD algorithm is validated by simulation experiments, clearly demonstrating that the convergent SAPD has excellent traits of stability and accuracy and that the PDC model is suitable for synchronization problems of clock clusters in novel DSAs. The version of improved SAPD is no longer out of synchronization once all nodes are synchronized, which can obtain a microsecond precision. This work represents our preliminary attempt to introduce a new logical clock model and its synchronization algorithm in contrast to previous ones. Future studies in this area are threefold. First, we shall improve the speed of SAPD such as through interpolation or dichotomy. Second, we plan to study the more detailed synchronizing errors of SAPD. Third, we intend to investigate the definite function relation between our PDC and existing physical clocks. Acknowledgement The authors thank the anonymous reviewers for their insightful comments and suggestions. This research is

partially supported by National Natural Science Foundation of China under Grant 61272336, and National Key Technology R&D Program of China under Grant 2009BAG12A05.

References [1] W. Wang, Y. Xu, M. Khanna, A survey on the communication architectures in smart grid, Computer Networks 55 (15) (2011) 3604–3629. [2] M. Armbrust, A. Fox, R. Griffith, et al., A view of cloud computing, Communications of the ACM 53 (4) (2010) 50–58. [3] I. Foster, I. Raicu, Y. Zhao, S. Lu, Cloud computing and grid computing 360-degree compared, in: Proceedings of the 2008 Grid Computing Environments Workshop, Austin, TX, US, November 12–16 2008. [4] D. Saba, A. Mukherjee, Pervasive computing: a paradigm for the 21st century, IEEE Computer 36 (3) (2003) 25–31. [5] L. Atzori, A. Iera, G. Morabito, The internet of things: a survey, Computer Networks 54 (15) (2010) 2787–2805. [6] S. Strassburger, T. Schulze, R. Fujimoto, Future trends in distributed simulation and distributed virtual environments: results of a peer study, in: Proceedings of the 2008 Winter Simulation Conference, IEEE, 2008, pp. 777–785. [7] W. Wang, W. Wang, Y. Zhu, Q. Li, Service-oriented simulation framework: an overview and unifying methodology, SIMULATION: Transactions of the Society for Modeling and Simulation International 87 (3) (2011) 221–252. [8] L. Lamport, Clock and the ordering of events in a distributed system, Communications of the ACM 21 (7) (1978) 558–565. [9] A. Sinha, P.K. Das, D. Basu, Implementation and timing analysis of clock synchronization on a transputer-based replicated system, Information and Software Technology 40 (5–6) (1998) 291–309. [10] D.L. Mills, Internet time synchronization: the network time protocol, IEEE Transactions on Communications 39 (10) (1991) 1482–1493. [11] J.C. Eidson, M. Fischer, J. White, IEEE-1588 standard for a precision clock synchronization protocol for network measurement and control systems, in: Proceedings of the 34th Annual Precise Time and Time Interval Meeting, IEEE, 2002, pp. 243–254. [12] B. Sterzbach, GPS-based clock synchronization in a mobile, distributed real-time system, Real-Time Systems 12 (1) (1997) 63– 75. [13] J. Elson, L. Girod, D. Estrin, Fine-grained network time synchronization using reference broadcasts, in: Proceedings of the 5th Symposium on Operating System Design and Implementation, ACM/IEEE, 2002, pp. 147–163. [14] M. Maróti, B. Kusy, G. Simon, Á. Lédeczi, The flooding time synchronization protocol, in: Proceedings of the 2nd International Conference on Embedded Networked Sensor Systems, ACM, 2004, pp. 39–49. [15] S. Ganeriwal, R. Kumar, M.B. Srivastava, Timing-sync protocol for sensor networks, in: Proceedings of the 1st International Conference on Embedded Networked Sensor Systems, ACM, 2003, pp. 138–149. [16] E. Farrugia, R. Simon, An efficient and secure protocol for sensor network time synchronization, The Journal of Systems and Software 79 (2) (2006) 147–162. [17] C. Lenzen, P. Sommer, R. Wattenhofer, Optimal clock synchronization in networks, in: Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems, ACM, 2009, pp. 225–238. [18] J.-H. Hoepman, A. Larsson, E.M. Schiller, P. Tsigas, Secure and selfstabilizing clock synchronization in sensor networks, Theoretical Computer Science 412 (40) (2011) 5631–5647. [19] R.M. Pussente, A.C. Barbosa, An algorithm for clock synchronization with the gradient property in sensor networks, Journal of Parallel and Distributed Computing 69 (3) (2009) 261–265. [20] F. Sivrikaya, B. Yener, Time synchronization in sensor networks: a survey, IEEE Network 18 (4) (2004) 45–50. [21] R. Ostrovsky, B. Patt-Shamir, Optimal and efficient clock synchronization under drifting clocks, in: Proceedings of the 18th Annual ACM Symposium on Principles of Distributed Computing, ACM, 1999, pp. 3–12. [22] R. Gotzhein, T. Kuhn, Black burst synchronization (BBS) – a protocol for deterministic tick and time synchronization in wireless networks, Computer Networks 55 (13) (2011) 3015–3031. [23] A. Boukerche, S. Hong, T. Jacob, Synchronization and handoff management schemes for wireless multimedia systems, Computer Networks 41 (3) (2003) 347–362.

L. Fan et al. / Computer Networks 57 (2013) 1474–1487 [24] I.-K. Rhee, J. Lee, J. Kim, E. Serpedin, Y.C. Wu, Clock synchronization in wireless sensor network: an overview, Sensors 9 (3) (2009) 56– 85. [25] D. Veitch, S. Babu, A. Pàsztor, Robust synchronization of software clocks across the internet, in: Proceedings of the 4th ACM SIGCOMM Conference on Internet measurement, ACM, 2004, pp. 219–232. [26] W.-A. Geoffrey, T. Geetika, P. Ankit, W. Matt, N. Radhika, Firenyinspired sensor network synchronicity with realistic radio effects, in: Proceedings of the 3rd ACM Conference on Embedded Networked Sensor Systems, ACM, 2005, pp. 142–153. [27] A. Hu, S.D. Servetto, On the scalability of cooperative time synchronization in pulse-connected networks, IEEE Trans on Information Theory 52 (6) (2006) 2725–2748. [28] Wikipedia, Synchronization, October 2012. . [29] R.M. Fujimoto, Parallel and distributed simulation, in: Proceedings of the 1999 Winter Simulation Conference, ACM, 1999, pp. 122–131. [30] Wikipedia, Criticalsection, October 2012. . [31] I.A. Dimitrovsky, The group lock and its applications, Journal of Parallel and Distributed Computing 11 (4) (1991) 291–302. [32] Wikipedia, Semaphore, December 2012. . [33] K.-L. Noh, E. Serpedin, K. Qaraqe, A new approach for time synchronization in wireless sensor networks: pairwise broadcast synchronization, IEEE Transaction on Wireless Communications 7 (9) (2008) 3318–3322. [34] P.H. Dana, Global positioning system (GPS) time dissemination for real-time applications, Real-Time Systems 12 (1) (1997) 9–40. [35] Q. Ye, Y. Zhang, L. Cheng, A study on the optimal time synchronization accuracy in wireless sensor networks, Computer Networks 48 (4) (2005) 549–566. [36] D.L. Mills, Network Time Protocol, Specification, Implementation and Analysis, RFC 1305, March 1992. . [37] D. Loy, GPS-linked high accuracy NTP time processor for distributed fault-Tolerant real-time systems, Ph.D. Thesis, Vienna University of Technology, April 1996. [38] D.L. Mills, Network Time Protocol (NTP), RFC 958, September 1985. . [39] D.L. Mills, Measured performance of the network time protocol in the Internet system, RFC 1128, October 1989. . [40] J. Wang, M. Zhou, H. Zhou, Clock synchronization for Internet measurements: a clustering algorithm, Computer Networks 45 (6) (2004) 731–741. [41] H. Khlifi, J.-C. Grégoire, Low-complexity offline and online clock skew estimation and removal, Computer Networks 50 (11) (2006) 1872–1884. [42] Y. Hong, A. Scaglione, A scalable synchronization protocol for largescale sensor networks and its applications, IEEE Journal of Selected Areas in Communications 23 (5) (2005) 1085–1099. [43] R. Mirollo, S. Strogatz, Synchronization of pulse-coupled biological oscillators, SIAM Journal on Applied Mathematics 50 (6) (1990) 1645–1662.

Linjun Fan received the M.S. degree in Management Science and Engineering from National University of Defense Technology, Changsha, China, in 2010. He is currently a Ph.D. candidate in operation research from National University of Defense Technology, Changsha, China. His research interests include distributed simulation application, run-time infrastructure based on SOA.

1487 Yunxiang Ling received the PH.D. degree in Computer Architecture from College of Computer in National University of Defense Technology, China, in 1999, He is currently a professor and doctoral supervisor in the College of Information System and Management at National University of Defense Technology, China. His research interests include Combat Simulation, Distributed Interactive Simulation, and Command Decision Analysis.

Tao Wang received his B.S and M.S degrees in management engineering and operation research from National University of Defense Technology, Changsha, China, in 2009 and 2011, respectively. He is currently a Ph.D. candidate in computer science from Faculty of Mathematics and Computer Science in University of Bremen (Germany). His research interests include Human–Computer Interaction, Distributed Simulation and Time Synchronization.

Xiaomin Zhu received the B.S. and M.S. degrees in computer science from Liaoning Technical University, Liaoning, China, in 2001 and 2004, respectively, and Ph.D. degree in computer science from Fudan University, Shanghai, China, in 2009. He is currently an assistant professor in the College of Information System and Management at National University of Defense Technology, Changsha, China. His research interests are cluster computing, fault-tolerant computing, green computing, and performance evaluation. He is a member of the IEEE, the IEEE Communication Society, and the ACM.

Xiaoyong Tang received his master degree from Hunan University, China, in 2007. He is currently working towards the Ph.D. degree at Hunan University of China. His research interests include modeling and scheduling for distributed computing systems, distributed system reliability, distributed system security, and parallel algorithms.