Efficient Deadlock-Free Wormhole Routing and Virtual-Channel Reduction in Shuffle-Based Networks

JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING ARTICLE NO.

46, 165–179 (1997)

PC971380

Efficient Deadlock-Free Wormhole Routing and Virtual-Channel Reduction in Shuffle-Based Networks 1 Hyunmin Park*,2 and Dharma P. Agrawal†,3 *Department of Computer Engineering, Myongji University, Yongin, South Korea; and †Electrical and Computer Engineering, Box 7911, North Carolina State University, Raleigh, North Carolina 27695-7911

Many aspects of shuffle-based networks have recently been studied by numerous researchers. However, no attention has been paid to deadlock-free wormhole routing algorithms. In this paper, for a set of shuffle-based networks, we introduce a graph-partitioning technique that enables a deadlock-free routing algorithm with fewer virtual channels than the known algorithms. This is achieved for the de Bruijn digraphs which are shown to require a maximum of m − b(m − 1)/rc virtual channels per physical channel, where m is the diameter and r is the radix. Algorithms for the generalized de Bruijn graph, the de Bruijn Cube (dBCube) graph and the Shuffle–Exchange network are introduced, and virtual channel requirements are determined. The dBCube graph of size (r, N b, n) requires a maximum of m − b(m − 1)/rc virtual channels for the outcluster channels, and a maximum of m + 1 − bm/rc virtual channels for the incluster channels in most cases, where m = blog r N bc, r is the radix of a generalized de Bruijn graph of size N b, and n represents the number of dimensions in a binary hypercube. We also show that a maximum of m − b(m − 1)/2c virtual channels are required in shuffle-exchange networks with 2 m nodes. © 1997 Academic Press

1. INTRODUCTION

As an interconnection network does affect the computational efficiency and the communication performance, different networks have been studied extensively by several researchers. Hypercube is the most popular direct interconnected network [1] investigated due to its regularity, low diameter and average distance, logarithmic connectivity, embedding of numerous topologies and simple mapping of many algorithms. But, for large systems, its connectivity becomes prohibitively large. The higher connectivity forced Intel Corporation to replace hypercube with a mesh topology in their new Touchstone and 1This work has been supported in part by the National Science Foundation under Grant MIP-9403191. Part of an earlier version of this paper was presented at the 9th IEEE International Parallel Processing Symposium [18] and at the 7th IEEE Symposium on Parallel and Distributed Processing [17]. 2 E-mail: [email protected]. 3 To whom correspondence should be addressed. E-mail: [email protected]. edu.

Paragon supercomputers [12]. On the other hand, shuffle-based networks have become increasingly important for large networks. A de Bruijn digraph is one such widely referenced shuffle-type network, as it can be defined for any number of nodes (even for prime numbers) [8–10, 20], and in most cases, it has the smallest diameter among all known networks with a prespecified node degree. Due to this property, JPL constructed a 8096 node de Bruijn network for NASA’s Galileo project [4]. Very recently, Sivarajan and Ramaswami [21] and Labourdette [11] have proposed and studied a de Bruijn digraph as a logical topology for optical networks, as a large number of nodes can be supported for a given degree. Another approach to overcome the connectivity problem, while maintaining the basic characteristics of a hypercube, is to use a class of hierarchical networks. A recently introduced new network called dBCube (de Bruijn-Cube) network [3] is a point-to-point twolevel hierarchical network, which consists of a number of basic hypercube clusters and interconnected by a de Bruijn graph. In direct networks, wormhole routing [7, 15] has become quite popular for multiprocessors. In wormhole routing, if the header flit of a packet is blocked by a busy channel, the remaining flit which follows in pipeline fashion, stays in the network along the established route, instead of being stored at the blocked node, which makes one packet usually block more than one channel. Wormhole routing provides a good network performance as long as the network load is light. When the network is heavily loaded, it has a poor communication latency compared to virtual cut-through routing, due to each message blocking several channels simultaneously [14]. Even when the network load is moderate, if the path length of a message is long, the performance may be poor since the blocking time, which is a function of the path length, increases [15, 16]. One of the critical problems of wormhole routing in a direct point-to-point network is how to accomplish a deadlockfree routing. Deadlock occurs when several messages traveling in the network develop dependency loops among themselves that prevents any further movement of data. In hypercube networks, simple e-cube routing (simple dimension ordering) [7] may be used for deadlock-free routing. In a cube of dimension n, each node has n output channels, one for each dimension. Since the e-cube routing algorithm routes a 165 0743-7315/97 $25.00 Copyright © 1997 by Academic Press All rights of reproduction in any form reserved.

166

PARK AND AGRAWAL

message in increasing order of dimension and hence increasing channel, there are no cycles in the channel dependency graph. However, in general, other networks need some other methods like buffer management or virtual channels [5–7, 13] to deal with the deadlock. Dally [7] introduced deadlock-free routing algorithms for arbitrary k-ary n-cubes, cube-connected cycles, and shuffle–exchange networks using the concept of virtual channels. This virtual channel approach can be applied to any switching method including store-and-forward, virtual cutthrough as well as wormhole routing. In an earlier paper [16], we introduced simple deadlock-free routing algorithms in a generalized de Bruijn (GdB) (di)graph, GdB(r, Nb ), with m virtual channels per physical channel, and in a de Bruijn Cube (dBCube) network, dBC(r, Nb , n), with m virtual channels per outcluster channel and m + 1 virtual channels per incluster channel, when m = dlogr Nb e. In this paper, a new deadlockfree routing algorithm is presented for de Bruijn (dB) digraph networks. We call it a β algorithm. The new β algorithm requires fewer virtual channels, m − b(m − 1)/rc, per channel for a large network. We apply this β algorithm to the other dB digraph based networks including GdB graph and de Bruijn Cube networks. It may be noted that a recent technical report [2] has independently presented a similar deadlock-free routing algorithm in r-ary de Bruijn graphs of size r m by employing an edge coloring scheme. Another class of shuffle network we have considered is a shuffle–exchange network. Dally [7] recently introduced an algorithm for a 2m -node shuffle–exchange network that requires m virtual channels per channel, while our proposed algorithm needs only maximum m − b(m − 1)/2c. Recently, Pifarré [19] introduced a deadlock-free routing algorithm using only 4 virtual channels per channel regardless of the network size in the shuffle–exchange networks. However, in his algorithm, the maximum number of hops increases from 2m to 3m. We extend our routing technique that requires slightly more virtual channels while keeping the path length of up to 2m. This paper is organized as follows: Section 1 introduces the paper. In Section 2, the general definitions and notations are presented. Section 3 covers a deadlock-free routing algorithm for a GdB digraph, and in Section 4, we extend it for a GdB graph. Section 5 shows a deadlock-free routing algorithm for a dBCube. In Section 6, an algorithm for a shuffle–exchange network is developed. Finally, Section 7 concludes the paper. 2. DEFINITIONS AND NOTATIONS

The following definitions cover the notations for describing different networks, and their routing schemes. DEFINITION 1. A graph G is represented as G = {V (G), E(G)}consisting of a nonempty set V (G) of vertices, a set E(G) of edges, and an incidence function that associates each edge of G with an unordered pair of (not necessarily distinct) vertices of G. If e ∈ E(G) and X, Y ∈ V (G) such

that e = XY , then e is said to join X and Y ; the vertices X and Y are called the ends of e. DEFINITION 2. A directed graph D is represented as D = {V (D), E(D)} consisting of a nonempty set V (D) of vertices, a set E(D) of edges, and an incidence function that associates each edge of D with an ordered pair of (not necessarily distinct) vertices of D. If e ∈ E(D) and X, Y ∈ V (D) such that e(X, Y ), then e is said to join X and Y ; X is the tail of e, and Y is its head. For convenience, we abbreviate “directed graph” as digraph. An edge with distinct ends is called a link, and an edge with identical ends is termed a loop. The edges of the graph G represent a set of communication channels in a computer network. With each digraph D, we can associate a graph G on the same vertex set; corresponding to each edge of D there is an edge of G with the same ends. This graph is the underlying graph of D. Conversely, given any graph G, we can obtain a digraph D by specifying, for each edge, an order for its ends. Such a digraph is called an orientation of G. A digraph H is a subdigraph of D (written H ⊆ D) if V (H ) ⊆ V (D), E(H ) ⊆ E(D), and the incidence function of H is the restriction on the incidence function of D to E(H ). A directed path in D is a finite nonnull sequence P = (X 0 , X 1 , . . . , X m ) such that X i ∈ V (D) for i = 0, 1, . . . , m and ei (X i−1 , X i ) for i = 1, 2, . . . , m. To develop a deadlock-free wormhole routing algorithm, it is assumed that a message arriving at its destination node is eventually consumed. A node can generate messages destined for any other node. The following notations are assumed throughout the paper: e, ei , a channel or its address; X, X i , Y, Yi , a node or its address; H (β), H (γ ), βth and γ th subdigraph, respectively; β pr ev , βnext , the β values of the previous incoming channel, and of the next outgoing channel, respectively; S, S0 , a source node or its address; D, D0 , a destination node or its address. We do not consider the loop channels in the path sequence and the routing path, since it can be easily excluded in an actual routing. However, when the characteristics of the (di)graph are considered, we include loop channels for simplicity of explanation. For example, in general, the degree of a node is defined as the number of edges (which are not loops) incident to the node. Since it does not affect the general characteristics of the (di)graph, we include a loop channel when we count the “degree.” 3. DEADLOCK-FREE ROUTING IN GENERALIZED DE BRUIJN DIGRAPH NETWORKS

Many research results on unidirectional dB digraph (dBD) have been obtained by Imase and Itoh [10], and Pradhan and Samatham [20]. Most of the time, the dBD has the smallest diameter with respect to the number of nodes in the graph. The dBD can be defined as follows. An r-radix unidirectional dBD(r, r m ), has the total number of nodes, N = r m and the address of a node X is represented as

ROUTING AND CHANNEL REDUCTION IN NETWORKS

167

LEMMA 1. In GdBD(r, Nb ), assume that X has a neighboring node Y . Then Y = (rX + α) mod Nb and X = b(Y + β Nb )/rc, where α, β ∈ {0, 1, . . . , (r − 1)}, β = b(rX + α)/Nb c and α = (Y + β Nb ) mod r.

FIG. 1. Examples of de Bruijn and Generalized de Bruijn digraph networks.

(xm−1 , xm−2 , . . . , x0 ) with xi ∈ {0, 1, . . . , (r − 1)} for 0 ≤ i ≤ m − 1. Its neighboring nodes are (x m−2 , x m−3 , . . . , α), where α = 0, 1, . . . , r − 1. Figure 1a shows an example of a dB digraph, dBD(2, 8). Esfahanian and Hakimi [9] researched possible routing paths in dBD. A unidirectional GdB digraph (GdBD) has been proposed [10] to extend the dBD for any general value of number of nodes rather than just a power of r. It has been proved that the GdBD has a short diameter very much like dBD. The GdBD can be defined as follows. In GdB digraph, GdBD(r, Nb ), the number of nodes is Nb , while the address of a node, X, is still represented as (x m−1 , x m−2 , . . . , x 0 ) with xi ∈ {0, 1, . . . , (r − 1)} for 0 ≤ i ≤ m − 1 and m = dlogr Nb e. The neighboring nodes Y ’s of node X are defined as the nodes such that Y = (rX + α) mod Nb , where α = 0, 1, . . . , (r − 1). An example of GdBD is shown in Fig. 1b. In GdBD(r, Nb ), we define theH (β) subdigraphs as follows: DEFINITION 3. For each β = 0, 1, . . . , r − 1, H (β) = {e(X, Y )|Xr + α = β Nb + Y }, provided e ∈ E(GdBD), X, Y ∈ V (GdBD) and α ∈ {0, 1, . . . , r − 1}. Figure 2 shows some examples of H (β) subdigraphs. The following property is obvious from the definition of H (β). Property 1. E(H (β)) ∩ E(H (γ )) = ∅ for β 6= γ and −1 E(H (β)) = E(GdBD). β, γ = 0, 1, . . . , r − 1 and ∪rβ=0 Using the definition of GdBD and Definition 3, we present several characteristics of the H (β) in the following lemmas.

Proof. It is obvious that Y = (rX + α) mod Nb and β = b(rX + α)/Nb c from the definition of GdBD(r, Nb ) and the modulus operation. Then Y = (rX + α) mod Nb = (rX + α) − β Nb . So Y + β Nb = rX + α, and b(Y + β Nb )/rc = bX + α/rc = X , since α/r < 1. Thus, X = b(Y + β Nb )/rc. Here, β is an integer such that 0 ≤ β ≤ r − 1, because 0 ≤ rX + α ≤ r(Nb − 1) + (r − 1) = (r − 1)Nb + (Nb − 1). And from Y + β Nb = rX + α, (Y + β Nb ) mod r = (rX + α) mod r = α, since 0 ≤ α ≤ r − 1 from the definition. Thus, α = (Y + β Nb ) mod r. LEMMA 2. No node can have more than one incoming channel belonging to the same H (β). Every node has one incoming channel belonging to a H (β) for each β = 0, 1, . . . , r − 1. Proof. Assume that a node, X 0 , has more than one incoming channel belonging to the same H (β). Then, because a node cannot have the same node as a neighbor twice, there must exist such nodes, X 1 and X 2 (X 1 6= X 2 ) that satisfy the following: X 1 r + α1 = β Nb + X 0 and X 2 r + α2 = β Nb + X 0 . From Lemma 1, X 1 = b(β Nb + X 0 )/rc and X 2 = b(β Nb + X 0 )/r c. Hence X 1 = X 2 and it contradicts the assumption. Therefore, a node can have only one incoming channel belonging to the same H (β). From the above result and the fact that every node in GdBD(r, Nb ) has the same degree r, for each β = 0, 1, . . . , r − 1, every node should have one incoming channel belonging to a H (β); otherwise, the above result or the fact has to be violated. All outgoing channels of a node could belong to the same H (β), or some of the outgoing channels could belong to H (β) while the remaining outgoing channels are members of H (β + 1), as α < Nb in the relation of Xr + α = β Nb + Y . The following lemma shows the condition of these cases. LEMMA 3. When Nb = kr, all outgoing channels of a node belong to the same H (β), where k = 1, 2, . . .. When Nb 6= kr (i.e., Nb = kr + j with j = 1, 2, . . . , r − 1, if a node address is b((β + 1)Nb )/r c for β = 0, 1, . . . , r − 2 such that (β + 1) j is not a multiple of r, each outgoing channel belongs to either H (β) or H (β + 1), and the outgoing channels belonging to the H (β + 1) are connected to the nodes with addresses from 0 up to (r − 2); otherwise all outgoing channels belong to the same H (β).

FIG. 2. Examples of H (β) subdigraphs in de Bruijn and Generalized de Bruijn digraph networks.

Proof. If a node X 0 has the outgoing channels satisfying the condition that X 0 r + α = β Nb + X 1 and X 0 r + (α + 1) = (β +1)Nb + X 2 , each outgoing channel of X 0 belongs to either H (β) or H (β + 1); otherwise, all outgoing channels of the node belong to the same subdigraph. Then X 1 − X 2 = Nb − 1, and because 0 ≤ X 1 , X 2 ≤ Nb − 1, X 1 = Nb − 1 and

168

PARK AND AGRAWAL

X 2 = 0. So it is obvious that the outgoing channels belonging to the H (β + 1) are connected to the nodes from 0 up to (r − 2). From Lemma 1 and the above assumption, such nodes must satisfy the condition that X 0 = b((β + 1)Nb )/rc = b((β + 1)Nb − 1)/rc. If Nb = kr, where k is a positive integer, b((β + 1)kr)/rc 6= b((β + 1)kr − 1)/rc because b((β + 1)kr)/rc = (β + 1)k and b((β + 1)kr − 1)/rc = (β + 1)k − 1. So, there exists no such node X 0 . Thus, if Nb = kr, all outgoing channels of a node belong to the same subdigraph. If Nb = kr + j , where j = 1, 2, . . . , r − 1, from the condition b((β + 1)Nb )/rc = b((β + 1)(kr + j))/rc = (β + 1)k + b((β + 1) j)/rc and b((β + 1)Nb − 1)/rc = b((β + 1)(kr + j ) − 1)/rc = (β + 1)k + b((β + 1) j − 1)/rc. If (β + 1) j = pr, where p is an positive integer, b((β + 1) j)/rc 6= b((β +1) j −1)/rc because b((β +1) j )/rc = p and b((β + 1) j − 1)/rc = p − 1. So, when Nb 6= kr, all outgoing channels of such a node satisfying that (β + 1) j is a multiple of r belong to the same subdigraph. If (β + 1) j = pr + q, where p = 1, 2, . . . , r − 1, b((β + 1) j )/rc = p, and b((β + 1) j − 1)/rc = p, so the condition is always satisfied. Thus, if Nb = kr + j , each outgoing channel of the nodes with addresses b((β + 1)Nb )/r c such that (β + 1) j is not a multiple of r, belongs to either H (β) or H (β + 1). LEMMA 4. Any path of length ≤ m −2, starting from nodes X 0 ’s, where 0 ≤ X 0 ≤ r − 1, routes along only the channels of the H (0), where m = dlogr Nb e. Proof. Let’s assume that a path of length m − 2 is P = (X 0 , X 1 , . . . , X m−2 ) In order to prove this lemma, it is sufficient to show that at the i th hop, the maximum value of each X i−1 r + α is ≤ r m−1 for all i = 1, 2, . . . , m − 2, because r m−1 < Nb ≤ r m . Among the starting node addresses, max(X 0 ) = r − 1. In the first hop, max(X 0 r + α) = (r −1)r +(r −1) = r 2 −1 < r m−1 and the second hop leads to max(X 1 r +α) = (r −1)r 2 +(r −1)r +(r −1) = r 3 −1 < r m−1 , and so on. At the final (m − 2)th hop, max(X m−3 r + α) = (r − 1)r m−2 + (r − 1)r m−3 + · · · + (r − 1)r + (r − 1) = (r − 1)((r m−1 − 1)/r − 1) = r m−1 − 1 < r m−1 . Thus, every path of length ≤ m − 2 employs only the channels of the H (0). LEMMA 5. Every H (β) of GdBD(r, Nb ) has no cycle with length m > 1. Proof. Assume that a H (β) has a cycle of (X 0 , X 1 , . . . , X m−1 , X 0 ) with length m > 1. Without loss of generality, we can assume that X 0 is the node with the smallest node address. Here we know that X 1 = (X 0 r + α0 ) mod Nb and X 0 + α0 = β Nb + X 1 . Because 0 ≤ α0 ≤ (r − 1), X 0 is the smallest (X 0 < X 1 ), and a self-looping channel is not considered, X 0 r + α0 < (X 0 + 1)r ≤ X 1 r + α1 And X 1 < X 2 as X 1 r + α1 = β Nb + X 2 and X 0 r + α0 = β Nb + X 1 < X 1 r + α1 = β Nb + X 2 . By a similar reasoning, X 0 < X 1 < X 2 < · · · < X m−1 < X 0 . Hence, a contradiction occurs.

In dBD(r, r m ), simple shift and exchange operations may be used for routing, and are defined as follows. A message can travel from any Source node S = (sm−1 , sm−2 , . . . , s0 ) to any Destination node D = (dm−1 , dm−2 , . . . , d0 ), in m hops, using the following path: Left-Shift Path in dBD(r, r m ). (sm−1 , sm−2 , . . . , s0 ) ⇒ (sm−2 , sm−3 , . . . , dm−1 ) ⇒ (sm−3 , . . . , dn−1 , dm−2 ) ⇒ · · · ⇒ (dm−1 , dm−2 , . . . , d0 . Du and Hwang [8] showed that GdBD(r, Nb ) digraph has the self-routing property and that there exists a length m = dlogr Nb e path for any pair of nodes. This Left-Shift Path is defined as follows: Left-Shift Path in GdBD(r, Nb ). For a given source node S and a destination node D, let (u m−1 , u m−2 , . . . , u 0 ) be an rary number representative of the residue (D − S · r m ) mod Nb . Then, there exists a path of length m from S to D: S = L 0 → L 1 → · · · → L m = D such that L j+1 = (r L j + u (m−1)− j ) mod Nb . We present a deadlock-free routing algorithm, which is called as the β algorithm for GdBD(r, Nb ). We define a virtual channel address as (v, β, τ ), where v is a virtual channel number, and β represents a subdigraph H (β) which the channel belongs to. Since from Lemma 5 every H (β) is acyclic, a total order can be assigned to its channels. τ is a unique channel number in each H (β) so that within H (β), all the outgoing channels of a node are greater than the incoming channel to that node. If the next outgoing channel has a lower value β than the previous incoming channel, v is increased by 1; otherwise, the same v of the incoming channel is used. A formal routing algorithm is shown in the following. ROUTING ALGORITHM 1. (β Algorithm). /* Routing Algorithm at node X = L j */ /* Assume (u m−1 , . . . , u (m−1)−( j−1) ) are already used. */ begin if (X = D) End Algorithm; Calculate the next node Y until Y 6= X , and βnext ; if (X = S) v := 0; else { Calculate β prev ; if (β prev > βnext ) v := v + 1; } Send a message to node Y ; end; We prove that the above algorithm is deadlock-free. THEOREM 1. The β Routing Algorithm 1 for GdBD(r, Nb ) is deadlock-free. Proof. Within the same H (β), τ of the channel address is in an increasing order, and if a path takes a channel in a higher numbered H (β) than the previous channel, they are also in an increasing order. Only if a channel in a lower numbered H (β) than the previous channel is used, are they in a decreasing order. However, the increase of v by 1 makes

ROUTING AND CHANNEL REDUCTION IN NETWORKS

169

them in an increasing order. Since all paths take the channels in an increasing order, the algorithm is deadlock-free.

LEMMA 7. If the maximum message path length is m, there exist at most (m − 1) − b(m − 1)/rc decreasing transitions.

Next, we illustrate how many virtual channels are required when the β Routing Algorithm 1 is used. If we know how many maximum times a path contains such transitions to a lower numbered H (β), then that primarily represents the number of virtual channels required to ensure a deadlock-free routing. Consider a path sequence P = (b1 , b2 , . . . , bm ), whose terms represent the value β of a H (β). There exist three possible transitions between bi and bi +1 for each i = 1, 2, . . . , m − 1, which are increasing, equal, and decreasing transitions. If bi < bi +1 , it is defined as an increasing transition, if bi > bi+1 , it is a decreasing transition, and when bi = bi +1 it will be an equal transition. We find the maximum number of decreasing transitions required along the path P. To generate maximum decreasing transitions, increasing transitions are needed when m > r, but not equal transitions. Actually, an equal transition only reduces the possibility of decreasing transitions. Hence, we consider only increasing and decreasing transitions in the following lemmas.

Proof. The path of length m has m − 1 transitions, and at least b(m − 1)/rc increasing transitions from Lemma 6, so at most (m − 1) − b(m − 1)/rc decreasing transitions can occur.

LEMMA 6. If the maximum message path length is m, there exist at least bm − 1/rc increasing transitions. Proof. Assume that the maximum message path length, m, is pr + 1 ≤ m ≤ ( p + 1)r, where p is an integer such that p ≥ 0. We prove that there exist at least p increasing transitions and p = b(m − 1)/rc. If p = 0, it is obvious because 1 ≤ m ≤ r and so there exists a path that contains only decreasing transitions without having any increasing transitions. Such an example is m, m − 1, . . . , 0. We prove the case of p ≥ 1 by a mathematical induction. If p = 1, the path must have at least one or more increasing transitions as r + 1 ≤ m ≤ 2r and bi is an integer such that 0 ≤ bi ≤ r − 1, and hence there is no path with only decreasing transitions. To prove that the path has at least one, not at least more than one, increasing transition, it is sufficient to find a path with only one increasing transition. Such a path is r − 1, r − 2, . . . , 0, r − 1, r − 2, . . . , r − (m − r). Assume that the case p = i , that is, if ir + 1 ≤ m ≤ (i + 1)r, there exist at least i increasing transitions in (b0 , b1 , . . . , bm ), is true. We show that the case p = i + 1 is also satisfied. From the assumption, it is obvious that (b0 , b1 , . . . , bir +1 ) should have at least i increasing transitions. Also because (i + 1)r + 1 ≤ m ≤ (i + 2)r when p = i + 1, the path length of (bir +1 , bir +2 , . . . , bm ) is ≥ r + 1 and ≤ 2r. For the same reason, (bir+1 , bir+2 , . . . , bm ) should have at least one increasing transition. Hence, when p = i + 1, there exist at least (i + 1) increasing transitions in a path. Next, we prove that p = b(m − 1)/rc. From pr + 1 ≤ m ≤ ( p + 1)r, p ≤ (m − 1)/r ≤ ( p + 1) − (1/r) and so b pc ≤ b(m − 1)/rc ≤ b( p + 1) − (1/r)c. As p is an integer such that p ≥ 0, p ≤ b(m − 1)/rc ≤ p. Hence p = b(m − 1)/rc.

THEOREM 2. When the maximum message path length is m = dlogr Nb e, the β Routing Algorithm 1 requires maximum m − b(m − 1)/rc virtual channels per channel. Proof. From Lemma 7, there exist at most (m −1)−b(m − 1)/rc decreasing transitions. Each decreasing transition needs one additional virtual channel. As the starting channel requires only one virtual channel, the algorithm needs a maximum of m − b(m − 1)/rc virtual channels.

4. DEADLOCK-FREE ROUTING IN GENERALIZED DE BRUIJN GRAPH NETWORKS

A GdB Graph, GdBG(r, Nb ) is the underlying graph of GdBD(r, Nb ). Figure 3 shows some examples of GdBG(r, Nb ) The total number of nodes is Nb and the address of a node X is also represented as (x m−1 , xm−2 , . . . , x0 ) with xi ∈ {0, 1, . . . , (r − 1)} for 0 ≤ i ≤ m − 1 and m = dlogr Nb e. Each node also has Left-Shift and Right-Shift neighbors. The neighboring node Y of node X are defined as follows: Left-Shift Neighbors are nodes Y ’s such that Y = (rX + α) mod Nb . Right-Shift Neighbors are nodes Y ’s such that X = (rY + α) mod Nb . Here, α = 0, 1, . . . , (r − 1). The Right-Shift Neighbors of X can be also represented as Y = b(X + β Nb )/rc according to Lemma 1. We construct two orientations of GdBG(r, Nb ), Left-Shift and Right-Shift. Left-Shift orientation, GdBG L , is a digraph consisting of Left-Shift neighbors and channels and RightShift orientation, GdBG R , has Right-Shift neighbors and channels. Some examples are shown in Fig. 4. The GdBG L is the same as a GdBD(r, Nb ) and the GdBG R is an edgereversed GdBD(r, Nb ). The H L (β) subdigraphs of the GdBG L are defined in the same way as the H (β)’s of the GdBD.

FIG. 3. Examples of de Bruijn and Generalized de Bruijn graph networks.

170

PARK AND AGRAWAL

Property 2. E(GdBG L ) ∩ E(GdBG R ) = ∅, and E(GdBG L ) ∪ E(GdBG R ) = E(GdBG L ) ∪ E(GdBG R ) = E(GdBG). E(H L (β)) ∩ E(H L (γ )) = E(H L (β)) ∩ E(H L (γ )) = ∅ for β 6= γ and β, γ = β, γ = 0, 1, . . . , r − 1, and −1 E(H L (β)) = E(GdBG L ). E(H R (β)) ∩ E(H R (γ )) = ∪rβ=0 E(H R (β)) ∩ E(H R (γ )) = ∅ for β 6= γ and β, γ = −1 E(H R (β)) = E(GdBG R ) β, γ = 0, 1, . . . , r − 1, and ∪rβ=0 r −1 R = ∪β=0 E(H (β)) = E(GdBG R ). LEMMA 8. In GdBG R , no node can have more than one outgoing channel belonging to the same H R (β). Every node has one outgoing channel belonging to a H R (β) for each β = 0, 1, . . . , r − 1. FIG. 4. Examples of Left-Shift and Right-Shift Orientations in de Bruijn and Generalized de Bruijn graph networks.

We can define a H R (β) of the GdBG R as a subdigraph of channels and nodes satisfying the condition that for each β = 0, 1, . . . , r − 1, H R (β) = {e(X, Y )|Y = b(X + β Nb )/rc}. Figure 5 shows some examples of H L (β) and H R (β) subdigraphs. Since from the definitions of H (β) and H R (β) a H R (β) is an edge-reversed H (β), the following property and lemmas can be easily proved.

LEMMA 9. In GdBG R , when Nb = kr, all incoming channels of a node belong to the same H R (β), where k = 1, 2, . . . . When Nb 6= kr (i.e., Nb = kr + j with j = 1, 2, . . . , r − 1), if a node address is b((β + 1)Nb − 1)/rc for β = 0, 1, . . . , r −2 such that (β +1) j is not a multiple of r, each incoming channel belongs to either H R (β) or H R (β + 1) otherwise, all incoming channels belong to the same subdigraph. LEMMA 10. In GdBG R , any path from a H R (β) (β > 0) to nodes X 0 ’s, where 0 ≤ X 0 ≤ r − 1, should use a H R (0) channel path of minimum length dlogr Nb e − 2. LEMMA 11.

FIG. 5. Examples of H L (β) and H R (β) subdigraphs in de Bruijn and Generalized de Bruijn graph networks.

In GdBG R , each H R (β) has no cycle.

In GdBG(r, Nb ) bidirectional graph, we have another path of length m, which may be called Right-Shift Path from S to D. It is defined as follows: Right-Shift Path. For this path we have an r-ary number representation, W = (wm−1 , wm−2 , . . . , w0 ), as the residue (S − D · r m ) mod Nb and the path can be constructed as S = R0 → R1 → · · · → Rm = D such that R j = (r R j+1 + w j ) mod Nb . This Right-Shift Path is actually the reverse of the Left-Shift Path from D to S. Lemma 1 and path representation W lead to Z = (z m−1 , z m−2 , . . . , z 0 ) such that z j = b(r R j +1 +w j )/Nb c and R j+1 = b(R j + z j Nb )/rc, where 0 ≤ j ≤ m − 1. We present a deadlock-free routing scheme for GdBG(r, Nb ). We define a virtual channel address as (LR, v, β, τ ), where LR = 0, 1 represents a GdBG L or a GdBG R channel, respectively, and v is a virtual channel number, and β represents a subdigraph, H L (β) or H R (β), to which the channel belongs. From Lemmas 5 and 11, every H L (β) and H R (β) are acyclic. So, an order can be assigned to the channels of each H L (β) (H R (β)). τ is a unique channel number in each H L (β) (H R (β)), so that within H L (β) (H R (β)) all the H L (β) (H R (β)) outgoing channels of a node are greater than the H L (β) (H R (β)) incoming channel to that node. If the next outgoing channel has the lower number β than the previous incoming channel, v is increased by 1; otherwise, the same v of the incoming channel is to be used. The formal routing algorithm is shown in the following.

ROUTING AND CHANNEL REDUCTION IN NETWORKS

ROUTING ALGORITHM 2. /* Routing Algorithm at node X = L i or Ri */ /* Assume (u m−1 , . . . , u (m−1)−( j−1) ) or (z 0 , . . . , z j−1 ) are already used. */ begin if (X = D) End Algorithm; if (X = S) { Decide a direction, Left-Shift or Right-Shift; Calculate U or Z according to the direction; } else direction := direction of the previous channel; Calculate the next node Y until Y 6= X, and βnext ; if (X = S) v := 0; else { Calculate β pr ev ; if (β pr ev > βnext ) v := v + 1; } } Send a message to node Y ; end; The following theorems show that the above algorithm is deadlock-free and lead to the number of virtual channels per physical channel. THEOREM 3. The Routing Algorithm 2 on a GdBG(r, Nb ) graph is deadlock-free. Proof. From the property of GdBG L and GdBG R , E(GdBG L ) ∩ E(GdBG R ) = ∅. So we can consider GdBG L and GdBG R separately. From Theorem 1, the messages on GdBG L do not create a deadlock situation. Let’s consider a Right-Shift message path of GdBG R . Within the same H R (β), τ of the channel address is in an increasing order. When a path takes a channel in the higher numbered H R (β) than the previous channel, they are also in increasing order. Only if a channel in the lower numbered H R (β) is used, are they in a decreasing order. However, the algorithm increases v by 1, so that it is in an increasing order. Since all paths take the channels in an increasing order, they are deadlock-free. THEOREM 4. With the maximum message path length m = dlogr Nb e, the Routing Algorithm 2 requires maximum m − b(m − 1)/rc virtual channels per channel. Proof. Because E(GdBG L )∩ E(GdBG R ) = ∅, we consider GdBG L and GdBG R separately. From Theorem 2, it is obvious for the GdBG L . In the GdBG R , if the maximum message path length is m, there may also exist at most (m −1)−b(m −1)/rc decreasing transitions from Lemma 7, and each requires an additional virtual channel. Since the starting channel requires one additional virtual channel, a maximum of m −b(m −1)/rc virtual channels are required. 5. DE BRUIJN CUBE GRAPHS

The de Bruijn Cube (dBCube) bidirectional interconnection network [3] proposed by Chen et al. is a direct point-to-point topology with the same degree for every node. As the nodes in dBCube are identical, the network can be easily extended to a larger size. The dBCube has been defined as a number of basic

171

clusters of hypercube topology, with each cluster representing a node of a dB graph. The hypercube topology for local interconnection allows easy embedding of parallel algorithms and the dB graph for cluster-to-cluster connection provides shortest distance among different clusters. The specific channel connections between nodes in various clusters were not clearly identified by the authors. Here, we present a methodology of how a node in one cluster is connected to a node in another cluster in a dBCube network. As dBCube network is constructed by replacing each node in the dB graph with a cluster of a cube, the local channels used to connect the nodes in the same cluster are called incluster channels, and the remote channels to connect different clusters are called outcluster channels. Each node has several incluster channels to communicate with the other nodes in the same cluster and one outcluster channel to communicate with another cluster. A dBCube Graph is characterized by dBC(r, Nb , n), where Nb is the number of cubes or clusters with radix r, where r is a power of 2 such that r ≤ 2n−1 , in the network and n is the dimension of a cube. The total number of nodes in dBCube is N = Nb × 2n . The address of a node is represented as X = (X C , X L ), where X C is the address of L L , xL , ..., xL a cluster and X L = (x n−1 n−2 n−log2 r −1 , . . . , x 0 ) L is the local address within a cluster with x j ∈ {0, 1} for 0 ≤ j ≤ n − 1. Each node in dBCube has two kinds of neighboring nodes: One is an incluster neighbor and the other is an outcluster neighbor. The incluster neighbors of a node are in the same cluster and connected by a cube topology with incluster channels. The outcluster neighbor connects a node to a node in a different cluster by a dB graph. InCluster Neighbors. The address of incluster neighbor Y to a node X is given by Y = (Y C , Y L ) L , . . . , xiL , . . . , x 0L )) for 0 ≤ i ≤ n − 1. = (X C , (x n−1

OutCluster Neighbor. Based on Lemma 1, the address of outcluster neighbor Y to a node X is defined as follows: L = 0, the node has Left-Shift Outcluster NeighIf x n−1 boring Node whose address is Y = (Y C , Y L ) = (((r X C + L , . . . , x 0L )), α0 ) mod Nb ), (1, ((r − 1) − β0 )2 , x n−log 2 r −2 L , ..., xL C + where α0 = (x n−2 n−log2 r −1 )r , and β0 = b(r X C C α0 )/Nb c such that X = b(Y + β0 Nb )/rc. L = 1, the node has Right-Shift Outcluster NeighIf x n−1 boring Node whose address is Y = (Y C , Y L ) = ((b(X C + L , . . . , x 0L )), where β1 = β1 Nb )/rc), (0, (α1 )2 , x n−log 2 r −2 L , ..., xL C (r − 1) − (x n−2 n−log2 r −1 )r and α1 = α1 = (X + C C β1 Nb ) mod r such that X = (rY + α1 ) mod Nb . Some examples of dBCube are shown in Fig. 6. Figure 6a shows a dBC(2, 8, 2), where each dotted circle represents a cluster consisting of nodes as a 2-cube. Each bold typeface number shows the address of a cluster, X C , and each plain

172

PARK AND AGRAWAL

FIG. 6. Examples of dBCube networks.

typeface number provides the local address within a cluster, X L . We can construct a larger cube in each cluster for dBCube. The dBC(2, 8, 3) in Fig. 6b uses a 3-cube for each cluster. The number of clusters could be any arbitrary value (including prime numbers), and such an example of dBC(2, 7, 2) is shown in Fig. 6c. The dBCube network, dBC(r, Nb , n), provides n incluster channels and one outcluster channel out of each node. Each incluster channel connects nodes within the same cluster by cube topology and an outcluster channel connects different clusters by dB graph. To obtain deadlock-free routing, we combine e-cube routing of hypercube network and virtual channel scheme of dB network. However, it is known that even though each hierarchy has deadlock-free routing scheme in a hierarchical network, it is not guaranteed that the entire hierarchical network could be deadlock-free by simply adopting each scheme for each hierarchy. We extend virtual

channel scheme of the dB graph and e-cube routing of the hypercube graph to obtain a deadlock-free routing in dBCube networks. Before we consider this, let’s see the self-routing property of a network which is helpful in selecting the route between a source and an arbitrary destination. Our early paper [16] shows the self-routing property of dBCube network, dBC(r, Nb , n). The length m = dlogr Nb e cluster-to-cluster routing paths have been constructed by the paths of the GdBG(r, Nb ) as follows. Left-Shift Cluster-to-Cluster Path. For given source node S = (S C , S L ) and destination node D = (D C , D L ), we have an r-ary number representative, U C = C C C C m (u C m−1 , u m−2 , . . . , u 0 ), of the residue (D − S ·r ) mod Nb . There exists a length m cluster-to-cluster path from S C cluster C C C to D C cluster: S C = L C 0 → L 1 → · · · → L m = D such C C C that L j +1 = (r L j + u (m−1)− j ) mod Nb .

ROUTING AND CHANNEL REDUCTION IN NETWORKS

Right-Shift Cluster-to-Cluster Path. We have an r-ary numC , w C , . . . , w C ), of the ber representative, W C = (wm−1 m−2 0 C C m residue (S − D · r ) mod Nb and the path can be conC = D C such structed as S C = R0C → R1C → · · · → Rm C C C that R j = (r R j+1 + w j ) mod Nb . Now we know the clusters through which a message passes when traversing from a source to a destination. In each traversed cluster, local routing path is determined by the dimension ordering e-cube routing of the hypercube. We now show how the local destination in a cluster can be determined. First, we assume the Left-Shift Cluster-to-Cluster Path. In L Cj cluster, the local destination is a node with a Left-Shift Cluster neighbor which is connected to L Cj+1 cluster. If we assume a local L , . . . , l L )), the local destination node with address (L Cj , (ln−1 0 C L L ) , l , (0, (u will be (L C 2 n−log2 r −2 , . . . , l0 )) from the i (m−1)− j interconnection definition. Let’s consider Right-Shift Clusterto-Cluster Path. From Lemma 1 and W C , we calculate Z C = C C C C C (z C m−1 , z m−2 , . . . , z 0 ) such that z j = b(r R j +1 + w j )/Nb c C C C and R j+1 = b(R j + z j Nb )/rc, where 0 ≤ j ≤ m − 1 to obtain the local destination address. From the dBCube interconnection definition, the local destination in R Cj cluster is of L the form (R Cj , (1, ((r − 1) − z Cj )2 , rn−log , . . . , r0L )). 2 r −2 We present the deadlock-free routing scheme in d BCube. A d BCube may have two orientations, Left-Shift and RightShift, as in a GdBG network. The Left-Shift orientation, dBC L , consists of all Left-Shift outcluster channels and all the outgoL , . . . , x L )). ing incluster channels from nodes (X C , (1, x n−2 0 These outgoing incluster channels are chosen since every LeftShift outgoing outcluster channel is connected to a node with L , . . . , x L )). Because the increasan address (X C , (1, x n−2 0 ing dimension ordering is adopted inside a cluster, these outgoing incluster channels can transfer a message to any node in a cluster. The Right-Shift orientation, dBC R , has all Right-Shift outcluster channels and all the outgoing inL , . . . , x L )). Simicluster channels from nodes (X C , (0, x n−2 0 lar discussions are applicable to the outgoing incluster channels of the dBC R as to those of the dBC L . It is obvious that E(dBC L ) ∩ E(dBC R ) = ∅ and E(dBC L ) ∪ E(dBC R ) = E(dBC). A H Ld BC (β) (H Rd BC (β)) subdigraph of the dBC L

173

(dBC R ) consists of the outcluster channels corresponding to the channels of H L (β) (H R (β)) of the GdBG L (GdBG R ) and the incluster channels of the clusters, X C ’s, that satisfies β = b(rXC )/Nb c = i. Examples of dBC L , dBC R , and their subdigraphs are shown in Figs. 7 and 8. As shown in Lemma 3, there may be two types of nodes in some GdBD’s and so two types of clusters in some dBC L ’s. In the dBC L of dBCube’s, we define a Type 1 cluster as a cluster with all outgoing outcluster channels belonging to the same subdigraph, and a Type 2 cluster as one whose outgoing outcluster channels belong to the two subdigraphs. The incluster channels of a Type 1 cluster have the same β as the outgoing outcluster channels. In a Type 2 cluster, β of the incluster channels is assigned the smaller β of the two subdigraphs. An incluster channel is viewed as a channel belonging to the H Ld BC (β). The incluster channels in a cluster X C have the same β = b(rXC )/Nb c because in rXC + α, the channel corresponding to α = 0 has the smaller or the same β among the incluster channels of a node. A similar definition can be applied to the clusters of the dBC R by changing “outgoing” to “incoming.” An example is shown in Fig. 8. ˜ X L ), We define a virtual channel address as (LR, v, β, τ, d, where LR = 0, 1 represents a dBC L or a dBC R channel, respectively, and v is a virtual channel number. β represents a subdigraph, H Ld BC (β) or H Rd BC (β) which the channel belongs to, and τ is a number assigned to each channel within the subdigraph. d˜ is a dimension number, and Y is a local source node address of a channel. Because we can assign τ of every outcluster channel to an arbitrary number in increasing order, the incluster channels of a dBC L have τ which is less than those of the outgoing outcluster channels, and greater than that of the incoming outcluster channel with the same β if it is not a loop. In a dBC R , τ of the incluster channels is assigned a value which is greater than those of the incoming outcluster channels and less than that of the outgoing outcluster channels if it is not a loop. d˜ = n for an outcluster channel and d˜ = 0, 1, . . . , n−1 for each incluster channel. Every time the next channel has the lower β than the previous channel, v is increased by 1. Cluster-to-cluster routing path is constructed

FIG. 7. Examples of Left-Shift and Right-Shift orientations in dBCube networks.

174

PARK AND AGRAWAL

FIG. 8. Examples of H Ld BC (β) and H Rd BC (β) subdigraphs in dBCube networks.

by the paths of the GdB graph and local routing path in a cluster is determined by the path of the dimension ordering e-cube routing of the hypercube. We now present the formal routing algorithm. ROUTING ALGORITHM 3. /* Routing Algorithm at node P ≡ (P C , P L ), P C = L Cj or R Cj during transmission */ C C C /* Assume (u C m−1 , . . . , u (m−1)−( j −1) ) or (z 0 , . . . , z j−1 ) are already used. */ begin if (P = D) End Algorithm; if (P = S) { v := 0; if (P C 6= D C ) { Calculate U C and Z C Calculate the next cluster Q C and local destination Q L until Q C 6= P C using U C ; Calculate the next cluster Q˜ C and local destination Q˜ L until Q˜ C 6= P C using Z C L if ((Q L = S L )k(sn−1 = 1)) direction := Left-Shift; L L L ˜ = 0)) { direction := Rightelse if (( Q = S )k(sn−1 L L Shift; Q := Q˜ ; } } } else if (P C 6= D C ) { direction := direction of the previous outcluster channel; if (direction = Left-Shift) { Calculate the next cluster Q C and local destination Q L until Q C 6= P C using U C ; if (the previous channel was an $ outclusterCchannel) { %  C r L j−1 + u (m−1)−( j−1) rP ; β prev = ; βnext = Nb Nb if (β pr ev > βnext ) v := v + 1; } }

else { /* (direction = Right-Shift) */ Calculate the next cluster Q˜ C and local destination Q˜ L until Q˜ C 6= P C using Z C ; if (the previous channel was an outcluster channel) { /* Check the special  */  C case. rP ; β prev := z Cj−1 ; βnext := Nb if (β prev > βnext ) v := v + 1; } else if (the next channel is an outcluster channel) { C C βnext = z C k+1 /* Assume each of (z j , . . . , z k ) has generated a loop. */ C) { if (P C 6= S  r PC ; β pr ev = Nb if (β pr ev > βnext ) v := v + 1; } } } } if (P C 6= D C ) { if (P L 6= Q L ) { Find min i such that piL 6= qiL Send a message over the incluster channel; } else Send a message to the outcluster channel; } else if (P L 6= D L ) { /* (P C = D C ) */ Find min i such that piL 6= diL ; Send a message over the selected incluster channel; } end; The following theorems prove that the routing algorithm in dBCube is deadlock-free and how many virtual channels it requires. THEOREM 5. lock-free.

The Routing Algorithm 3 in dBCube is dead-

Proof. From the definitions and properties of the dBC L and dBC R , they have no common channels and the proofs of

ROUTING AND CHANNEL REDUCTION IN NETWORKS

both are very similar. Therefore, we show the proof only for the dBC L . When the network contains no Type 2 clusters, within the same H Ld BC (β) subdigraph τ ’s of the channel addresses are all in the increasing order, if we view all the incluster channels of the same cluster as one channel. In a cluster, a path takes incluster channels in dimension ordering, so in the increasing order. Some networks may contain Type 2 clusters such that the outgoing outcluster channels belong to the H Ld BC (β) and H Ld BC (β + 1). From Lemma 2 there exists an incoming outcluster channel belonging to the H Ld BC (β + 1). If the incoming outcluster channel belonging to the H Ld BC (β + 1) routes to the outgoing outcluster channel belonging to the H Ld BC (β + 1), the incoming outcluster channel and incluster channels are in a decreasing order, as the incluster channels in that cluster belong to the H Ld BC (β). However, in that case the virtual channel number v is increased by one because a message is transmitted to the lower numbered H Ld BC (β) The outgoing outcluster channel uses the same increased virtual channel number. If a path takes a channel in higher numbered H Ld BC (β) than the previous channel, they are in the increasing order. Only if a channel in lower numbered H Ld BC (β) than the previous channel is used, they are in a decreasing order. However, the increase of v by 1 makes them in the increasing order. Since all paths take the channels in the increasing order, the algorithm is deadlockfree. THEOREM 6. With the maximum cluster-to-cluster path length m = dlogr Nb e, if Nb is not a multiple of r and m = pr ˜ + 1 for p˜ = 0, 1, 2, . . ., Algorithm 3 requires m + 1 − b(m − 1)/rc virtual channels for outcluster channels. If Nb is not a multiple of r and m = pr for p = 1, 2, . . ., it requires m +2−bm/rc virtual channels for the incluster channels. In all other cases, m − b(m − 1)/rc virtual channels are required for the outcluster channels, and m + 1 − bm/r c virtual channels for the incluster channels. Proof. As dBC L and dBC R have no common channels and the proofs of both are very similar, we show the proof only for the dBC L . If the network does not contain Type 2 clusters, that is, Nb = kr where k = 1, 2, . . ., then the required number of virtual channels will be m − b(m − 1)/rc for outcluster channels. To obtain the required number of virtual channels for incluster channels, we should consider a path of length m + 1 since the incluster channels will have the same β as the outgoing outcluster channels. The incluster channels require m + 1 − bm/rc virtual channels when Nb = kr. When the network contains Type 2 clusters (Nb = kr + j for j = 1, 2, . . . , r − 1), some special cases may occur. Suppose there exists a Type 2 cluster such that the outgoing outcluster channels belong to the H Ld BC (β) and H Ld BC (β + 1). From Lemma 2 there exists an incoming outcluster channel belonging to the H Ld BC (β + 1). If the incoming outcluster channel belonging to the H Ld BC (β + 1) routes to the outgoing outcluster channel belonging to the

175

H Ld BC (β + 1), it requires the virtual channel number v to increase by one, even though it is not a decreasing transition between the outcluster channels, as the incluster channels belong to the H Ld BC (β). However, from Lemmas 3 and 4, it is obvious that such a special case can occur only once in a path of length m, and as it employs only the channels of the H Ld BC (0) afterwards, it should occur in the latest part of the path. If the special case replaces a decreasing transition, it does not increase the required number of virtual channels, so it should take the place of an increasing transition in the latest part of the path as much as possible so that the required number of the virtual channels is increased at most by one. It is obvious that the only possible path sequences that satisfy both aforementioned conditions and Lemma 7 are as follows: When m = pr for p = 1, 2, . . ., P = (r − 1, r − 2, . . . , 0, r − 1, r − 2, . . . , 0, . . . , r − 1, . . . , 2, 1, 0), and if m = pr ˜ + q for p˜ = 0, 1, 2, . . ., and q = 1, 2, . . . , r − 1, P = (r − 1, r − 2, . . . , 0, r − 1, r − 2, . . . , 0, . . . , r − 1, . . . , 2, 1, 0, bm−(m mod r )+1 , bm−(m mod r )+2 , . . . , bm ) > bm−(m mod r )+i+1 for where bm−(m mod r )+i i = 1, 2, . . . , (m mod r) − 1. When m = pr, it is obvious that the special case cannot affect the required number of virtual channels because the only possible place of the special case is between b( p−1)r and b( p−1)r+1 , and so P = (r − 1, r − 2, . . . , 0, r − 1, r − 2, . . . , 0, . . . , r − 1, . . . , 2, 1, 1, 0, 0, 0, . . . , 0). If the special case occurs when m = pr + q, P = (r − 1, r − 2, . . . , 0, r − 1, r − 2, . . . , 0, . . . , r − 1, . . . , 2, 1, 1, 0, bm−(m mod r )+2 , . . . , bm ), where bm−(m mod r )+i = 0 for i = 2, 3, . . . , (m mod r). If m = pr ˜ + 1, it requires one more virtual channel to cover the special case; otherwise, the special case does not increase the required number of virtual channels because, in the partial path (b1 , . . . , bm−(m mod r )+1 ), it increases the required number of virtual channels only by one, and in (bm−(m mod r )+1 , . . . , bm ), it reduces at least one decreasing transition. So, only when m = pr ˜ +1, m +1−b(m −1)/rc virtual channels are required for outcluster channels; otherwise, m − b(m − 1)/rc virtual channels for outcluster channels. As mentioned before, we should consider a path of length m + 1 for the incluster channels. So, only when m + 1 = pr + 1, that is, m = pr for p = 1, 2, . . ., m + 2 − bm/r c virtual channels are required for incluster channels; otherwise, m + 1 − bm/r c virtual channels are required for outcluster channels. 6. DEADLOCK-FREE ROUTING IN SHUFFLE-EXCHANGE DIGRAPH NETWORKS

The shuffle–exchange directed (SED) network [7] shown in Fig. 9 has one shuffle and one exchange channel out of each node. The number of nodes is N = 2m and a node X 0 can be represented as a binary representation (x 0, m−1 , x 0, m−2 , . . . , x 0, 0 ). The shuffle channel from node X 0 has a node X 1 as a neighbor if the binary representation of

176

PARK AND AGRAWAL

LEMMA 12. No H S E D (β) of the SED has two channels e1 (Y0 , Y1 ), e2 (Y1 , Y2 ) such that e1 , e2 ∈ E(S E D exchange ).

FIG. 9. An example of a Shuffle–Exchange network.

X 1 is the left rotation of the binary representation of X 0 , which can be described as X 1 = (2X 0 +x m−1 ) mod N . The exchange channel from X 0 routes messages to X 2 where the binary representation of X 0 and X 2 differ in the least significant bit, i.e., X 2 = (x 0, m−1 , x 0, m−2 , . . . , x 0, 0 ). Each node has two incoming channels; one shuffle and one exchange channel as any two shuffle channels cannot have the same node as a neighbor. We can combine one shuffle channel and one exchange channel as a grouping pair to each other. If e1 (Y0 , Y1 ) ∈ E(S E D shu f f le ), then the corresponding exchange channel is e2 (Y1 , Y2 ) ∈ E(S E D exchange ). To construct a H S E D (β) subdigraph, we first consider only the shuffle channels. Then the corresponding exchange channels to the shuffle channels are added to the H S E D (β) subdigraphs which the shuffle channels belong to. H S E D (β) subdigraphs shown in Fig. 10 are defined as H S E D (β) = {e1 (Y0 , Y1 ), e2 (Y1 , Y2 )|2Y0 + y0, m−1 = β N + Y1 , Y2 = (y1, m−1 , y1, m−2 , . . . , y1, 0 )} for each β = 0, 1, where e1 ∈ E(S E D shu f f le ), e2 ∈ E(S E D exchange ) and Y0 , Y1 , Y2 ∈ V (S E D). Here, from the definition, Y2 can be represented using Y0 , not Y1 , as Y2 = (2Y0 + y0, m−1 ) mod N . There exists a simple way to identify which subdigraph a channel of node X 0 belongs to. From the definition, if x 0, m−1 of node X 0 is 1, its shuffle channel belongs to the H S E D (1) otherwise, the shuffle channel is in the H S E D (0). The exchange channel of node X 0 belongs to the H S E D (1) if x0, 0 is 0; otherwise it is in the H S E D (0). Property 3. E(H S E D (β)) ∩ E(H S E D (γ )) = ∅ for β 6= γ and β, γ = 0, 1 and ∪1β=0 E(H S E D (β)) = E(S E D).

FIG. 10. Examples of H S E D (β) subdigraphs in a Shuffle–Exchange Directed network, N = 8.

Proof. Assume that there exist such channels e1 , e2 in H S E D (β) as described above. Then, there should be two shuffle channels such that e3 (Y3 , Y0 ), e4 (Y4 , Y1 ), and Y3 6= Y4 as the same node cannot have two shuffle channels. From the definition, Y1 = (2Y3 + y3, m−1 ) mod N and Y1 = (2Y4 + y4, m−1 ) mod N . As all channels are in the same H S E D (β), 2Y3 + y3, m−1 = β N + Y1 and 2Y4 + y4,m−1 = β N + Y1 . From Lemma 1, Y3 = Y4 = b(β N + Y1 )/r c. Hence, it violates the assumption Y3 6= Y4 . LEMMA 13. With length m > 1, there exist no cycles in every H S E D (β) of the SED. Proof. From Lemma 12, no exchange-to-exchange channel paths exist in H S E D (β). So, we need to consider only shuffleto-shuffle and shuffle-to-exchange channel paths. Assume that there exists a cycle m > 1. A cycle consists of the shuffle channels and the shuffle-exchange channel paths. From the definition of shuffle channel and H S E D (β), it is obvious that those shuffle channels and shuffle-exchange channel paths can be viewed as the channels of the H (β) of GdBD(2, 2m ). Then, it contradicts Lemma 5. A node address X 0 in a shuffle-exchange network of N = 2m is defined as a binary representation (x0, m−1 , x 0, m−2 , . . . , x 0,0 ). The routing algorithm moves a message toward its destination one bit at a time, beginning with the most significant bit. Assume that the j most significant bits are compared and matched. At a node, the (m − j − 1)th significant bit of the destination address is compared to the least significant bit of the current node address. When the two bits match, the message is sent over the shuffle channel to rotate the node address around to the next bit. Otherwise, the message is forwarded over the exchange channel to make the two bits match, and then the next node sends it over the shuffle channel to rotate the address. Dally [7] developed a deadlock-free routing algorithm with m virtual channels for each physical channel. We show that our algorithm requires maximum m − b(m − 1)/2c virtual channels for each channel. So, in a large network, our algorithm requires approximately half the number of virtual channels as compared to Dally’s. We present a formal deadlock-free routing scheme in S E D with N = 2m . We define a virtual channel address as (v, β, τ ), where v is a virtual channel number, and β is a subdigraph number which the channel belongs to. Since from Lemma 13, H S E D (β) is acyclic, a total order can be assigned to the channels of E(H S E D (β)). τ is a unique channel number in each H S E D (β) so that all the channels of the H S E D (β) are in the increasing order. If the next outgoing channel has the lower numbered β than the previous incoming channel, v is increased by 1; otherwise, the same v of the incoming channel is used. The formal routing algorithm is shown in the following.

177

ROUTING AND CHANNEL REDUCTION IN NETWORKS

ROUTING ALGORITHM 4. /* Routing Algorithm at node X 0 */ begin if (X 0 = D0 ) End Algorithm; else if (X 0 = S0 )k := 0; else k := j; /* Assume the j bits are already matched. */ do { Y1 ≡ (y1, m−1 , . . . , y1, 0 ) := X 0 ; if (y1, 0 = d0, m−k−1 ) { Y1 := (y1, m−2 , . . . , y1, 0 , y1, m−1 ); k := + + j; } else Y1 := (y0, m−1 , . . . , y0, 1 , y0, 0 ); } while (Y1 = X 0 ); /* Check a loop. */ if (X 0 = S0 ) v := 0; else { v := the virtual channel number of the previous channel; if (x 0, 0 = d0, m−i−1 ) { βnext := x 0, m−1 ; if (the previous channel was a shuffle channel) β pr ev := x 0, 0 ;

else β pr ev := x0, m−1 ; if (β pr ev > βnext ) v + +; } } Send a message to node Y1 ; end; THEOREM 7. The proposed routing algorithm in a shuffle– exchange network is deadlock-free. Proof. The path generated by Algorithm 4 does not contain exchange–exchange sequences, but only shuffle–shuffle and exchange–shuffle sequences. Even though the path can start over the exchange channel, we may assume the use of shuffle– exchange and not the exchange–shuffle sequences. If we assume that the source node received a message over the shuffle channel of the previous channel, an exchange–shuffle sequence can be viewed as a shuffle–exchange sequence. The shuffle and the exchange channels in a sequence are a pair in the same subdigraph. So, they are always in an increasing order, and only a shuffle channel changes the subdigraph. If the path takes a shuffle channel with greater than or equal to the β value of the previous channel, they are in an increasing

TABLE I Virtual Channel Requirements in an Earlier Algorithm and Our Proposed Algorithm for de Bruijn-Based Networks GdB(D or G)(r, N b)

dBC(r, N b, 2 log r+1)

Number of virtual channels per channel

Number of virtual channels per channel Earlier Algorithm [16]

New Algorithm

r

Nb

m

Earlier Algorithm [16]

New Algorithm

outcluster channel

incluster channel

outcluster channel

incluster channel

2a

512 1024 2048 5102 8192 16400

9 10 11 13 13 15

9 10 11 13 13 15

5 6 6 7 7 8

9 10 11 13 13 15

10 11 12 14 14 16

5 6 6 7 7 8

6 6 7 8 8 9

3

512 1024 2048 5102 8192 16400

6 7 7 8 9 9

6 7 7 8 9 9

5 5 5 6 7 7

—b — — — — —

— — — — — —

— — — — — —

— — — — — —

4

512 1024 2048 5102 8192 16400

5 5 6 7 7 8

5 5 6 7 7 8

4 4 5 6 6 7

5 5 6 7 7 8

6 6 7 8 8 9

4 4 5 6 6 7

5 5 6 7 7 7

5

512 1024 2048 5102 8192 16400

4 5 5 6 6 7

4 5 5 6 6 7

4 4 5 5 5 6

— — — — — —

— — — — — —

— — — — — —

— — — — — —

a

Also applies to SED(2 m). dBCube does not exist.

b

178

PARK AND AGRAWAL

TABLE II Summary of Maximum Path Lengths and Maximum Virtual Channel Requirements in Different Shuffle-Based Networks Networks

Algorithm

Size

Maximum path length

Maximum virtual channel requirements

dBD or dBG

Existing [16]

rm

m

m

Ours

rm

m

m − b(m − 1)/r c

Existing [16]

Nb

dlogr Nb e

dlogr Nb e

Ours

Nb

dlogr Nb e

dlogr Nb e − b(| logr Nb | − 1)/r c

Existing [16]

Nb × 2n

dlogr Nb e (cluster-to-cluster)

outcluster channel dlogr Nb e

Ours

Nb × 2

GdBD or GdBG

dBC(r, N b, n)

n

dlogr Nb e (cluster-to-cluster)

incluster channel dlogr Nb e + 1 outcluster channel a dlogr Nb e − b(dlogr Nb e − 1)/r c incluster channel b dlogr Nb e + 1 − b(dlogr Nb e)/r c

SED or SEG

Dally’s [7] Pifarré’s [19] Ours

2m

2m

m

2

m

3m

4

2

m

2m

m − b(m − 1)/r c

a

This applies to most cases except the following:   dlogr Nb e − 1 dlogr Nb e + 1 − ˜ + 1( p˜ = 0, 1, 2, . . .). when Nb 6= kr (k = 1, 2, . . .) and dlogr Nb e = pr r

b

This applies to most cases except the following:   dlogr Nb e dlogr Nb e + 2 − when Nb 6= kr (k = 1, 2, . . .) and dlogr Nb e = p˜ ( p˜ = 1, 2, . . .). r

order. When a channel in the lower numbered H S E D (β) than the previous channel is used, they are not in the increasing order, but the increase of v by 1 makes them in the increasing order. Because all the channels of a path are in the increasing order, it is deadlock-free.

the comparisons between our new routing algorithms and the existing ones.

THEOREM 8. Algorithm 4 requires maximum m − b(m − 1)/2c virtual channels.

1. Agrawal, D. P. Advanced Computer Architecture. IEEE Computer Society Press, 1986.

Proof. The path length is at most 2m−1 with m−1 shuffles and m exchanges. If we consider an assumed shuffle channel to the source node, there are m shuffles. As only the shuffle channels can change the β value, we consider a path of length m. Since a shuffle or a shuffle-exchange sequence can be viewed as a channel of a d BG(2, 2m ), then from Theorem 2, Algorithm 4 requires a maximum of m − b(m − 1)/2c virtual channels. 7. CONCLUSIONS

A new deadlock-free algorithm called β algorithm for a generalized de Bruijn digraph has been introduced in this paper. We have shown that with this algorithm, m − b(m − 1)/rc virtual channels are required in each channel when the maximum path length is m. We extended it to a GdBG, a dBCube and a SED. Table I shows some examples of the virtual channel requirements in an earlier algorithm and our new algorithm with various r’s and Nb ’s. Table II summarizes

REFERENCES

2. Boppana, R. V., and Chalasani, S. A new framework for designing deadlock-free wormhole routing algorithms, Tech. Rep. CS-94-1, Div. Math., Comp. Sci. and Statistics, The University of Texas at San Antonio, San Antonio, May 1994. 3. Chen, C., Agrawal, D. P., and Burke, J. R. dBCube: A new class of hierarchical multiprocessor networks with area efficient layout. IEEE Trans. Parallel Distrib. Systems 4, 12 (Dec. 1993), 1332–1344. 4. Collins, O., Pollara, F., Dolinar, S., and Statman, J. Wiring Viterbi decoders (splitting de-Bruijn graph), Tech. Rep. 42-96, JPL TDA Progress Report, Feb. 1989. 5. Dally, W. J. Virtual-channel flow control. IEEE Trans. Parallel Distrib. Systems 3, 2 (Mar. 1992), 194–205. 6. Dally, W. J., and Seitz, C. L. The torus routing chip. Distrib. Comput. 1, 3 (1986), 187–196. 7. Dally, W. J., and Seitz, C. L. Deadlock-free message routing in multiprocessor interconnection networks. IEEE Trans. Comput. 36, 5 (May 1987), 547–553. 8. Du, D. Z., and Hwang, F. K. Generalized de Bruijn digraphs. Networks 18 (1988), 27–38. 9. Esfahanian, A., and Hakimi, S. L. Fault-tolerant routing in de Bruijn communication networks. IEEE Trans. Comput. 34, 9 (Sept. 1985), 777– 788.

ROUTING AND CHANNEL REDUCTION IN NETWORKS 10. Imase, M., and Itoh, M. Design to minimize diameter on building-block network. IEEE Trans. Comput. 30 (June 1981), 439–442. 11. Labourdette, J.-F. P. Performance impact of partial reconfigurability in lightwave networks. Proceedings of the IEEE Infocom’95. Apr. 1995, Vol. 2, pp. 683–691. 12. Lillevik, S. Touchstone program overview. Proceedings of The Fifth Distributed Memory Computing Conference. Charleston, SC, 1990. 13. Linder, D. H., and Harden, J. C. An adaptive and fault tolerant wormhole routing strategy for k-ary n-cubes. IEEE Trans. Comput. 40, 1 (Jan. 1991), 2–12. 14. Ngai, J. Y. A framework for adaptive routing in multicomputer networks. Ph.D. thesis, California Institute of Technology, 1989. 15. Ni, L. M., and Mckinley, P. K. A survey of wormhole routing techniques in direct networks. IEEE Comput. 26, 2 (Feb. 1993), 62–76. 16. Park, H., and Agrawal, D. P. WICI: An Efficient hybrid routing scheme for scalable and hierarchial networks. IEEE Trans. Comput. 45, 11 (Nov. 1996), 1272–1281. 17. Park, H., and Agrawal, D. P. Efficient deadlock-free wormhole routing in shuffle based networks. Proceedings of the Seventh IEEE Symposium on Parallel and Distributed Processing. Oct. 1995, pp. 92–99. 18. Park, H., and Agrawal, D. P. A novel deadlock-free routing technique for a class of de Bruijn graph based networks. Proceedings of the 9th International Parallel Processing Symposium. Apr. 1995, pp. 524–531. 19. Pifarré, G. D., Gravano, L., Felperin, S. A., and Sanz, J. L. C. Fully adaptive minimal deadlock-free packet routing in hypercubes, meshes, and other networks: Algorithms and simulations. IEEE Trans. Parallel Distrib. Systems 5, 3 (Mar. 1994), 247–263. 20. Samatham, M. R., and Pradhan, D. K. The de Bruijn multiprocessor network: A versatile parallel processing and sorting network for VLSI. IEEE Trans. Comput. 38, 4 (Apr. 1989), 567–581. 21. Sivarajan, K. N., and Ramaswami, R. Lightwave networks based on de Bruijn graphs. IEEE Trans. Networking 2, 1 (Feb. 1994), 70–79.

HYUNMIN PARK was born in Pusan, Korea, on March 7, 1962. He received the B.S. in electronics engineering from the Seoul National September 1, 1995; revised November 12, 1996; accepted August 29, 1997

179

University, Seoul, Korea, in 1985; and the M.S. and Ph.D. in computer engineering from North Carolina State University, Raleigh, in 1988 and 1995, respectively. He is an assistant professor in the Department of Computer Engineering at Myongji University in South Korea. His research interests include routing protocols, deadlock-free routing, multicasting, broadcasting, parallel processing and computer architecture. DHARMA P. AGRAWAL received the B.E. in electrical engineering from the Ravishanker University, Raipur, in 1966, the M.E. (Hons.) in electronics and communication engineering from the University of Roorkee, Roorkee, 1968, and the D.Sc. Tech degree in electrical engineering from the Swiss Federal Institute of Technology, Lausanne, 1975. He is currently a professor of electrical and computer engineering at the North Carolina State University, Raleigh, NC. His current research interests are in parallelism detection and scheduling, mobile and communication networks, and system reliability. He is an author of a self-study guide on “Parallel Processing” (IEEE Press, 1991), an editor of a tutorial text on “Advanced Computer Architecture” (1986, IEEE Computer Society Press), and co-editor of tutorial texts on “Distributed Computing Network Reliability,” 1990, and “Advances in Distributed System Reliability.” 1990. He is a recipient of the “Certificate of Appreciation” and three “Meritorious Service Awards” from the IEEE Computer Society. He has served as an editor of the IEEE Computer, the IEEE Transactions on Computers, and the “IEEE Computer Society Press,” and is currently an editor of Journal of Parallel and Distributed Computing (Academic Press) and the International Journal of High-Speed Computing (World Scientific Publishing Co.). Dr. Agrawal has served as the General Chair for MASCOTS 1996 workshop, and the founding Chair of the 1995 ICPP Workshop on Challenges for Parallel Processing, the Program Chair of the 1994 International Conference on Parallel Processing, (Chicago, Aug. 1994), the General Chair for the 6th International Conference on Parallel and Distributed Computing Systems (Louisville, Oct. 1993), the Program Chairman of the 11th International Symposium on Computer Architecture (1984), the Army Research Office Workshops on Future Directions in Computer Architecture and Software (1986). Currently, he is the Chairman of the IEEE–CS Harry Goode Memorial award and the IEEE–CS W. Wallace McDowell Award. He has been a member of the ACM Lecturership Committee, the Computer Science Accreditation Board, and the IEEE CS Distinguished Visitor Committee.