LCMSC: A lightweight collaborative mechanism for SDN controllers

Accepted Manuscript

LCMSC: A Lightweight Collaborative Mechanism for SDN Controllers Ming CHEN , Ke DING , Jie HAO , Chao HU , Gaogang XIE , Changyou XING , Bing CHEN PII: DOI: Reference:

S1389-1286(17)30165-2 10.1016/j.comnet.2017.04.029 COMPNW 6168

To appear in:

Computer Networks

Received date: Revised date: Accepted date:

27 July 2016 19 December 2016 7 April 2017

Please cite this article as: Ming CHEN , Ke DING , Jie HAO , Chao HU , Gaogang XIE , Changyou XING , Bing CHEN , LCMSC: A Lightweight Collaborative Mechanism for SDN Controllers, Computer Networks (2017), doi: 10.1016/j.comnet.2017.04.029

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

LCMSC: A Lightweight Collaborative Mechanism for SDN Controllers

Ming CHEN1, Ke DING1, Jie HAO3, Chao HU1, Gaogang XIE2, Changyou XING1, Bing CHEN3 (1 PLA University of Science & Technology, Nanjing, 210007, China 2 Institute of Computing Technology, CAS, Beijing, 100190, China

CR IP T

3 Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China)

Abstract: SDN-based applications can be deployed on large-scale networks only with east/westbound interfaces. However, there is still a long way to standardize SDN east/westbound due to the high complexity and comprehensive optimization goals. To make it possible to deploy SDN in the Internet to support both traditional and emerging network applications, we propose a

AN US

lightweight collaborative controller mechanism which has advantageous characteristics such as application independent, controller independent, minimum functions, reliability and etc. We, also design its software program named LCMSCware running on each controller and its application layer protocol enabling corporation among the controllers. The prototype system is implemented and the experimental results demonstrate that LCMSC enables efficient cooperation among

existing and emerging services.

M

heterogeneous controllers in multiple SDN domains connected by Internet and support both

ED

Keywords: SDN, controller, east/westbound interfaces, cooperation mechanism, lightweight protocol

PT

1. Introduction

CE

To deal with the ossification of TCP/IP architecture, Software-Defined Networking (SDN) is

born. The area dominated by a SDN controller is called SDN administrative domain (the domain

AC

for short), each is limited by the capacity of the single controller and thus only includes tens of switches [1]. In this way, a large enterprise or carrier network can be divided into a number of non-overlapping SDN domains. However, the centralized mechanisms of SDN cannot be directly used to manage the different domains in the Internet. What‟s worse, the network applications generally involve two or more domains. If the controllers in these domains cannot collaborate efficiently, the domains are just some isolated information islands that definitely stunts the high efficiency of SDN. It is obvious that the cooperative mechanisms of controllers are indispensable for various SDN applications across the Internet. In order to cope with the increasing demands for the controller collaboration mechanism,

1

ACCEPTED MANUSCRIPT

scientists from academia and industry have been devoting to design distributed mechanisms for controller management in large scale networks [3, 4]. For example, Google‟s B4 [2] collaborates tens of controllers in its datacenters around the world, which greatly improves the utilization of high speed links in the wide area networks. East/westbound interfaces are a special case of interfaces required by distributed controllers. The functions of these interfaces include import/export data between controllers and monitoring/notification capabilities [24]. Specifically, the east/westbound interfaces are APIs used for interaction among the distributed controllers. The

CR IP T

functions of these interfaces include data transmission, supervision, notification and etc.

For distributed controllers, east-westbound APIs are basic components. East/westbound is very valuable for the following reasons: 1) Scalability. A reasonably large network may need to deploy multiple SDN controllers for the network expansion. 2) Reliability. The use of multiple controllers avoids the risk of network paralysis caused by a single controller failure. 3) Privacy

AN US

flexibility. A carrier may choose to implement different privacy policies in different SDN domains. 4) Incremental deployment. Dividing the network into multiple individually manageable SDN domains allows for flexible incremental deployment. However, the design of the east/westbound is an extremely complex engineering issue, involving SDN development direction, several kinds of advanced technologies, dedicated protocol design, heterogeneity and etc [5]. The community has

M

not yet reached an agreement on its standard, thus there are a variety of controller set and their own specific APIs. At present there exist several problems about existing east/westbound research:

ED

1) Various dedicated controller coordination mechanisms have been developed, which will lead to great waste of manpower and material resources. 2) There is too much attention on long-term requirements while ignoring the current urgent needs of the applications. 3) Inheriting the Internet

PT

resources has not been paid enough attention. 4) The advantages of SDN have not been fully exploited. All these inspire us to think about such a question: can we develop a general,

CE

lightweight SDN controller coordination mechanism, build a running system which can be used to connect private SDNs conveniently through the Internet and then make it evolve depending on the

AC

requirements?

In order to answer this question, this paper proposes a lightweight SDN controller

coordination mechanism, which can be rapidly deployed on the Internet to satisfy the requirements of both inheriting the Internet software and hardware resources and accommodating the emerging applications. The contributions of this paper include: i) A lightweight collaborative mechanism for SDN controllers (LCMSC) and its model are proposed; ii) The software program of LCMSC named LCMSCware running on the controllers and the LCMSC protocol for controller communication are designed;

2

ACCEPTED MANUSCRIPT

iii) The LCMSC prototype is implemented, and the effectiveness of the mechanism is verified. The organization of this paper is as follows. Section 2 overviews the related works. Section 3 describes the principles of LCMSC and presents a model. Section 4 discusses the key technologies of LCMSCware implementation in detail. The performance of LCMSC is analyzed in Section 5. Section 6 implements and tests a prototype system. Finally, Section 7 concludes the paper.

CR IP T

2. Related Work

To construct multiple SDN domains, east/westbound APIs are the interfaces between controllers, which are usually regarded as extended functions of the network operating system. There are specific communication protocols among controllers in SDN domains [6] whose basic

AN US

functions include the coordination of flow setup, information exchange of the reachability of different SDN domains, maintenance of the network states consistency and etc. As an example, SDNi [3] defined the common requirements of the intra-domain flows to coordinate flow setup and exchange reachability information.

Programming the network management application through a uniform centralized API to hide

M

the details of different kinds of devices is the goal of distributed operating system. This relates to the issues such as the distributed architecture of the controllers, which are significantly complex.

ED

The distributed architecture can be hierarchically arranged, e.g. Onix [7], which supports multi-controller around the wide-area network and provides flexible primitives to distribute network status. Onix is rather complex which has multiple management levels and quite a few

PT

components to support sorts of synchronization and other goals. It‟s not easy to deploy Onix to the already-existing SDNs located at the edge of the Internet which owned by different properties.

CE

HyperFlow [23] is an Openflow oriented, event-based platform where the controller uses the “push” method to distribute the local state changes as network events to the other controllers.

AC

Although event sensitive, this mechanism performs inferior in the highly dynamic network environment. HyperFlow is heavily depending on WheelFS - a distributed file system which should be properly configured via a set of performance parameters. This is not convenient for deploy and use. Kandoo [22] used a layered-distribution, which deploys a higher-layer controller beyond multiple distributed lower-layer controllers. This hybrid mechanism takes advantage of both centralized and distributed systems: only the requests for the centralized view will be sent to the higher-layer controller. Kandoo use 2-layered structure and is a standalone controller implementation, which is not fitful for existing SDNs which have their own controllers. In addition, a lot of advanced complex techniques are related, e.g. the input/output function of Onix

3

ACCEPTED MANUSCRIPT

[7], CE-CE interfaces of ForCES [8, 9], highly available cold backup mechanism of ForCES Intra-NE [10] and distributed storage [11]. The east/westbound APIs require advanced data distribution technologies, such as Advanced Message Queuing Protocol (AMQP) [12], distributed dispatching and consistent policy composition [13], DHT [14], strong consistency or fault-tolerant algorithms [11, 15], etc. Although much progress has been made, some problems still need to be addressed: 1) There is a huge obstacle to design a global network operating system. It‟s a long way for OS level distributed communication or subscribing/dispatching

CR IP T

function implementation. 2) The coordination mechanisms of normal applications are often not globally issues but only involve a few controllers. For example, the C/S mode applications only need 2 controllers to collaborate. 3) The Internet is a game among ISPs, ICPs, governments and users [21]. We should achieve the good tradeoff between the current requirements and long-term requirements when developing east/westbound interfaces.

AN US

Heterogeneity is another issue that east/westbound interfaces should address. Besides peer-to-peer communications in the same layer, one controller also needs to communicate with the controllers in the lower-layer or the non-SDN controllers [16, 17]. East/westbound interfaces may need to traverse the borders of multiple management domains and support all sorts of controller interfaces. These controller interfaces have their specific service set and the characteristics of the

M

low level infrastructure including the technology diversity, network span or network scale and differences between WAN and LAN. In the heterogeneous cases, the controllers must exchange

ED

different kind of information, including adjacency relation, capability discovery, account information, etc [16]. SDN compass [18] discriminates eastbound interfaces and westbound interfaces, and the eastbound interfaces define the SDN-to-SDN protocols while the westbound

PT

interfaces define the protocols of the traditional network control plane (e.g. PCEP [19], GMPLS [20]). If we limit the function of east/westbound interfaces to the information interactions of

CE

network applications neglecting the semantics and the states of the applications processing and assume the controllers communicate with each other only through IP network, it would be much

AC

easier for communication expansion in the heterogeneous controller environment. Fibbing [27] introduces fake nodes and links into an underlying linkstate routing protocol,

then routers compute their own forwarding tables based on the augmented topology. SDN Partitioning [28] uses SDN switches as border nodes to partition the original distributed routing domain into sub-domains and then either SDN control messages or legacy routing messages can flow between domains. InitSDN [29] bootstrapping the distributed SDN architecture and deploying the distributed controllers from legacy network to the new architecture. However, all these efforts are not aiming to address the issue of collaboration problem between existing SDN domains which owned and being fully controlled by different organizations.

4

ACCEPTED MANUSCRIPT

In summary, existing research focus on developing a comprehensive collaborating controller system from bottom to up or based on some corporation‟s own WAN infrastructure to develop new mechanisms to cope their own specific requirements. Quite a few of them are much successful. However, best to our knowledge, none of the existing work was aiming to build a lightweight mechanism to connect existing SDNs‟ heterogeneous controllers. On the developing of SDN technology, many corporations or research organizations have built up their own SDNs at the edge of the Internet. LCMSC is lightweight, easy to use and ready to be deployed to existing controllers.

CR IP T

With this lightweight running system, we can connect all kinds of private SDNs via the Internet to inherit the legacy, and what makes it more important is we can always get an improved running system when we find something is important enough to add to the core of the LCMSC. This causes incremental and rolling development.

AN US

3. Principles of LCMSC 3.1 Prerequisite

SDN domains normally locate at the edge of the Internet, which are connected by the core

M

network that is composed of the IP routers. In each domain, the end systems running applications connect to OpenFlow switches, and thus the enterprise network is formed in which the controller

ED

dominates packet transmission by installing flow entries in the switches. Fig. 1 shows a simple scenario. Suppose an application of Server 1 in domain 1 wants to communicate with that of Server2 in domain 2, the packets would pass through the end-to-end path, which usually consists

PT

of three sub-paths. Sub-path 1 is from Server 1 to the port of the egress switch connecting to R1, which is established by the controller C1. Sub-path 2 is from router R1 to router R2, which is

CE

determined by the IP routing algorithms. Sub-path3 is from the port of the ingress switch

AC

connecting to R2 to Server2, which is established by the controller C2.

Controller C1 Servers1

R1

Path1 Domain 1

Controller C2

East/westbound

Egress switch

IP network Path2

Servers2 IP router

R2

Ingress Domain 2 switch

The core network Domain 4

Domain 3 End system

Path3

The edge network

Fig 1. An Abstract Working Environment for LCMSC The design of east/westbound interface should solve the following properties: P1: The inter-domain path from the source host (the ingress switch) to the egress switch (the 5

ACCEPTED MANUSCRIPT

destination host) is built by installing flow entries to related switches by local controller such as C1 or C2. Whether the path can be built depends on not only their routing but also the policies of the security, QoS, resource management, etc. P2: The end-to-end connection requested by the application can be properly set up only if all the controllers have reached an agreement after negotiation. In Fig. 1, Path1 and Path3 are built by controller 1 and controller 2 respectively, while Path2 is established by the distributed algorithm of the IP network. Here, we suppose there always exists Path2 satisfying application requirement.

CR IP T

Since a controller is aware of all the resource information of its domain, it can properly set up the path within the local domain satisfying application requirement, such as Path1 or Path3. The key to address P2 is: controllers can negotiate each other based on the application requirement and their domain status. Only if they reach an agreement an end-to-end path can be established.

P3: To support each kind of applications, controllers should maintain their corresponding

AN US

state space and computational resources. Due to the complexity and diversity of the applications, what the state space should contain, how to keep consistency of the state space and etc. are all challenges to be addressed. On one hand, when the state space is global, the domain information updating will consume a large amount of network bandwidth, especially in large scale networks. On the other hand, various applications require different state spaces with slim sharing opportunity.

M

For these reasons, if we separate the collaborative mechanism and the state space that the coordination mechanism only involves the rules of interaction and the semantic processing of the

ED

applications is left to the applications themselves, it will greatly simplify the controller collaboration mechanisms and benefit the development of emerging network applications. P4: The controllers may be heterogeneous. If we extract the common part of the controller

PT

communication requirements as a functional minimum set, it will be much convenient to realize the interaction of the controllers.

CE

The design of LCMSC primarily aims to provide interactions among controllers. The concise and efficient addressing mechanism in IP network can efficiently support the controller

AC

cooperation. This facilitates large scale deployment of LCMSC, establish rapidly distributed service capacity as well as inherits the hardware resources of the Internet infrastructure. Different from magnificent goal of network operation system (NOS) design, LCMSC expects

to develop a lightweight and generally appropriate cooperation mechanism to sustain various network applications, including both traditional Internet applications and emerging network applications. This means LCMSC should be application-independent, so that it can support both existing traditional and new-style network applications. Its simplicity makes it‟s easy to add new functions.

6

ACCEPTED MANUSCRIPT

3.2 Design Consideration If we design the controller cooperation mechanism as a canonical NOS, the high technical complexity blocks us to develop a usable system in the near future. Furthermore, current network applications only involve finite controllers and resources, and their requirements have major discrepancies. Due to the diversity and complexity of network applications, there are no perfect NOS for all applications. In fact, since network applications are aware of their own demands, we

CR IP T

only need to design a lightweight controller cooperation mechanism, which takes charge of the basic tasks and leaves the application-related work to the applications themselves. Specifically, this lightweight mechanism should possess the following properties: 

Application Independence. This mechanism should be independent of specific network

applications, so as it can sustain broad network innovation. This mechanism should provide basic

AN US

cooperation support among finite involving controllers for different application patterns and leave the application dependent tasks and states to the applications. That is „An application understands its relative SDN controllers' coordination and application level needs best‟. 

Function Minimization. This mechanism should provide fundamental functions, e.g.

establishing connections among coordinating entities. Concise function set is convenient for



M

accelerating application response and expanding network coverage.

Controller Independence. Different types of SDN or non-SDN controllers should

ED

conveniently integrate into the cooperation mechanism, and achieve cooperation via brief interfaces regardless of the specific functions of a certain controller. 

Reliable Communication. This mechanism should offer reliable communication

PT

connection for the controllers.

CE

3.3 LCMSC

In this subsection, we illustrate how LCMSC works.

AC

Within SDN management domains Dj (j=1, 2, …, n), the entities in Dj are connected with the

OpenFlow switches which are controlled by the controller Cj of the domain. The communication between Dj must pass through Dcore which is composed of routers and links. If Cj has an IP address IPCj in Dcore, Cj can join the collaboration controller set CoCtlerSet after authentication and then it can obtain the other controller‟s IP addresses. Assume entities in Dj are connected with Cj by default and we can use the entity name to locate Cj and the IP address of the controller. Before an application appi in Di interacts with an application appj in Dj, Dk, …

(i  j ,k ,...) , it will trigger LCMSC which contains the collaboration controller set

7

ACCEPTED MANUSCRIPT

CoCtlerSet and support the related interactive communication mode. There are 2 methods to trigger LCMSC, one is using the first packet of the application flow, and the other is actively collaboration of unicast, multicast or anycast please refer to the following discussion about „Controller collaboration modes‟). All the controllers in CoCtlerSet check security permission constrains Securityis and Securityid of the flow in their home domain s and d, arrange related resource constrains Resourceis and Resourceid, and then build up the path Pathi as, Pathi = {Securityis^Resourceis^Securityid^Resourceid}

CR IP T

(1)

In the flow duration of appi, the collaborated controllers can monitor and control the Pathi based on the 4-tuple of the flow.

According to this model, Fig 2 shows the collaborating process of C1 in D1 and C2 in D2. This process supports the C/S mode network application APPweb running on the end system Hs and Hd1

LCMSCWare deployed here D1:Hs

D1:DP1

D1:C1

first packet to controller

AN US

(such as Web service). The detailed procedure is as follows.

LCMSCWare deployed here

Dcore:R1

Dcore:R2

D2:C2

D2:DP2

D2:Hd1

collaboration msg

IP forwarding

M

...

first packet IP forwarding

data

...

ED

data IP forwarding

...

. . .

data

PT

data

setup rules

collaboration msg

collaboration msg

setup rules

collaboration msg

data

data IP forwarding

...

data

data

Fig 2. Collaboration between C1 in D1 and C2 in D2

AC

CE

. . .

(1) Launce: when a packet of app1 on end system Hs reaches DP1 without any matched rules,

the packet would be delivered to C1. (2) Invoke LCMSC: the collaboration program LCMSCware on C1 parsed the packet

information. To achieve this, the program should identify the collaboration controller subset {C1, C2} and the unicast mode. (3) Deliver collaboration information: according to collaborating control mode, set up a controller set by invoking the LCMSCware service, i.e.C1 uses reliable unicast mode (e.g. TCP) to deliver the related information (such as characteristics of the flow) to C2.

8

ACCEPTED MANUSCRIPT

(4) Perform control by collaborating the controllers: after C1 and C2 in CoCtlerSet analyze app1‟s security requirements Security1 and Security2 and resource requirements Resource1 and Resource2, if both controllers agree and notify each other, forwarding rules are placed in D1 or D2 and a path from Hs to Hd is built up. Then the client (browser) and the server (web server) are both operational. In the meantime, both C1 and C2 can manage this flow. The lightweight collaboration between the controllers is dependent on the network application mode. Presently, there are only two network application modes: C/S and P2P.

CR IP T

Therefore, collaboration only has limited patterns, e.g. C/S mode only involves two controllers in two related domains to collaborate, while P2P mode may involve i relative domains and i controllers need to be collaborated where i is the controller number.

In Fig 3 we use Directed Acyclic Graph (DAG) to describe collaboration methods among the controllers. Fig. 3(a) shows the unicast method used by C/S applications. This is the simplest and

AN US

most widely used method. Fig. 3(b) shows the multicast method used by P2P applications: a source controller should collaborate with all the controllers in the related domains. Fig. 3(c) shows the anycast method: a source controller only collaborates with anyone of the controllers in the related domains.

Cj

Ci …

…

Cg

Cg

(b) Multicast mode

(c) Anycast mode

PT

(a) Unicast mode

Cj

Ck

Ck

Ci

ED

Ci

M

Cj

CE

Fig 3. Controller collaboration modes

4. Implementation of LCMSC

AC

As described in section 3.3, LCMSC consists of two implementation modules, LCMSCware

and LCMSC protocol parser. As shown in Fig.4, LCMSCware runs on each controller and using LCMSC protocol to sets up logical communication channels for controller collaboration. In this section, we‟ll present the detailed implementation respectively.

9

ACCEPTED MANUSCRIPT

4.1 LCMSCware

Ci

Cj

Ck

LCMSC Protocol LCMSCware

IPi

Socket()

...

IPj

IP Infrastructure

Fig 4. LCMSC framework

LCMSCware

E space

IPk

C space

CR IP T

LCMSCware

LCMSCware is the core part of the controller collaboration mechanism that consists of 3 key function modules.

AN US

1 ） Collaboration initialization module: This module provides functions to identify collaboration mode. The collaboration mode includes coordinating the collaboration controller set and providing communication methods. The former includes at least two controllers‟ IP addresses of the core network, and the latter specifies the method of unicast, multicast or anycast. The collaboration mode can be expressed as: (coop-ID, {IPaddress set}), where coop-ID is the collaboration code representing unicast, multicast or anycast, IPaddress set is the set of all the

M

controller‟s IP addresses involved in the collaboration. Despite the huge number of networks, the number of coop-ID is small. The traditional applications of IP network correspond to C/S and P2P

ED

modes: C/S to (unicast, {IPC1, IPC2}) and P2P to (multicast, {IPC1, IPC2, IPCi, …}). The future emerging network applications should have similar collaboration modes. LCMSCware provides

PT

two interfaces to the applications. One is to identify the collaboration mode directly from the first packet of the flow of the application. This can be applied without any modification of current C/S

CE

or P2P applications. The other interface is to provide dedicated APIs whose parameters are specified by the controllers to satisfy some novel special needs. 2）Collaboration information delivery module: Taking advantage of the IP infrastructure,

AC

this module uses standard sockets to implement unicast, multicast and anycast functions and reliably deliver the necessary collaboration information to the other controllers in the collaboration controller set. The collaboration information includes flow attributes and the actions that should be applied to the flow. By default, it unicasts the collaboration information based on TCP, or multicast it via reliable UDP communication. 3）Local cooperative operation module: When all the controllers in the collaboration controller set have received the collaboration information, they immediately dispatch the flow entries and enforce corresponding actions. The collaboration information delivery module and the local cooperative operation module can be considered as two functions for the two phases 10

ACCEPTED MANUSCRIPT

respectively in the two-phase commit (2PC) protocol. Without loss of generality, we consider the communication between two hosts in different domains. The path between the two endpoint hosts should be setup by the controllers to which the hosts belong. We call the controller „initiator‟ if the communication was initiated by the host managed by the controller. We call the controller „responder‟ if the communication‟s destination was the host inside the controller‟s management area.

of the responder is similar. Algorithm 1. LCMSC for the initiator Input:

The first packet p in a flow or coop-ID, {IPaddress set}, management policies C

Output: H - handler of reliable connections to all controllers, NIL on failure 1.

H=NIL;

2.

if LCMSCAPI is invoked

// an application process invokes LCMSC API

extract coop-ID and{IPaddress set} from input parameters;

4.

if coop-ID and {IPaddress set} are in accord with C

AN US

3.

5.

make reliable connections to all other controllers;

6.

if all related controllers response

7.

append connections to H; //success

8.

end if end if

M

9. 10.

else

// p is from a traditional internet application

11.

extract port number and destination IP from p;

// obtaining port number

if port number belongs to well-known port &&destination IP in accord with C

ED

12. 13.

make reliable connections to the other controller;

14.

if the related controller response append connection to H; //success

PT

15. 16.

end if

19.

end if

CE

17. 18.

CR IP T

Algorithm 1 shows the pseudo algorithm of the initiator in LCMSCware, and the algorithm

end if

return H

AC

We implement LCMSCware modules in Python. Thanks to the cross platform characteristics

of Python, LCMSCware is easily planted to various kinds of platforms. In the POX, NOX and Ryu controllers, we extend related interfaces to support LCMSC by

adding less than 100 lines source code. We realize the collaboration among these 3 kinds of controllers by using LCMSC. The basic idea behind LCMSCware is simple: it accepts commands and data from the controller and then rearranges and properly dispatches related information to peer LCMSCware modules by TCP connections or reliable UDP channels. LCMSCware is convenient to use. The initiator can create reliable connection with the 11

ACCEPTED MANUSCRIPT

module by invoking CoOp_Create_Conn()and getting the connection descriptor h (in this step, the controller will register to a specific collaboration group). Then the controller can invoke function CoOp(h, msg)to send message msg. All registered controller will receive the message msg through an event. The response methods to asynchronous events are different among different kinds of controllers. There exists a little difference when receiving messages from LCMSCware among different controllers. Specifically, for NOX controller, CoOp_handler()is used to process the messages from LCMSCware:

CR IP T

register_handler(CoOp_event.static_get_name(), CoOp_handler).

For POX controller, to enable the response function handle_CoOp_event(), the collaboration initiator should add the following code in the proper location: _eventMixin_events = set([CoOp_event]) core.CoOp_event.addListeners(self)

AN US

For Ryu controller, to enable _coop_handler(), the following code should be added: @set_ev_cls(ofp_event.EventCoOp, MAIN_DISPATCHER) _coop_handler(self, ev):

M

4.2 LCMSC Protocol

LCMSC protocol is an application layer protocol for reliable transmission, which is used for

ED

information exchange between LCMSCware. First, when the initiator wants to coordinate with the other controllers, it establishes a TCP connection or reliable IP multicast group from the source

PT

controller to the destination controllers, and then the LCMSC message with cooperation information will be sent to the receivers. Second, once all members in the collaboration controller

CE

set respond to this packet, the initiator will notify all members to perform local cooperation. Subsequently, all members return the execution results to the initiator. Finally, the initiator examines whether this cooperation is successful or not, and recycles the resources if it is

AC

successful. Otherwise, it restarts a new cooperation request or abandons the preceding cooperation according to the situation. Fig 5 shows the fields of LCMSC protocol packet. 1 byte

Version

1 byte

Type

2 bytes

Identification

1 byte

Seqno

2 bytes

Body Length

42~1498 bytes

Data Body

Fig 5. The fields of LCMSC protocol packet The length, definition and description of each field are as follows.

12

ACCEPTED MANUSCRIPT

Version: 1-byte length, the version of protocol, and current version is 1. Type: 1-byte length, the type of packet, including request packet, responding packet and their sending direction. Identification: 2-byte length, used for distinguishing different cooperation process, and the identification value of request packet should be same as that of responding packet in the same cooperation process. Seqno: 2-byte length, the sequence number of the next packet in the same cooperation

CR IP T

process is increasing by 1. This field is only used in implementation of the reliable UDP delivery. Body Length: 2-byte length, the length of data body in cooperation messages.

Data Body: 42~1498 bytes length, the payload of LCMSC packet. This field can be defined by proprietary data structures in json coding according to Type, Identification and Seqno fields.

AN US

5. Discussions

In this section, we‟ll discuss several design considerations of LCMSC regarding cooperation delay tcoll, cooperation state number scoll, and cooperation fault probability fcoll. Cooperation delay tcoll measures how quickly the cooperation is accomplished, cooperation state number measures the

M

complexity of state measurement and cooperation fault probability

Assuming the controller processing delay is tproc, and the transmission delay between the two

ED

controllers is ttrans, then tcoll = 2tproc+ ttrans can be obtained. No matter how large the network is, the fluctuation of the value of tcoll is relative small for it would usually being related only to a few nodes, which does not have much negative influence on LCMSC mechanism having flat structure.

PT

Considering the impact of its security mechanism, we assume the authentication delay of the controller participating in the coordinated controller set is tasymmetry and the authentication delay

CE

when exchanging LCMSC messages is tsymmetry, tasymmetry≫tsymmetry. Because tasymmetry only exists in the cooperation initialization phase while tsymmetry affects the cooperation process frequently,

AC

LCMSC has acceptable delay property. For fcoll, the fault includes the fault of two local controllers and LCMSC fault. Assuming the

probabilities of these two types of fault are p1 and p2, respectively, then fcoll = 1-(1-p1)(1-p2 ). Kandoo [22] also proposed a multiple-controller mechanism that the communication between

the controllers in different domains must resort to higher-layer controllers. With the extending network scale, the controllers logically form a multiple-layer controller model. For tcoll, we assume the average processing delay in each layer is tproc, and the transmission delay between two layers is ttrans. For n+1 layers controller model, if a controller has all information for decision in the best case and the cooperation is unnecessary, tcoll is thus 0. In the worst case, two controllers in the

13

ACCEPTED MANUSCRIPT

lowest layer have to implement cooperation via the root controller, and then tcoll = 2n(tproc + ttrans). So the average value of tcoll = n(tproc + ttrans). Obviously, the increase of the network scale and the layer number will lead to the augment of cooperation delay, and have negative influence on network scalability. Contrasted with LCMSC, tcoll is a small constant. For scoll, the controller states are kept in each controller, and LCMSC only maintains the basic information of controller connections, so scoll is independent of the number of the controllers involved. Now, let‟s consider a single layer with m controllers, each has u states. The upper

CR IP T

controller of these m controllers must maintain (m!u!) states, so it is very difficult to synchronize such a large number of states.

For fcoll, once the ith layer controller has fault (assuming its fault probability is pi), all controllers below ith layer will be impacted. Suppose that faults are independent, the fault probability in nth layer is 1 − ∏𝑛−1 𝑖=0 (1 − 𝑝𝑖 ). Therefore, multiple-layer controller model has much

AN US

higher fault probability.

Based on the above analysis, we design the controller architecture to a plain one. This is reasonable since the collaboration processes each is related to only a very small number of all the controllers and is independent of one another.

M

6. Experiments and Analysis

ED

In order to validate the feasibility of LCMSC mechanism, a LCMSC prototype was implemented and tested in practical network environment.

PT

6.1 Experimental Scenario

CE

An experiment environment is constructed as shown in Fig 6, which is composed of 3 SDN subnets and an IP network, to test the LCMSC prototype. The IP routers are Huawei S3700, OpenFlow switches are Pica8, and APs are Cisco AIR-AP1151A. The end systems are DELL

AC

Vostro 270-D698-JN, and OS is CentOS v6.3. Three types of controllers are pox-carp, NOX and RYU respectively, on which LCMSCware is running, that reflects LCMSC mechanism has the controller-independence property.

14

ACCEPTED MANUSCRIPT

Hs IP: 192.168.1.1/24

D1

Hd1 IP:192.168.2.1/24

D2 C2

OFS-A1

C1

DCore

OFS-B1

Core network

OFS-B3

OFS-A3

R2

R1 OFS-A2

Key:

OFS-B2

AP2

C3

Controller

R3 OFS-C3

IP router OpenFlow switch

OFS-C1

Mobile

OFS-C2

D3

Hd2 P:192.168.3.1/24

CR IP T

End system

AP3

Fig 6. The experimental environment of LCMSC prototype

We evaluate the performance of LCMSC in terms of transparency for traditional applications

AN US

and compatibility to emerging applications.

6.2 The Transparency for Traditional Internet Applications This experiment evaluates that, in the duel space environment, LCMSC can support traditional applications transparently when the security requirements are satisfied and the

M

resources are sufficient.

A web server is operated on Hs in D1, and a browser is used on Hd1 in D2. The browser can

ED

access the web server normally without modifying any programs and network facilities. The measurement result demonstrates that LCMSC extracts the information from the first packet of HTTP flow, and automatically fills in the cooperation mode with (unicast, {IPC1, IPC2}) based on

PT

socket and well-known port number. After C1 cooperating with C2, the end-to-end path from Hs to Hd1 is established with LCMSC, so that HTTP flow could pass through the dual space network.

CE

For e-mail and the other applications, the experiments have similar results. UDP ping server and client programs are run on Hs and Hd1 respectively to measure the delay

generated by LCMSC. Fig 7 illustrates the RTT CDFs of the first packet of the UDP flows using

AC

LCMSC and using hard-coded method. First packet delay in Openflow networks can denote the controller‟s path-building overhead. We can see that there exists no distinct difference between these two methods, this means LCMSC would bring little delay while providing convenience.

15

ACCEPTED MANUSCRIPT

Fig 7. CDF comparison of the first packets‟ RTT with and without LCMSC

CR IP T

Once the first packet of the flows triggers the controller cooperation mechanism to establish an end-to-end path, the subsequent packets have much lower RTT. Fig 8 shows the CDF of the RTT values of the non-first packets. The length of the measured packets are set to 64, 128, …, 1360 bytes. We also tested the way by using LCMSC and the hard-coded method. As expected, we get little difference. (Hard-coded result not shown in fig) And we found there‟s only slight

M

AN US

differences in RTT when packet length is above 128 bytes.

Fig 8. CDF of the following packets‟ RTT

ED

We measure the throughput of the end-to-end system in the Gigabit wired linkage environment by using Iperf [25] which is a commonly used tool to measure TCP/UDP bandwidth.

PT

Fig 9 shows the throughput of TCP flow and UDP flow. UDP has better performance than TCP

AC

CE

and both can achieve high throughput more than 670Mbps.

Fig 9. The throughput of UDP/TCP Summary: (1) LCMSC can support traditional C/S application transparently in duel space networks, properly inheriting the applications and the infrastructure of the Internet. (2) LCMSC has only little influence on the first packet of a flow (delayed about 200ms) and the following packets (delayed about 1.5ms). (3) LCMSC can properly support C/S and P2P applications.

16

ACCEPTED MANUSCRIPT

6.3 Security and Performance Advantages of LCMSC SDN has intrinsic security that every controller can fully control the establishment of each flow. In this test, we verify that this important character is maintained. To evaluate the security and performance of LCMSC, we enforce different policies in controller C1 in domain D1 and controller C2 in domain D2, add 200Mbps background traffic and then deliver an UDP flow AppUDP from Hs in domain D1 to Hd1 in domain D2.

CR IP T

When LCMSC is used between C1 and C2 in CoCtlerSet, and the 4-tuple in (1) can be satisfied, the path PathUDP can be properly set up. Otherwise, PathUDP can‟t be built and no network resource is wasted or incur security risk.

When LCMSC is not used between C1 and C2, and C1 granted the security or resource of D1 but C2 denied the security or resource of D2 for the security or resource reasons, PathUDP only

AN US

includes D1 part and Dcore part. In this case, AppUDP would cause serious waste in network

ED

M

resources and may bring security risks to D1 and Dcore (as shown in Fig 10).

AC

CE

PT

(a) The network traffic in D1

(b) The network traffic in Dcore

17

CR IP T

ACCEPTED MANUSCRIPT

(c) The network traffic in D2

Fig 10. The effect on the networks with LCMSC

Summary: (1) LCMSC retains the flexibility and security of SDN. (2) By using LCMSC to collaborate between controllers in advance can greatly reduce unnecessary traffic being injected

AN US

into the core transportation network.

6.4 The Support for Emerging Applications

This subsection examines how LCMSC mechanism supports emerging applications such as

M

mobile communication.

To illustrate that LCMSC can be used by the controllers to conveniently support moving

ED

terminal to communicate with a server in a different SDN domain and monitor the changing of the communication bandwidth, firstly we use Iperf [25] at a mobile device to measure the bandwidth it can reach during its moving from an area to another. And then using a media-play client at the

PT

mobile device to connect a remote streaming service, the bandwidth usage is monitored as well as user experience (UE) is evaluated.

CE

Assume an Iperf server is run in Hs, and its monitor port is 10086. At the same time, a mobile

host Hd2 run Iperf client and accesses Hs via TCP connection. During the connection, Hd2 is

AC

continuously moved between AP2 and AP3. Combining the software process with our measurement analysis, the following processes were validated. When C3 senses the client from Hd2 and parses cooperation parameters, coordinates with C1 by invoking LCMSC APIs, a TCP connection is established via AP3. When Hd2 switches off between AP2 and AP3, C2 coordinates with C1 and C3 by invoking LCMSC APIs respectively, and correlative routing tables need to be modified. Whether the TCP connection is kept or not is checked, and also the connection bandwidth is recorded.

18

ACCEPTED MANUSCRIPT

Bandwidth(Mbits/Sec)

25 20 15 10 AP2-AP3

5

AP3-AP2 0 0

5

10 15 Time(Sec)

20

25

Fig 11. The change of bandwidth

CR IP T

Fig 11 shows the bandwidth changes between Hs and Hd2 as Hd2 switches off between AP2 and AP3. The curve with circular sign indicates the change of bandwidth when Hd2 moves from AP2 to AP3, and the curve with rhombic sign shows the change of bandwidth when Hd2 moves from AP3 to AP2. It can be found that although the handoff time between AP3 and AP3 is about 1.5 seconds, the TCP connection is kept. Furthermore, the bandwidth is about 22Mbps when Hd2 is

AN US

associated with AP3 and 12Mbps when Hd2 is associated with AP2, respectively.

We use VLC [26] not only as a media-play client at the mobile host but also a streaming server in another SDN area. The mobile host access the server‟s streaming service. As the mobile host moves, the AP it was connected to must be switched. Fig 12 shows the bit rate notified by the

CE

PT

ED

M

VLC client.

Fig 12. Access VLC streaming service

AC

The stream is a VBR (variable bit rate) video. AP switch occurred at around the 12th second.

After connecting the server, VLC client lagged for a while and buffered some data. When the AP switch occur, no obvious delay can be observed. This may be improved by the former buffered data. Summary: (1) LCMSCware can take advantage of SDN and LCMSC APIs to support emerging applications with dedicated functions or complex collaboration. This makes software development much easier. (2) There‟s no interruption to TCP sessions when using LCMSC. What‟s more, LCMSC is efficient and scalable. These experiment results show us LCMSC can support emerging applications by invoking 19

ACCEPTED MANUSCRIPT

LCMSC APIs and adding some software defined logic.

7. Conclusions The theoretical analysis and experimental results demonstrate that LCMSC mechanism have advantages such as application-independence, controller-independence, function minimization, and reliable communication. Under the environment of multiple SDN controllers connected by IP

CR IP T

network, LCMSC can easily support traditional and emerging network services. Our future works focus on extending LCMSC mechanism in other mainstream controllers and developing a universal way to support more network applications.

Using SDN to improve the Internet control and management capabilities, it‟s of high priority to solve the basic problem of how to use the east/west bound to coordinate the SDN controllers.

AN US

Rather than pursuing perfect and comprehensive goal, LCMSC proposed in this paper a lightweight controller cooperation mechanism, which is simple, efficient, with limited functions. It aimed for inheriting the Internet software and hardware resources, providing support for emerging applications and incremental deployment. Theoretical and experimental results show that LCMSC has the characteristics of application independent, controller independent, minimum functions and

M

reliable communication. LCMSC can support traditional IP services and novel services by coordinating multiple SDN domains connected by an IP network. This has important theoretical

ED

and practical value. The next step is to improve LCMSC to provide stable and reliable support for a variety of mainstream controllers, and to study the new APIs to support a wider variety of new

PT

network applications.

Acknowledgements：This paper is supported by the State Key Development Program for Basic Research of China

CE

under Grant No. 2012CB315806, the National Natural Science Foundation of China under Grant No.61379149, and Jiangsu Province Science & Technology Plan under Grant No.BY2013095-1-06.

AC

References

[1] G. Natasha, et al., NOX: Towards an Operating System for Networks, CCR 2008 [2] S. Jain, et al., B4: Experience with a globally-deployed software defined WAN. In SIGCOMM, 2013. [3] H. Yin, et al., SDNi: A Message Exchange Protocol for Software Defined Networks across Multiple Domains,Internet Draft, Internet Engineering Task Force, June 2012. [4] Gupta, D., and Jahan, R. Inter-SDN Controller Communication: Using Border Gateway

20

ACCEPTED MANUSCRIPT

Protocol. Tata Consultancy Services White Paper, 2014. http://www.tcs.com [5] Kreutz, D., et al, Software-Defined Networking: A Comprehensive Survey. Proceedings of the IEEE, January 2015 [6] W. Stallings, Software-defined networks and OpenFlow, The Internet Protocol Journal, vol. 16, no. 1, 2013. [7] T. Koponen, et al., Onix: a distributed control platform for large-scale production networks, Proceedings of the 9th USENIX conference on Operating systems design and implementation,

CR IP T

ser. OSDI‟10. Berkeley, CA, USA: USENIX Association, 2010, pp. 1–6

[8] A. Doria, et al., Forwarding and Control Element Separation (ForCES) Protocol Specification, Internet Engineering Task Force, Mar. 2010.

[9] Z. Wang, T. Tsou, J. Huang, X. Shi, and X. Yin, “Analysis of Comparisons between OpenFlow and ForCES,” Internet Draft, Internet Engineering Task Force, December 2011.

AN US

[10] K. Ogawa, W. M. Wang, E. Haleplidis, and J. H. Salim, ForCES Intra-NE High Availability, Internet Draft, Internet Engineering Task Force, October 2013.

[11] F. A. Botelho, F. M. V. Ramos, D. Kreutz, and A. N. Bessani, On the feasibility of a consistent and fault-tolerant data store for SDNs, Proceedings of the 2013 Second European Workshop on Software Defined Networks, ser. EWSDN ‟13. Washington, DC, USA: IEEE

M

Computer Society, 2013, pp. 38–43.

[12] S. Vinoski, Advanced message queuing protocol, IEEE Internet Computing, vol. 10, no. 6, pp.

ED

87–89, Nov. 2006.

[13] M. Canini, P. Kuznetsov, D. Levin, and S. Schmid, Software transactional networking: concurrent and consistent policy composition, Proceedings of the second ACM SIGCOMM

PT

workshop on Hot topics in software defined networking, ser. HotSDN ‟13. New York, NY, USA: ACM, 2013, pp. 1–6.

CE

[14] A. Ghodsi, Distributed k-ary system: Algorithms for distributed hash tables, Ph.D. dissertation, KTH-Royal Institute of Technology, 2006.

AC

[15] F. Botelho, A. Bessani, F. M. V. Ramos, and P. Ferreira, On the design of practical fault-tolerant SDN controllers, Third European Workshop on Software Defined Networks, 2014.

[16] ONF,

SDN

architecture,

June

2014.

https://www.opennetworking.org/images/stories/downloads/sdnresources/technical-reports/T R SDN ARCH 1.0 06062014.pdf [17] R. Hand and E. Keller, ClosedFlow: OpenFlow-like control over proprietary devices, Proceedings of the Third Workshop on Hot Topics in Software Defined Networking, ser. HotSDN ‟14. New York, NY, USA: ACM, 2014, pp. 7–12.

21

ACCEPTED MANUSCRIPT

[18] M. Jarschel, T. Zinner, T. Hoßfeld, P. Tran-Gia, and W. Kellerer, Interfaces, attributes, and use cases: A compass for SDN, IEEE Communications Magazine, vol. 52, no. 6, pp. 210–217, 2014. [19] J. Vasseur and J. L. Roux, “Path Computation Element (PCE) Communication Protocol (PCEP),” RFC 5440 (Proposed Standard), Internet Engineering Task Force, Mar. 2009. [20] E. Mannie, Generalized Multi-Protocol Label Switching (GMPLS) Architecture,” RFC 3945 (Proposed Standard), Internet Engineering Task Force, Oct. 2004, updated by RFC 6002.

CR IP T

[21] Marjory S. Blumenthal, David D. Clark, Rethinking the design of the Internet: The end to end arguments vs. the brave new world, ACM Transactions on Internet Technology, Vol 1, No 1, August 2001.

[22] S. Hassas Yeganeh and Y. Ganjali, “Kandoo: A framework for efficient and scalable offloading of control applications,” Proceedings of the First Workshop on Hot Topics in

AN US

Software Defined Networks, ser. HotSDN ‟12. New York, NY, USA: ACM, 2012, pp. 19–24. [23] A. Tootoonchian and Y. Ganjali, HyperFlow: a distributed control plane for OpenFlow, Proceedings of the 2010 internet network management conference on Research on enterprise networking, ser. INM/WREN‟10. Berkeley, CA, USA: USENIX Association, 2010, pp. 3–3. [24] Shin M K, Nam K H, Kim H J. Software-defined networking (SDN): A reference architecture

M

and open APIs[C]//2012 International Conference on ICT Convergence (ICTC). IEEE, 2012: 360-361.

ED

[25] Tirumala A, Qin F, Dugan J, et al. Iperf: The TCP/UDP bandwidth measurement tool[J]. htt p://dast. nlanr. net/Projects, 2005. [26] http://www.videolan.org/vlc/

PT

[27] Vissicchio S, Tilmans O, Vanbever L, et al. Central control over distributed routing[J]. ACM SIGCOMM Computer Communication Review, 2015, 45(4): 43-56.

CE

[28] Caria M, Jukan A, Hoffmann M. SDN Partitioning: A Centralized Control Plane for Distributed Routing Protocols[J]. arXiv preprint arXiv:1604.04634, 2016.

AC

[29] Patil P, Gokhale A, Hakiri A. Bootstrapping Software Defined Network for flexible and dynamic control plane management[C]//Network Softwarization (NetSoft), 2015 1st IEEE Conference on. IEEE, 2015: 1-5.

22

ED

M

AN US

CR IP T

ACCEPTED MANUSCRIPT

PT

Ming Chen

was born in 1956. Professor and PhD supervisor of PLA University of Science and Technology,

CE

Nanjing, China. His research interests include network architecture, network performance analysis and modeling, network management and measurement, future Internet and softwore-defined

AC

networking.

Ke Ding

23

ACCEPTED MANUSCRIPT

was born in 1978. He is a PhD student of PLA University of Science and Technology, Nanjing, China. His research interests include network architecture, cloud computing and softwore-defined

CR IP T

networking.

Jie Hao

received her BS degree from Beijing University of Posts and Telecommunications, China, in 2007,

AN US

and the PhD degree from Uni- versity of Chinese Academy of Sciences, China, in 2014. From 2014 to 2015, she has worked as post-doctoral research fellow in the School of Computer Engineering, Nanyang Technological University, Singapore. She is currently a lecture at College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China.

AC

CE

PT

ED

M

Her research interests are wireless sensing, wireless sensor networks, VLC and etc.

24

AC

CE

PT

ED

M

AN US

CR IP T

ACCEPTED MANUSCRIPT

Chao Hu

received the Ph.D. degree from PLA University of Science and Technology, Nanjing, China, in 2013. He is currently an assistant-professor in the college of command information systems at PLA University of Science and Technology, Nanjing, China. His research interests include software-defined networking and distributed computing.

25

CR IP T

ACCEPTED MANUSCRIPT

AN US

Gaogang Xie

received the PhD degree in computer science from Hunan University, in 2002. He is the professor at ICT/CAS. His research interests include Software-Defined Networking, Internet architecture,

AC

CE

PT

ED

M

and Internet measurement.

Changyou Xing was born in 1982. Received his PhD degree from PLA University of Science and Technology, Nanjing, China in 2009. His research interests include network modeling, p2p multimedia communications and software-defined networking.

26