Context-aware RAON middleware for opportunistic network

Accepted Manuscript Context-aware RAON middleware for opportunistic network G. Lau, M. Al-Sabah, M. Jaseemuddin, H. Nejad, M. Bhuiyan

PII: DOI: Reference:

S1574-1192(16)30355-8 http://dx.doi.org/10.1016/j.pmcj.2017.07.004 PMCJ 870

To appear in:

Pervasive and Mobile Computing

Please cite this article as: G. Lau, M. Al-Sabah, M. Jaseemuddin, H. Nejad, M. Bhuiyan, Context-aware RAON middleware for opportunistic network, Pervasive and Mobile Computing (2017), http://dx.doi.org/10.1016/j.pmcj.2017.07.004 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Context-Aware RAON Middleware for Opportunistic Network G. Lau, M. Al-Sabah, M. Jaseemuddin, H. Nejad, and M. Bhuiyan Department of Electrical and Computer Engineering Ryerson University, 350 Victoria Street, Toronto, Canada M5B 2K3 {jaseem, m1bhuiyan}@ryerson.ca, [email protected], [email protected] Abstract Application development and deployment on Mobile Ad Hoc Networks (MANET) is a major challenge in the widespread use of MANET. D2D communication in 5G networks renewed interest in effective middleware design for MANET where application developers need to deal with unstable connectivity, high error rate, mobility induced disruption and disconnection, and limited battery power. We find that unstructured overlay network provides a good abstraction to facilitate application development and deployment on MANET. In this paper, we present the design of a middleware that builds Resource-Aware Overlay Network (RAON), which is an unstructured overlay network of nodes engaged in the application that employs a query-reply mechanism for resource discovery. RAON is enhanced with features such as proactive neighbor replacement, congestion-aware data download and cooperative caching. Simulation results show that these features are effective in reducing query delay, improving data availability, and balancing node power consumption with protocol performance. We also present the middleware software design that offers the API based on node and path abstractions to applications. The middleware implements a generic context framework for acquiring device and user context. We discuss the implementation of application-level multicast and credit-based file-sharing applications using the middleware API. The middleware is implemented in Java J2ME on Android, which is tested in an ad hoc network of Nexus 7 devices running OLSR. Key Words Middleware, ad hoc network, unstructured P2P overlay network, cooperative caching, context-aware, resource-aware, modified biased random walk, D2D communication, 5G networks.

1. Introduction With enormous availability, adoption and popularity of variety of mobile devices, the interest in developing pervasive applications that run on ad hoc network of mobile nodes is growing. Mobile Ad-Hoc Network (MANET) also forms opportunistic networks for devices that need ad hoc and temporary connectivity for a particular service delivery. D2D communication in 5G networks [40, 41] renewed interest in effective middleware design for MANET. The applications design on MANET requires cooperation of the nodes to perform distributed tasks and deals with common challenges of MANET, which can be effectively addressed through a middleware. Peer-to-Peer (P2P) overlay network is a promising approach for designing and deploying applications on MANET since it requires only a subset of nodes to maintain application related states while other nodes perform only underlying network function, such routing and packet forwarding [1]. It avoids significant overhead of maintaining application states on the non-application nodes that join the network for an unpredictable transient time. Thus, an application remains immune to the transience and instability of the nodes that are not engaged in its execution. Overlay design in MANET faces many challenges posed by link instability, node transience, intermittent disconnection, limited battery power, and node resource (CPU, memory, and storage) constraints. The peer nodes can be used as landmarks for path metrics such as reliability and throughput using MMHC approach [24] allowing applications to use the most suitable path. It also facilitates performing dynamic service composition based on distance and load estimation [25]. In this paper, we present the design of a middleware that builds Resource-Aware Overlay Network (RAON) [1] of nodes engaged in the application that employs a query-reply mechanism for resource discovery in an unstructured overlay network. It uses overlay abstraction to offer API for developing applications such as file-sharing, multicast and multimedia streaming, etc. It provides common functionalities that every application on an overlay network has to deal with, such as joining and leaving the network, searching for resources and data, downloading data from the source, and caching data in the network. It also provides features to deal with intermittent connectivity through the underlying network and power conservation. Finally, it gives application developers great flexibility by hiding the lower

layer implementation details. We implemented RAON in ns-2 simulator and in Java on Android platform. P2P overlay networks are distinguished by how they are constructed and how do they perform the file search.

They can be classified as unstructured and structured.

In

unstructured overlay networks such as Freenet [9], Gnutella [11], Gia [4] and Bittorrent [10], nodes form no specific topology rather they are organized in random graphs and flood the network with the query during the file search operation. Searching for resources in unstructured overlay networks can be centralized or distributed.

Napster [12] and

Bittorrent are centralized unstructured overlay networks where search operation is centralized while file downloads occur in a distributed manner. The centralized search networks are less scalable and suffer from single-point-of-failure problems. To solve these problems, distributed-search overlay architectures, such as Freenet, Gnutella and Gia, are introduced. Some decentralized systems use TTL-controlled flooding search while other systems, such as Gia, use biased random walk to route queries and locate resources.

In

contrast, structured P2P networks, such as CAN [6], Chord [7], Pastry [5], and Tapestry [8], often use distributed hash tables to locate directly the nodes storing the file pointers. Existing P2P systems are designed for stationary nodes that have reliable network connectivity, high computing power, and unlimited power supply. Unfortunately, none of these assumptions holds good for MANET nodes. The question of which P2P architecture is more suitable for MANET requires further analysis. In structured P2P networks, a node stores index for the files located at other nodes in the network regardless of its distance with those nodes. If the node leaves the network, an expensive operation is invoked to relocate its file indices to another node. Frequent disruption of connectivity in MANET would require full redundancy or expensive relocation of file indices on a structured P2P network. Furthermore, unstable links that cause transient disconnection may trigger expensive relocation of file indices to more stable nodes. In contrast, unstructured P2P networks do not need to relocate or force redundancy of file indices for dealing with most common cases of transient and permanent disruptions. Hence, we consider unstructured P2P overlay for RAON. In this paper, we present and evaluate the design of RAON middleware that addresses above issues. We discuss in section 2, RAON along with its features including proactive

neighbor replacement and congestion-aware download. In section 3, we discuss the design of co-operative caching for RAON, followed by performance analysis of RAON and its features in section 4. In section 5, we discuss the middleware software design with its subsystems and context framework. Finally, we conclude the paper in section 6. 2. Resource-Aware Overlay Network (RAON) The RAON builds unstructured overlay network based on Gia [4], while it retains main design features of Gia but modifies the biased random walk algorithm to incorporate dynamic factors in forwarding a query such as link instability and node power constraints. Gia [4] is an unstructured P2P network that uses the capacity of a node to perform topology adaptation, flow control, one-hop replication, and biased random walk. The capacity of a node is defined as the number of queries it can handle within a period of time, which is mainly determined by static factors in wired networks such as CPU speed, memory, and bandwidth.

Topology adaptation ensures that only high capacity nodes are highly

connected. The flow control mechanism uses tokens to limit the number of queries a node can receive from its neighbor that is determined by its capacity. One-hop replication makes a node replicate the file pointers of its neighbors. The biased random walk scheme favors query forwarding to high capacity nodes, which significantly reduces the number of queries and consequently the number of messages generated during flooding (broadcast). The biased random walk is shown to significantly improve the query success rate (by increasing the probability of query hit) when compared to pure random walk. Although Gia is not designed by considering wireless networks, the above features are useful for its formation on MANET. The performance of Gia is sensitive to the factors that define node capacity, which does not consider power constraints of nodes and instability of their access links. It engenders two problems when Gia is formed on MANET. First, unlike flooding that simultaneously forwards a query along multiple links; biased random walk forwards only one copy of a query at a time, which if sent over an unstable link would increase the query response time due to excessive delay or loss. The average response time will increase significantly if the link to a high capacity node becomes unstable. Since link instability is common in MANET due to mobility of devices, the biased random walk can hit performance snag. Second, a high capacity node tends to receive and process more queries, which drains its battery power faster and eventually leads to its death. Hence, forwarding

large number of queries to a high capacity node without considering its power may result in node failure and consequently query losses. The RAON addresses above issues by redefining capacity of the node and modifying biased random walk to make it a function of link stability and node’s residual power. Unlike Gia, RAON defines the capacity of a node to be a static function of CPU speed and memory and excludes bandwidth that varies with the radio channel condition in MANET. It introduces a ranking function for a node to classify its link with its neighbor according to the residual power of the neighbor (Pr) and the link stability (Ls). It defines a Link Coloring Scheme (LCS) for a node to color links to its neighbors based on a function defined in Table 1(a). The link colors are ranked as GREEN > YELLOW > RED. The modified biased random walk (MBR) algorithm, as shown in Table 1(b), employs link color in addition to the neighbor’s capacity for forwarding a query to the neighbor.

Link Color

Condition

GREEN

Pr=HIGH && Ls=LOW

YELLOW

(Pr = MEDIUM && Ls != HIGH) || (Ls = MEDIUM && Pr != LOW)

RED

Pr=LOW || Ls=HIGH

Algorithm MBR: Node x receives a query with qid and determines the next hop for the query. Coli  color of the link to neighbor i Capi  capacity of neighbor i ncolor ← RED; ncapacity ← 0; nnode ← x For every neighbor i such that x has tokens from i and it has not forwarded query with qid to i before { If (Coli > ncolor) or ((Coli == ncolor) and (Capi > ncapacity)) Then { ncolor ← Coli; ncapacity ← Capi; nnode ← i }} return (nnode)

Table 1: (a) Link Color Function (b) MBR Algorithm

In RAON, the LCS parameters, Pr and Ls, are measured periodically, hence color of a link varies with measurement window. RAON defines two thresholds (PHigh and PLow) for Pr and two thresholds (LHigh and LLow) for Ls. The measures of Pr and Ls may be gleaned from the routing metric of the underlying MANET routing protocol. For example, path stability in MANET routing based on routing metrics, such as hop count, route repairs, or ETX [29], can be used as a measure of link stability for RAON because a link in RAON (overlay link) is a path in MANET. In case a MANET routing metric does not measure Pr or Ls, a node measures the missing parameter by sending a probe to the neighbor. For example, it measures the stability of a link to its neighbor as a function of the round trip time (RTT) on the link by computing the response time of the probe message. It also receives the residual power level (energy) of its neighbor (Pr) in the probe message. The LCS colors a link based on RTT and the neighbor’s energy. For example, the color (rank) of a link is lowered if either the RTT is high or the energy level is low. The probe is a control message that can be piggybacked in another control or data message to reduce the bandwidth overhead. The periodicity of the probe message should consider: (i) overhead in processing the message that determines no or small change in energy, and (ii) varied maximum power of nodes due to different battery sizes. To reduce the probing overhead, a node defines three energy states: HIGH, MEDIUM, and LOW. It monitors its own power level and determines the energy state relative to its battery size. It sends Pr update to its neighbors only when a state transition occurs. The MBR algorithm described above outlines the steps a node takes in determining the next hop for a query. The algorithm uses the link color marked by LCS to evaluate a link for the forwarding decision. It avoids unstable links (to improve query search performance) and low-energy nodes (to increase the battery life of the neighbors). It also employs Gia’s flow control mechanism to further control the number of queries a node sends to its neighbor. It designates a neighbor node to be available if the node has tokens for the neighbor. It then selects an available node with the highest color and capacity. It gives color precedence over the capacity. For example, it would select a neighbor with higher color and lower capacity over the neighbor with lower color and higher capacity. The flow control mechanism at a node can assign tokens to a neighbor proportional to the color of the link to the neighbor. For example, it can assign least tokens to a neighbor

reachable through a RED link. For token assignment, it can distinguish a RED link due to low power from the RED link due to low link stability. A low power node can also use the link coloring scheme to extend its battery life. In a P2P network, file download typically consumes more power than query processing. A node that runs low on energy can take the following power conservation measures. First, it adopts one of the following approaches to deal with the download request: (i) it denies the download requests, or (ii) queues the requests and serves them when it regains the energy, or (iii) it responds with a list of nodes that also stores the file, which are the nodes that have previously downloaded the files. Second, it can reduce the number of tokens it distributes to its neighbors. Finally, it can reduce the number of queries and control messages it receives for processing by dropping some of the neighbors. It can use the link color for choosing a neighbor to drop, for example, it drops a neighbor accessible through a RED link first. However, each node must keep a minimum number of neighbors (e.g. two) to avoid network partitioning. 2.1. Pro-active Neighbor Replacement We augmented RAON with Proactive Neighbor Replacement (PNR) mechanism that further improves the overall performance of LCS by changing the overlay topology adaptively [2]. If a node n measures the link stability to be Low of the overlay link l with a neighbor g, it tries to form a new neighbor relationship with an overlay node that is not currently a neighbor and is reachable through a low latency MANET route. There are two major steps involved in PNR: 1) node discovery to search for a neighbor candidate; and 2) establish a new overlay link. We discuss below three node discovery methods that return a list of neighbor candidates to n. Then, we discuss the process of selecting a neighbor to establish a new link. Ring Search: The node n broadcasts a search message to all the nodes within its transmission range. If a receiving node is a RAON node, it replies to n; otherwise, it expands the ring search by broadcasting the message to all the nodes in its transmission range. The message is propagated in the network until the TTL expires. Initially n can include the list of known neighbors, so that a node does not reply to n if it is in the list. When the search completes, n can form the candidate list with the nodes that have sent the replies.

Ping-Pong: The node n sends a PING message to its neighbor that is connected through a link with highest stability. It includes a list of known neighbors in the PING message. If the receiving node finds a node x among its neighbors that is not in the neighbors list, then it sends a PONG message with the information about x back to n via the reverse route. If it does not find such a node, then it forwards the PING to one of its neighbors connected through a link with highest stability. The PING message is propagated in the network until its TTL expires. The probability of finding a node that n can establish a stable/low latency connection increases by forwarding the PING through links with highest stability. At the end of the process, n forms the candidate list with the information it receives in PONG messages. Location Server: If all the overlay nodes are equipped with GPS (or a localization support) and cellular interfaces, they can periodically report their location, velocity and direction to a location server. When a node n needs to discover a new RAON node, it can send a request to the server with its location information and list of its neighbors. The server prepares a list of nodes that are physically close to n and sends the list in its reply to n. The server can also include nodes that are moving towards n and/or exclude those that are moving away from n. Although physical proximity does not necessarily guarantee stable connectivity, the intent here is to hope that n would be able to find at least one node with that it can establish a stable connection. The objective of the selection process is to find a node from the candidate list that is connected through a path of highest stability. It requires n to first probe every node in the list and then determine the best among them. Probing a non-neighbor node requires n to establish a TCP connection that may trigger the underlying layer to find a new route. This process may consume substantial amount of bandwidth and power, which can potentially defeat our aim to conserve those resources. Hence, instead of probing all the candidates, n may order the candidate list based on certain criterion, for instance in the descending order of node capacity. It then begins probing from the highest capacity node and stops at the node that can be connected through a path whose latency is below certain threshold. It may use different criterion supported by the search mechanism under use. For example, in case of Ring Search the first non-neighbor node that responds the probe is practically the one with the smallest delay, so n can select the candidate in the same sequence as it receives the

replies. In case of location server, it can return the candidate list sorted in ascending order of the distance, which n can use to establish connection one by one. Once n finds a suitable candidate it sends overlay connection request to the candidate. If the candidate accepts the connection request, n closes its connection with g and establishes overlay link with the candidate. If the connection request is rejected, n probes another candidate from the list and repeats the above procedure. Although this method might not find the best candidate, it avoids selecting one that is unacceptable in terms of performance. When PNR is used for replacing multiple neighbors concurrently, the candidate list formed for one neighbor replacement can be used for the subsequent replacements. In case that no candidate is found in the node discovery process or all the candidates are connected through paths with low stability, then n continues its neighbor relationship with g while the link l is colored as RED. The node n periodically probes all neighbors that are connected through RED links and if finds the links to remain RED then invokes PNR. Both ring search and location server methods do not require any knowledge of the RAON network, thus they can also be used for bootstrapping. However, they are both expensive operations – ring search floods the network, and location server incurs new service charges. In contrast, the Ping-Pong method incurs low overhead because it requires establishing no new connection to discover candidate neighbors. Therefore, we suggest using Ping-Pong for node discovery if n is already connected to at least one peer. For bootstrapping, location server is preferred over ring search because flooding for ring search incurs a recurring cost of consuming bandwidth and power of all the nodes within the TTL range. In contrast, the service cost of accessing location server at n can be amortized over subsequent invocation of PNR with marginal expense of resources of other nodes. The PNR carries a significant overhead in terms of the costs of messaging, computation and connection. Hence, it should be invoked judiciously when the benefit offsets the cost. The low link stability causing the link to be colored RED may be a transient occurrence due to link congestion or temporary link failure. Hence, more reliable indicator of link instability should be used as a measure of invoking PNR. For example, if the link shows high latency trend over several measurement windows, or the neighbor’s speed and direction indicate the widening distance between the node and its neighbor. 2.2. Energy-aware Topology Adaptation and Flow Control

In RAON, we consider link instability as the only trigger for PNR and not the low energy state of the neighbor. When a node is running low on energy it is in the best position to adopt one or more measures to rectify the situation instead of simply its neighbor invoking PNR, which we argue may not be that effective to improve the overall query performance. There are three measures available to a low energy node to improve its power level. First, since the file download consumes more power than query processing, it can disable the file download requests selectively until the power level improves, as discussed above. Second, it can invoke the flow control to attract less number of queries by reducing the number of tokens assigned to its neighbors. As a complementary step, the neighbor can reduce its tokens assigned to the node so that the node will expend less energy in forwarding the query. Third, it can invoke the topology adaptation by dropping some of its overlay links. However, the node should keep a minimum number of neighbors (e.g. two) to avoid network partitioning. 2.3. Congestion-aware File Download The file download traffic in RAON deteriorates the query success rate and delays. We measured packet losses, duplicate packets and RTT of the overlay links to trace the source of the performance degradation [13]. All those measures are high during the download period. We further observed that the number of unstable links and their use for query forwarding increased. These measurements lead us to believe that: (1) the download traffic causes congestion in the network, (2) the download paths intersect with the paths that form overlay links causing more links to become unstable, and (3) RAON query forwarding fails to avoid unstable links. The challenges of dealing with network congestion at the overlay level are due to lack of precise congestion indicator and lack of control on managing congestion. However, the source of the traffic causing congestion can be controlled at the overlay level. The following two modifications to RAON improves the quality of query forwarding decision, and mitigates the effect of download traffic in causing congestion. 1. Local Feedback: The MBR algorithm in RAON only considers the quality of links to the next hop neighbors and tends to avoid using unstable links. We modified the MBR algorithm by a feedback from the neighbor to the forwarding node if the neighbor is unable to forward the query over a stable link. The forwarding node then duplicates the query and

forwards to a new neighbor. With this local feedback, the forwarding decision of a node is also affected by the quality of links one-hop away from the node, which is expected to increase the use of stable links in query forwarding. Side effect of this feedback is replication queries in the network. 2. Download Site Selection: The congestion due to download traffic can affect the query performance if it intersects the query-forwarding path. We developed site selection algorithm to avoid using unstable links at the intersection of query and download paths. We introduce a query color scheme. A query becomes RED if it traverses at least one RED link. The response of a query returns the color of the corresponding query. Thus, a RED query generates RED response, and so on. The querier (query source) node avoids downloading from a site that has generated RED response.

Figure 1: Query Forwarding and Download Site Selection in RAON Figure 1 explains the modified query forwarding and download site selection process. Node A is the querier that generates a GREEN query. When the query arrives at node B, it selects node E as the next hop for the query, because E is accessible through a GREEN link

that is also the most highly connected neighbor. The node E changes the color of the query to RED when it forwards the query over a RED link to I, because all the neighbors of E are connected through RED links. After forwarding the query on a RED link, E sends the local feedback to B about the use of unstable link. The node B after receiving local feedback from E decides to send a copy of the query over a GREEN link to F, which does not generate any local feedback because it can forward the query over a GREEN link to G. Thus, a copy of the query arrives at both L and J. The node L receives a RED query hence it generates the RED Response, whereas J generates a GREEN response since it receives a GREEN query. The querier node A selects and downloads file X from J. 3. Co-operative Caching in RAON The Cooperative Caching in RAON [23] is a hybrid scheme that exploits the strengths of both caching data (CacheData) [34] and caching path information to a node serving data (CachePath) [34] using a demand-supply framework [35]. We chose cooperative caching approach because in cooperative caching nodes make local caching decisions in collaboration with other nodes in the network that leads to optimal decision based on both local and global parameters [36 – 38]. It modifies the MBR at intermediate nodes to make two decisions: (a) CachePath decision after receiving query reply from a destination (supplier) for data item di, and (b) CacheData decision when it receives Down Load (DW) request. Both decisions are based on estimates of demand and supply of the data item in the network. The query processing at an intermediate node is modified to generate query response if it either finds the data item in its data cache or the path to a query destination in its path cache. Demand for a data item can be estimated at a query destination node. The query contains a demand vector, which is the list of queriers that are sources of outstanding queries for the data item known to intermediate nodes along the query path. When an intermediate node n receives a query for data item di, it appends the list of queriers of outstanding queries for di before forwarding the query to the next hop. These are the queriers from which n has received their queries for di but has not yet seen the replies. When a query destination that is ready to serve di receives the query, it calculates the demand estimate as the number of queriers in the demand vector and puts this information in the query reply. It also includes its residual power (Pr), and expiry time (Te) of di that indicates the time until that the data is

available at the destination, which is especially important if the destination is caching the data. The intermediate nodes along the reverse path receive the demand estimate and make their CachePath decisions as described below. An intermediate node decrements demand estimate by one when it receives a DW request, because DW request shows demand fulfillment of an erstwhile querier. The number of suppliers for a data item can be estimated by keeping track of the potential caches in the network, which can be computed along each download path. The download DW request contains supplier vector that is the list of potential caches along the download path. Every intermediate node maintains a local running count of supplier estimate for di, which is the number of potential caches it knows within a time window tw. A node expresses its ability to cache di by including its node-id and expected cache expiry time te in the supplier vector. When a querier decides to download di, it generates DW request and includes its node-id and te in the supplier vector if it wants to cache and serve di. When an intermediate node receives a DW request, it expresses its intent of caching the data by adding its node-ID and te in the supplier vector of the DW request. The destination serving the DW request returns the supplier vector and di in the download DW reply packet. It also includes its own node-id and te in the supplier vector. An intermediate node along the DW reply path updates it supplier estimate = supplier estimate + number of potential caches in the supplier vector. The supplier estimate is a step function of time as shown in Figure 3. It increases with DW request and decreases when a te elapses. The intermediate nodes make following CachePath and CachData decisions: Cache the Data Path (CachePath): When an intermediate node receives a query reply it makes a decision of keeping the path information (next hop) to the destination node (supplier) if Pr > PHigh of the destination node. It associates expiry time to the path information so that it can forward subsequent queries along that path until the expiry time ends. It sets expiry time to te that it receives from the destination. Cache the Data (CacheData): When an intermediate node receives a download request for data item di, it adds its (node-id, te) in the demand vector of the DW request if it intends to cache di. It employs a heuristic such as demand estimate > supplier estimate and its Pr > PHigh. The cache expiry time it computes as ((Pr – PLow)/α), which is an estimate of the time it takes Pr to reach below PLow given power depletion rate α at the node. When it receives

DW reply message, then it i makes thhe CacheDaata decisionn as it has better estim mate of supplliers in the nnetwork andd caches thee data arrived in the reply packet. Figuure 2 showss a network where X, Y and Z are iintermediatees nodes, Si is a querierr and Dj is a query desstination, annd all three queries, q Q1 ((S1, D2), Q2 (S2, D2), annd Q3 (S3, D3) are Node X is loccated at the intersectionn of two queery paths, Q2 and launcched for dataa item di. N Q3, node n Y lies oon a single qquery path Q2, and Z is located on the paths off two queriees Q1 and Q2. The dem mand estimatte at an inteermediate noode dependss upon the oorder of the qquery arrivaal. For exam mple, if X reeceives Q3 bbefore Q2, thhen the dem mand estimatte at X and Y will be tw wo at tq2 (arrrival time off Q2), whereeas if the ordder of queryy arrival is reeversed, theen X and Y know onlyy one demannd (from S2) at tq2. Sincce Y only reeceives Q2 fr from X, its ddemand estim mate remainss one at a latter time tq3 w when X learrns an additional demannd through Q3. Figurre 3 shows tthe evolution of demandd estimate and a supplierr estimate fuunctions for data item di with timee at Z. In thee figure, tQ1 and tQ2 are the Q1 andd Q2 arrival times; tDWRRep1 and S1 and S2; aand teD2, teS2 e the tDWReep2 are the arrrival time oof DW Rep packets to S S and tez are cachee expiry tim mes at D2, S22 and node Z respectiveely. Node Z incrementss its demandd estim mate every tiime it receivves query foor the di and decrementss it by one w when it receeives downnload reply m message. Thhe supplier vector in the downloadd reply packets are: DWR Rep1<(Z, teZ), (D2, teD2)> and DWR Rep2<(Z, teeZ), (D2, teDD2), (S2, teS2))>. The cachhe expirry times are ordered as tteD2 < teS2 < teZ. Node Z decrementts supplier eestimate eveery time a suppplier cache time expirees that show ws that di is nno longer avvailable at thhe supplier.

Figure 2: Cache Patth (1) and Cache C Data (2) ( decisionss.

Figure 3: Demand Estimate and Supplier Estimate step functions.

4. Numerical Results and Discussion We simulated both Gia and RAON to evaluate their performance. We first discuss the simulation results of both RAON and RAON with PNR. We then discuss the simulation results of cooperative caching support for RAON.

4.1 Simulation of RAON We simulated Gia, RAON, and (RAON – PNR) using the Network Simulator NS2 [30] with peer-to-peer extension developed at Georgia Tech, where RAON includes all three features namely LCS, MBR, and PNR whereas (RAON – PNR) is RAON without PNR. Our simulation objective is to study the behavior of RAON and Gia and evaluate their performances under different system configurations and network conditions.

We

implemented mobile nodes equipped with the IEEE 802.11b wireless interface with the transmission range of approximately 250m. We used the energy model provided by NS2 to model the node’s energy consumption and we set all nodes to start with the same initial energy level. We used random waypoint mobility model, where each node chooses a destination randomly within the simulation area using a uniform random speed between 0 and maximum speed. When the node reaches the destination, it pauses for 30 seconds and then move to another destination.

We generated different MANET scenarios by varying the node density and the maximum speed of mobile nodes. We consider 2m/s, 5m/s, and 20m/s to model the movements of pedestrians, non-motorized vehicles, and motor vehicles respectively. We keep the number of mobile nodes constant in two different areas: (500m x 500m) and (1000m x 1000m), to simulate dense and sparse node distribution respectively. We use the notation maximumspeed:dimension to refer to a scenario as a combination of speed and dimension, where dimension is the dimension of the simulation area (i.e. 500 and 1000). Thus, we generated six scenarios for all combinations of speed and dimension. For example, 2:500, 5:500, 20:500, 2:1000, 5:1000, 20:1000 are scenarios 1 to 6 respectively. Each simulation run lasts 1000 seconds. The simulation begins with a defined MANET topology where the nodes are randomly placed over the simulation area. We used AODV as the underlying routing protocol in MANET [3]. Each overlay node in the P2P network uses a uniform random number between [0, simulation-time/2] to determine when to join the network. Therefore, no P2P topology is defined in the beginning of the simulation. A virtual bootstrap server contains a complete list of existing P2P nodes, so new nodes can learn about other P2P nodes by contacting it. After a node is connected to the overlay network, it periodically exchanges with its neighbors update messages every update interval.

Since the AODV

implementation in NS2 sets the routing cache expiry time to be 60 seconds, we set the update interval to be 30 seconds. Queries are generated at a minimum rate of 1 query per 10 seconds to model the behavior of aggressive P2P users. Each query contains a keyword that is represented by an integer between [0, 99], and every keyword is mapped to a set of files. The files located on each node are generated randomly. We set the same capacity level, number of files and their sizes for each node. Table 2 summarizes all the P2P related simulation parameters. Parameters Number of peers Start Time Query Generation Interval No. of files stored at each peer update_interval query_timeout RTT thresholds TH1, TH2 Energy thresholds TH1, TH2 max_responses TTL

Values 10, 20, 30, 40, 50 Uniform random [0, 500] Uniform random [0, 10] Uniform random [0, 80] 30 sec 60 sec 1 sec, 3 sec 50%, 20% 1 32

Table 2: P2P level simulation parameters 4.2 Query Forwarding Performance Both Gia and RAON use biased random walk to forward queries. The critical difference between Gia and RAON is that the capacity level of a neighbor is the only factor that biases the random walk in Gia. In contrast, link color is also a factor biasing the random walk in RAON, which is expected to improve the query success rates and delays by improving the quality of the forwarding decision. Further improvement in the performance of RAON is expected due to PNR’s topology adaptation. The main performance metrics we used for evaluating the relative query forwarding performance of Gia and RAON are query success rates and query delays. Nevertheless, before discussing those measures we need to look at the quality of the forwarding decisions. In our implementation, each node measures the RTT value for the links to its neighbors. It also measures the energy value of its neighbors. Based on those measures it marks the links green, yellow or red. Although those measures are not used in Gia, we measure those values even for Gia in our simulation. Figure 4 plots the relative percentage of link color X (% of links colored X in RAON – % of links colored X in Gia) where X is green, yellow or red. We distinguish the red links for high delay (RTT) and red links for low energy, which are labeled in the figure as Red (d) and Red (e) respectively. Since the path chosen for a query response is independent of the path chosen for the query, a query that is sent over a green link its response may traverse a link of different color. Therefore, we traced the link colors separately for forward and reverse paths. The negative value of a graph means Gia shows more links of that color than RAON. Percent difference of color links used

40% 30%

Forward Green Forward Yellow Forward Red(d) Forward Red(e)

Total Green Total Yellow Total Red(d) Total Red(e)

20% 10% 0% -10% -20% -30% Number of Peers

Figure 4: Percent difference in colored links used by Gia and RAON for the 20:1000 scenario The above graph illustrates that in general RAON tends to use more green links than Gia along the forward path. Another expected trend is that Gia uses more red links than RAON. However, quite unexpectedly, the total number of red links used in RAON is rather close to Gia. This indicates that RAON uses more red links for reverse routes than Gia. Furthermore, since LCS can change link color only once in update interval, a node may continue using the link as green and prefer that link to others even if the link latency increases shortly after the link is marked green. Decreasing the value of update interval to achieve better accuracy of the link conditions will increase the overhead due to excessive number of control (update) messages. Increasing the protocol overhead will drain more energy of the processing node. Therefore, we cannot arbitrarily decrease the value for update interval. To evaluate the performance of a P2P system, we measure the success rate of query search and the query delay. A query is considered successful if the query originator receives at least one query hit response before the query times out. The query delay is the time it takes for the originator to receive the query hit response. In this Section, we compare the performance of Gia and RAON under various MANET environments. The query success rate and query delay are plotted in Figure 5 and 6 for Gia, (RAON – PNR) and RAON respectively. We observe the following: (i) As the number of peers increases, the success rate in Gia tends to drop while the query delay shows a rising trend. The network traffic also increases with the number of nodes, increasing the traffic load on every node thereby resulting in higher query delay. 100% 90%

SuccessR ate

80% 70% 60% 50% 40% 30%

2:500 2:1000 5:500 5:1000 20:500 20:1000

20% 10% 0% 10

20 30 40 Number of Peers

50

100% 90% 80% SuccessR ate

70% 60% 50% 40% 30% 20%

2:500 5:500 20:500

10%

2:1000 5:1000 20:1000

0% 10

20 30 40 Number of Peers

50

100% 90%

S u cce ssR a te

80% 70% 60% 50% 40% 30% 20%

2:500 5:500 20:500

10%

2:1000 5:1000 20:1000

0% 10

20 30 40 Number of Peers

50

Figure 5: Query Success Rate for Gia, (RAON-PNR) and RAON

5

Query Delay(sec)

4

3

2 2 :5 0 0 5 :5 0 0 2 0 :5 0 0 2 :1 0 0 0 5 :1 0 0 0 2 0 :1 0 0 0

1

0 10

20

30 40 N u m b e r o f P e e rs

50

30 40 N u m b e r o f P e e rs

50

5 2 :5 0 0 5 :5 0 0 2 0 :5 0 0 2 :1 0 0 0 5 :1 0 0 0 2 0 :1 0 0 0

Query Delay(sec)

4

3

2

1

0 10

20

5 2 5 2 2 5 2

Query Delay(sec)

4

:5 0 :5 0 0 :5 :1 0 :1 0 0 :1

0 0 0 0 0 0

0 0 0 00

3

2

1

0 10

20

30 40 N u m b e r o f P e e rs

50

Figure 6: Query delay for Gia, (RAON-PNR), and RAON (ii) The performance degrades when the simulation area and the node’s maximum speed are increased. The impact of node density on the relative performance is noticeably higher than node speed. This observation is further confirmed when we found that the variation in the success rates and delays are more pronounced for networks of different dimensions than nodes of different speeds. Therefore, we can conclude that node density has a higher impact on the performance than node speed. (iii) RAON exhibits low variation in query success rate with increasing number of peers as compared to Gia as shown in Figure 5. For example in the highly dynamic scenario of 20:1000, the query success rate for Gia varies from 72% to 48% whereas for RAON – PNR it varies from 70% to 60%. The low variation in RAON – PNR is primarily due to the quality of its forwarding decision. The PNR improves the query success rate of RAON from 70% to 85%. In case of RAON, the source does not generate queries unless it finds a good link, which is either a green or yellow link. Holding a query for a good link by the source may cause delay in query generation time but it contributes to increase the chance of query success. This is because the stability of a good link reduces the query drop probability, and it becomes more likely for the next hop having high residual power to process and forward the query. However, since RAON tends to use good links only in the forwarding path and does not guarantee that when the reverse path uses the same links they remain good links, we see some drop in the query success rate. (iv) PNR overhead: PNR involves probing of new peers that triggers underlying route discovery for on-demand routing protocol such as AODV [15]. Figure 7 shows the number

of AODV route requests during the simulation lifetime. From the graphs, we notice that generally RAON generates slightly more route requests than Gia, indicating that it explores more routes in the underlying network. Since AODV uses broadcast to discover a new route, the small increase of route requests generates a large number of route request messages in the network. Table 4.2 summarizes the average number of messages each system generates in all MANET scenarios. It is worth noting that the number of messages generated in (RAON – PNR) is comparable to Gia, which implies that LCS alone does not create any significant overhead. On the other hand, RAON that implements PNR generates probe messages to discover new neighbors, resulting in approximately 5000 more AODV request messages than Gia on average. This indicates the overhead of PNR in exploring routes to the neighbor candidates. It also shows PNR’s tendency of exploring new routes, which can create excessive control traffic if invocation of PNR is not regulated.

Type of Message Query, Query Hit, Ping, Pong Token Updates + ACK Connection Probe + ACK Retransmissions (Token Update and Query Hit) AODV request messages Total

Gia 10187.48 3886.42 161.54 0 1652.85

RAON – PNR 9504.60 4010.72 166.57 0 1582.40

RAON 7887.06 4084.76 159.82 87.72 1318.24

33860.61 49748.9

33947.09 49392.66

38622.23 52159.82

Table 3: Average number of messages generated over all MANET scenarios for Gia, (RAON – PNR), and PNR. 6000 2:500 2:1000 5:500 5:1000 20:500 20:1000

A O D VR e q u e sts

5000 4000 3000 2000 1000 0 10

20 30 40 Number of Peers

50

6000 2:500 5:500 20:500 2:1000 5:1000 20:1000

A O D VR e q u e sts

5000 4000 3000 2000 1000 0 10

20 30 40 Number of Peers

50

Figure 7: AODV Requests for Gia and RAON. 4.3 Cooperative Caching performance We simulated RAON with cooperative caching support and called it RACC. We implemented RACC in OMNet++ [31] with OverSim [32] overlay and INETMANET [33] ad-hoc network frameworks. Our objective is to compare the performance of RACC with Simple Cache, CacheData, and CachePath algorithms. We simulated mobility using random waypoint mobility model, as described above, in a 1000m x 1000m area at the maximum speed of 5m/s to implement the low mobility scenario of non-motorized vehicles. We used AODV as the underlying routing protocol in MANET. Each node joins the P2P network randomly. After a node is connected to the overlay network, it periodically sends a message to find its neighbors every update-interval time of 10 sec. In this model, all nodes, except the file server, are mobile nodes. Each file server holds 100 files. The file size varies from 100kb to 300 kb. When a node joins the network, its cache is empty. Each node sends a request for the file that it does not have in its cache. Queries are generated at a random interval of [10s, 30s] to model the behavior of aggressive P2P users. Each file has a unique file-Id for which nodes send a request. The cache sizes of nodes are different to reflect the diversity of resources. The cache size is considered large enough to avoid cache replacement. In this model when a query source generates a request for a specific file and receives the file, it keeps the file until the end of the simulation. Intermediate nodes set expiry time for their caches to 30 seconds that is reset by the arrival of a query for the data item before it expires. We evaluated performance using three metrics: average query delay, number of hop counts and average residual energy. We compared performance of RACC with Simple Cache, CacheData and CachePath schemes. The Simple Cache scheme is a non-cooperative scheme where a node caches data only when it has generated the query for

that ddata. If it dooes not find data in its cache c durinng query proocessing, it forwards f thee query to thee next hop. In a CachePath schem me, an interrmediate noode caches the path onnly to a queriier (data cacche) if the qquerier is few wer hops aw way that is typically onne. While itt caches the data item in a CacheDatta scheme iff its Pr > PHHigh and it recceives multiiple queries for the multiple prevvious hop noodes. In samee data item iif at least soome of thosee queries coome from m both schemes, iff an intermeediate nodee finds eitheer path or ddata item inn the cache, then it mentation off RACC, ann intermediaate node sendss query respponse to the querier. Inn our implem cachees the data item if Pr > PHigh for its i residual ppower otherwise it cacches the patth in its path cache. It keeeps multiplee suppliers w with their Pr and te vallues that it ssends to the querier node to select the best suppllier. Aveerage Querry Delay: Fiigure 8(a) shhows the deemands for data increasses with inccreasing numbber of nodess, which inccreases com mpetition am mong nodes for f limited bbandwidth causing c averaage query delay grow aalbeit at diffferent rates for all fourr schemes. T The Simplee Cache schem me shows thhe highest query q delayy since all queries q end at the servers incurrinng large ms, the queryy delays. Since intermeediate nodess in CacheP Path schemee do not cacche data item path to a suppllier is longger than RA ACC that ccauses highher query ddelays. In case c of CachheData, an inntermediatee node that ddoes not havve data item m in the cacche forwards query downnstream to a cache or a server; hennce, it show ws higher qquery delay than RACC C where the inntermediate node if dooes not find data item in i the cachee returns reesponse by looking l into its i path cacche. Furtherr, RACC caaches are m more available and closeer to high ddemand of caches thhrough their expiry tim path because cacching decisiions track availability a mes and demaand-supply iin the netwoork.

Figgure 8: (a) A Average Quuery Delay ((b) Averagee Query Hopp count

Aveerage Hop Count: Reeducing thee number of hops betw ween a queery source aand the destinnation imprroves powerr consumpttion in the nnetwork andd the queryy delay. In RACC, each intermediatte node traccks the hopp count to the t supplierrs in order to t find the nearest suppllier with HIIGH residuaal energy. AODV A provvides the hopp count of a path. Figuure 8(b) show ws the averagge hop counnt between qquery sourcee and destinnation. Longger paths in Simple Cachhe due to lacck of data caaching in thhe network rreflects highher hop counts. The hopp count trendd of Figure 88(b) shows higher querry delay of sschemes othher than RA ACC in Figuure 8(a) due too their tendency of chooosing longeer paths. Aveerage Resid dual Energyy: RACC coonsiders residual energyy of nodes bbefore makiing cachiing decisionns. Thus, thee probabilityy of having a dead nodee in the netw work duringg the simullation time iis small. Figgure 9(a) shhows that RA ACC outperrforms otherr three schem mes in termss of residuall energy of the t nodes. It shows CacchePath schheme consum mes less eneergy than CacheData and sometim mes even leesser than RA ACC; since according to t CachePatth me nodes onnly cache thhe path not thhe data, which consum mes less enerrgy. The CD DF schem graphh of residuall energy in Figure F 9(b) for 2m/s sppeed shows less l variancce of residuaal energgy in case off RACC as compared too other scheemes. Hencee, demand-ssupply basedd coopeerative cachhing improvves the network lifetimee.

(a) Aveerage Residdual Energy

F of Residuaal Energy foor 2:1000 w with 50 peerss (b) CDF Figu ure 9: (a) A Average Residual Energgy (b) CDF of o Residual Energy 5. C Context-Aw ware RAON N Middlewaare Softwarre Design Thee aim of ourr middleware software iis to implem ment the functionalities of RAON aand proviide commonn API to MA ANET appliications. The middlewaare software provides follow wing functioonal and non-functionaal requiremeents: bootstrrapping, rouuting, resourrce discoovery, abilityy to disconnnect, portabiility, and eff fficiency 5.1 Middlewarre Layered Design Thee proposed R RAON midddleware resiides between the appliccation and thhe network layers. We aadopted a layyered designn of middleware softwaare, which is structuredd in three layyers as show wn in Figure 10. The appplication-suupport layer implementss the API ussed by ad hooc applications. Thee overlay layer implem ments the RA AON overlayy functionallities, such aas bootsstrapping, roouting, resouurce discovery etc. It allso implemeents contextt frameworkk to gatheer context innformation ssuch as residdual power, location etcc. that are used u by RAO ON. The ccontext APII is also avaiilable for thhe applicatioons to retrievve context ddata. The add hoc layerr is an optionnal layer thaat implemennts network support funnctions. For example, soome ad hoc rrouting protoocols measuure the RTT T and residuaal power off neighbors aas a part of ttheir routinng metrics tthat this layeer retrieves from the prrotocol objeccts and provvides them tto the LCS..

Figure 10: Layered Architecture of RAON Middleware Software 5.2 Middleware Sub-systems We modularized implementations of layers in the following subsystems. The components of the subsystems are services that a subsystem provides, such as LCS, or they are abstractions of entities that exist in the overlay network, such as Node, Neighbor and Peer. Application Support: It provides an interface to the applications of the middleware and is mainly composed of three components: Node, Message Handler and Path. The Node provides an abstraction of overlay node to the application with functions such as getting an overlay address, joining and leaving the network, launching query for a resource in the network, such as files, and returning a path to the resource. The middleware provides support for multiple simultaneous applications by instantiating multiple nodes. Its resource discovery mechanism relies on a single query packet, unlike some ad hoc middleware, such as [16], in which resource discovery is based on broadcasting resources. The user application registers a Message Handler with the Node that gives it a mean of communicating with other overlay nodes. It is invoked to handle application-level messages, generate error reports and exceptions to the application. The Path is a component that provides abstraction of a path to a resource in the overlay network. At the end of a query reply, the application gets access to the path to the destination through the Path component that can be used, for example, to download the file. Cross-Layer Services: Some utilities and services are needed across different layers of the middleware, such as overlay packet processing, LCS, and local databases for managing

node, neighbor and data download information. The LCS implements RAON’s link coloring scheme that opens a single duplex connection between a node and its neighbor. It implements periodic probing to measure link stability and residual power for MBR. In case of deriving path stability metric from the routing metrics [26, 27], it interacts with underlying routing protocol through the ad hoc layer. Bootstrapping Subsystem: The role of distributed bootstrapping algorithm is crucial for the implementation of RAON since the overlay topology does not change unless a peer node disappears due to shut down or power failure. We implemented a simple bootstrapping procedure where each node joins the network with a static list of overlay peers by multicasting connection request message using java.net.MulticastSocket API. The bootstrapping subsystem is tightly coupled with the underlying ad hoc layer, as it needs multicast support from the ad hoc routing protocol. More sophisticated bootstrapping algorithm may build overlay topology as a function of the underlying ad hoc topology, network path stability, mobility pattern of peer nodes etc. Overlay Management: Nodes rely on Connection manager to establish and manage connections with their neighbors, perform query forwarding, generate query replies and handle overlay packets. It is also responsible for setting up an overlay path between source and destination and maintaining that path. When a query arrives at a destination it knows the querier’s address from the query that also carries an option of sending query reply either directly to the querier or along the reverse overlay path. Some applications require query reply through the reverse path to establish overlay path between the querier and the destination for resource delivery. For example, application level multicasts need overlay paths to build the multicast distribution tree, congestion-aware file download for RAON needs overlay paths for efficient file download through congestion-free path, and cooperative caching needs to make CachePath and CacheData decision along the reverse path. However, the reverse overlay path may not always be available, such as when some intermediate nodes leave the network after forwarding the query. Hence, our middleware gives application an option of choosing either the reverse overlay path or direct connection to the querier from the destination. The MBR forwarding algorithm is implemented in the Overlay layer. Since next hop calculation is based on MBR, query forwarding may encounter a loop. To solve this

problem, a query packet carries sequence number and if it revisits a node, then the node simply forwards it to a different neighbor. The node keeps track of all the neighbors to which it forwards a query. If the node has already sent the query to all of its neighbors, then it drops the query. 5.3 Context Framework RAON middleware implements a context framework that provides node residual power, Pr to LCS and other services. A mobile device acquires context information, such as ambient light, temperature, humidity etc. from external sensors, and power, location data etc. from internal sensors. The sensors are activated at the behest of an application, which identifies a set of context information to be monitored and collected by the mobile device. Figure 11 shows the context-monitoring framework that is implemented in Android. The framework allows monitoring and storage of a number of context sensors accessed through sensor managers. The sensor broker in the framework coordinates activation, operation and shutdown of these sensor managers that run in different threads. RAON context handler in the overlay layer acts as the client of this framework, which installs context filters in and retrieves context information from the context processor. The change detection module checks context data it receives from the sensor manager for conditions detecting the change, such as Pr crossing PLow or PHigh thresholds. The notification module generates notification corresponding to changes detected. Thus, RAON context handler is not invoked for every new power reading from the power sensor. More elaborate preprocessing of context data can be done for high level feature extraction instead of sending raw data to the framework client [28].

Figure 11: Context Framework integrated with RAON

5.4 Application Deployment We implemented RAON middleware in Java J2ME Android 4.2.2 and deployed on a MANET of Nexus 7 tablets configured in ad hoc mode running OLSR [14, 21].The middleware provides an abstraction of a path to the application for locating resource at a destination node in the overlay network through query-reply mechanism. We discuss the implementation of an Application-Level Multicast (ALM) and Credit-based File Sharing (CFS) applications using the middleware API. The ALM is a popular application [17], [18], [19], and [20], that scales well on overlay networks. Figure 12 shows its implementation using RAON middleware. The node that serves as the root of a multicast distribution tree creates a file with the same name as of group ID by calling Node.CreateFile(). A receiver node launches a query by calling Node.Query() to locate a parent node in the tree by locating the group ID file. Once a destination is located, it becomes a parent node as the receiver uses Path.SendData() to send Join message. The new receiver then creates its own group ID file and thus becomes a

node in the multicast distribution tree that can further entertain join from new receivers. To multicast data to the group members, each node simply sends a data packet to its parent by calling Path.SendData(). A receiver can leave the tree by sending Leave message through Path.SendData() to its parent that removes the node from the list of its children.

Figure 12: ALM Implementation using RAON Middleware

We implemented a credit-based file sharing application using RAON middleware where a node stores files it is sharing in a specific directory. As shown in Figure 13, a node searching for a file launches an overlay query by calling Node.Query(). After receiving reply and Path to the destination, it calls Node.Download() to download the file. We implemented a credit-based scheme in which each node that offers a file gains credit at the rate of share-credit, and every node that forwards data packets also gains credit at the rate of forward-credit. We conducted three tests of CFS to measure the impact of OLSR [14, 15] and RAON middleware on node’s power consumption. Table 4 shows the battery lifetime for the four cases: 1) Only Android, 2) OLSR, 3) OLSR + RAON middleware, and 4) OLSR + RAON + File Transfer. The test result shows that OLSR has considerable impact on node’s power consumption and file download (for smaller file) has marginal impact. RAON LCS probing for link stability and neighbor’s power reduces node’s battery lifetime significantly.

Figure 13: File Sharing Application using RAON Middleware

Battery Lifetime Node 1

Node 2

Node 3

Android

7h45m

7h52m

-

OLSR

6h47m

7h1m

6h51m

OLSR+RAON

4h2m

3h40m

3h36m

OLSR+RAON+ File Download

3h48m

3h21m

3h27m

Table 4: Power Consumption 6. Conclusion and Future Work Developing applications for MANET is a challenging task as it needs to deal with ad hoc and unstable connectivity, abrupt disconnection, and resource constraints including limited

power and computing resources. In this paper, we presented RAON middleware that is designed using RAON unstructured overlay network on MANET as a substrate to engage nodes that are involved in the application with application-level state information. It offers proactive neighbor replacement to avoid disconnected overlay due to loss of power at a node, congestion-aware data download through overlay reverse path to reduce download delay, and co-operative caching to improve data access latency and prevent power drainage at fewer nodes. Simulation results show that these features are effective in reducing query delay, improving data availability, and balancing node power consumption. We designed and implemented middleware software in J2ME on Android that provides application-level support and API to application developers through Node and Path abstractions to access overlay features. The middleware incorporates with a generic Context Framework to acquire user and device context data, which RAON as well as middleware applications can use. We demonstrated its viability by implementing two applications: Application-level Multicast (ALM) and a Credit-based File Sharing (CFS) application. We also tested CFS in an ad hoc network of Nexus 7 devices running Android 4.2.2 and OLSR. We identified some future work that will enhance its features and enable it for D2D caching in 5G networks [41]. One, the overlay link stability computation can be improved to mitigate the mobile OS induced bloating factors [26]. The measure can be extracted from path stability metric of the underlying MANET routing [27], such as BATMAN [22], which can be used as a measure by LCS. Second, a proactive neighbor replacement algorithm based on link stability metric can be implemented to ensure that each node remains connected to its physically closest nodes, which can help build a more localityaware topology. Third, one-hop replication can improve performance by exploiting locality of reference. Fourth, a credit-based flow control such as the one used in Gia can be implemented. Fifth, query forwarding algorithm can be enhanced to discover caches in small cell base stations in 5G network [39] and augment cooperative caching demand and supply framework with social network metrics [40]. Sixth, the resource discovery approach can be used in sensor [42] and cognitive networks [43].

Acknowledgment We acknowledge the support of NSERC Discovery Grant for this work.

References [1] G. Lau, M. Jaseemuddin, and G. Ravindran “RAON: A P2P Network for MANET,” Proc. of 2nd IEEE/IFIP WOCN, 2005. [2] G. Lau, M. Jaseemuddin and G. Ravindran, Proactive Neighbor Replacement for RAON, Proceedings of IEEE Wireless and Mobile Computing, Networking, and Communications (WiMobs), Montreal, August 2005. [3] D. Doval, and D. O’Mahony, “Overlay Networks: A scalable alternative for P2P,” IEEE Internet Computing, Jul-Aug 2003. [4] Y. Chawathe, et al, “Making Gnutella-like P2P Systems Scalable,” in Proc. of SIGCOMM 2003. [5] A. Rowstron and P. Druschel, “Pastry: Scalable, Distributed Object Location and Routing for Large-Scale Peer-to-Peer Systems,” in Proc. of the IFIP/ACM Middleware 2001. [6] S. Ratnasamy, et al, “A Scalable Content Addressable Network,” in Proc. of the ACM SIGCOMM, pp. 161–172, 2001. [7] I. Stoica, et al, “Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications,” IEEE/ACM Transactions on Networking, vol. 11, no. 1, pp. 17– 32, 2003. [8] B. Y. Zhao, et al, “Tapestry: A Resilient Global-Scale Overlay for Service Deployment,” IEEE Journal on Selected Areas in Communications, Vol. 22, No. 1, pp. 41–53, January 2004. [9] I. Clarke, et al, “Freenet: A Distributed Anonymous Information Storage and Retrieval System,” In Proc. of the ICSI Workshop on Design Issues in Anonymity and Unobservability, 2000. [10] http://www.bittorrent.org/bittorrentecon.pdf [11] http://www.gnutellaforums.com/ [12] Napster, http://www.napster.com/ [13] M. Bhuiyan, and M. Jaseemuddin, “Congestion-Aware Overlay Networks”, Proc. Of IEEE VTCFall 2006. [14] E. T. Clausen and E. P. Jacquet, “Optimized Link State Routing Protocol (OLSR), http://www.ietf.org/rfc/rfc3626.txt. [15] M. Abolhasan, T. Wysocki, and E. Dutkiewicz, "A review of routing protocols for mobile ad hoc networks," Ad Hoc Networks, Vol.2, No.1, pp.1–22, Jan. 2004. [16] P. Engelstd, et al, “Middleware Supporting Adaptive Services in On-Demand Ad hoc Networks,”in Proc. of Int. Conf. on Intelligence in service delivery Networks, France, 2004. [17] M. Castro, et al, “SCRIBE: A Large-Scale and Decentralized Application-Level Multicast Infrastructure”, IEEE Journal on selected areas in communications, Vol. 20, No. 8, October 2002. [18] S. Ratnasamy, et al, “Application-Level Multicast using Content-Addressable Networks,” in Proc. of Int. Workshop on Networked Group Communication, London, UK, pp. 14–29, Nov. 2001. [19] S. Kandula, J. Lee, and J. Hou, “LARK: A Light-weight, Resilient Application-Level Multicast Protocol,” in Proceedings of IEEE Computer Communication Workshop, November 2003. [20] F. Dabek, et al, “Towards a Common API for Structured Peer-to-Peer Overlays”, Second International Workshop on Peer-to-Peer Systems (IPTPS 2003), February 2003. [21] J. Robble, “The SPAN Project”, 01- 09 2012. [Online: https://github.com/ProjectSPAN. [22] L. Delosieres, and S. Nadjm-Tehrani, “BATMAN store-and-forward: The best of the two worlds”, IEEE Pervasive Computing and Communications Workshops (PERCOM Workshops), 2012. [23] H. Nejad, M. Jaseemuddin, and A. Al-Karkhi, “Co-operative Caching in P2P MANET”, the 12th ACM Int. Symp. On Mobility Management and Wireless Access (MobiWac 2014), 2014. [24] P. Bellavista, A. Corradi, and C. Gianneli, “Mobility-aware Middleware for Self-organizing Heterogeneous Network with Multihop Multipath Connectivity”, IEEE Wireless Communications, Vol. 15, No. 6, pp. 22—30, December 2008. [25] U. Sadiq, et al, “Service Composition in Opportunistic Networks: A Load and Mobility Aware Solution”, IEEE TOC, Vol. 64, No. 8, pp: 2308 – 2322, 2015. [26] W. Li et al, “On the Accuracy of Smartphone-based Mobile Network Measurement”, IEEE INFOCOM, 2015. [27] R. Sheshadri and D. Koutsonikolas, “An Experimental Study of Routing Metrics in 802.11n Wireless Mesh Networks”, IEEE Trans. on Mobile Computing, Vol. 13, No. 12, pp: 2719—2733, Dec. 2014. [28] Y. Lee, et al, “MobiCon: A Mobile Context-Monitoring Platform”, Communications of the ACM, Vol. 55, No. 3, p: 54 – 65, March 2012.

[29] D. S. J. D. Couto, et al., “A high-throughput path metric for multi-hop wireless routing.” in Proc. of ACM MobiCom, 2003. [30] http://www.isi.edu/nsnam/ns/ [31] www.omnetpp.org [32] www.oversim.org [33] https://inet.omnetpp.org/ [34] L. Yin and G. Cao, “Supporting cooperative caching in ad hoc networks”, IEEE Transactions on Mobile Computing, Vol. 5, No. 1, pp. 77–89, 2006. [35] U. C. Kozat, et al, “Peer assisted video streaming with supply-demand-based cache optimization”, IEEE Trans. On Multimedia, Vol. 11, No. 3, pp. 1515–1532, 2006. [36] J. Zhao, P. Zhang, G. Cao, and C. R. Da, “Cooperative caching in wireless p2p networks: Design, implementation, and evaluation”, IEEE Transactions on Parallel and Distribution System, Vol. 21, No. 2, pp. 229–241, 2010. [37] W. Wu and J. Cao, “Efficient Cache Discovery for Cooperative Caching in Wireless Ad Hoc Networks”, The 18th IEEE Int. Conf. on Parallel and Distributed Systems (ICPADS), 2012. [38] W. Wu, J. Cao, and X. Fen, “Design and Performance Evaluation of Overhearing-Aided Data Caching in Wireless Ad Hoc Networks”, IEEE Trans. on Parallel and Distributed Systems, Vol. 24, No. 3, pp. 450 – 463, 2013. [39] N. Golrezaei, et al, “FemtoCaching: Wirleess Video Content Delivery through Distributed Caching Helpers”, IEEE INFOCOMM, 2012. [40] E. Bastug, M. Bennis, and M. Debbah, “Living on the Edge: The Role of Proactive Caching in 5G Wireless Networks”, IEEE Communication Magazine, Vol. 52, N0. 8, pp: 82 – 89, August 2014. [41] J. Rao, et al, “Optimal Caching Placement for D2D Assisted Wireless Caching Networks”, IEEE ICC 2016. [42] M. Amjad, M. Sharif, M. K. Afzal, and S. W. Kim, TinyOS-New Trends, Comparative Views, and Supported Sensing Applications: A Review. IEEE Sensors Journal, Vol. 16, No. 9, pp.2865-2889, 2016. [43] A. A. Khan, M. H. Rehmani, and M. Reisslein, Cognitive radio for smart grids: Survey of architectures, spectrum sensing mechanisms, and networking protocols. IEEE Communications Surveys & Tutorials, Vol. 18, No. 1, pp.860-898, 2016.