A loss aware MPTCP scheduler for highly lossy networks

Computer Networks 157 (2019) 146–158

Contents lists available at ScienceDirect

Computer Networks journal homepage: www.elsevier.com/locate/comnet

A loss aware MPTCP scheduler for highly lossy networks Enhuan Dong a,b, Mingwei Xu a,b,∗, Xiaoming Fu c, Yu Cao d a

Department of Computer Science and Technology, Tsinghua University, China Beijing National Research Center for Information Science and Technology (BNRist), China c Institute of Computer Science, University of Goettingen, Germany d National Digital Switching System Engineering & Technological Research Center, China b

a r t i c l e

i n f o

Article history: Received 24 January 2018 Revised 15 January 2019 Accepted 4 February 2019 Available online 5 February 2019 Keywords: Multipath TCP Scheduler Highly lossy networks

a b s t r a c t Varieties of wireless communication links, such as HSPA+ access in high speed trains, balloon-based aerial wireless networks and satellite Internet connections have high loss rates. In such environments, Multipath TCP (MPTCP) offers a robust solution compared to regular TCP. However, MPTCP and existing schedulers suffer from performance degradation for both constant bit rate (CBR) and bulk traffic. To tackle this challenge, we develop LAMPS, a novel scheduler for MPTCP, which considers both the loss and delay when selecting subflows, and chooses segments based on subflows’ state. The design goal of LAMPS is to achieve a steady performance for different traffic and significantly reduce the unnecessary bandwidth consumption, especially in case of bursty losses. We have implemented LAMPS and evaluated its performance for Dynamic Adaptive Streaming over HTTP (DASH), CBR, and bulk traffic. Our experiment results show that LAMPS preserves application latency, keeps low memory consumption, and significantly reduces extra bandwidth consumption in the presence of high packet loss rate. © 2019 Elsevier B.V. All rights reserved.

1. Introduction Modern wireless technologies provide a critical infrastructure for data delivery for billions of users. However, in a variety of wireless access scenarios, data transmissions are still subject to high losses and usually the losses have a bursty fashion. In the case of increasingly deployed high speed rails (HSRs), the packet loss rates of TCP flows rise sharply in urban areas over HSPA+ networks [1,2]. Table 1 shows the probabilities that the packet loss rate is higher than 20% or 30% in 3 min TCP flow transmission for different handoff times when trains run at a stable high speed [1]. For example, if a 3 min TCP flow suffers 11 to 16 handoffs when train speed is stable and high, the packet loss rate is more than 20% at the probability of 53%. Li et al. [1] have also observed that in urban areas TCP flows suffered more frequent handoffs. In their measurement, a 3 min flow has experienced handoff 12 times when the train went through Beijing. As a result, in urban areas, TCP flows suffer unneglectable high packet loss rate on High-speed Rails.

∗ Corresponding author at: Department of Computer Science and Technology, Tsinghua University, China. E-mail addresses: [email protected] (E. Dong), xumw@tsingh ua.edu.cn, [email protected] (M. Xu), [email protected] (X. Fu), [email protected] (Y. Cao).

https://doi.org/10.1016/j.comnet.2019.02.001 1389-1286/© 2019 Elsevier B.V. All rights reserved.

Link-level ARQ/Hybrid-ARQ (HARQ) schemes are able to significantly reduce packet loss rates in general. However, it has been observed that, although link-level ARQ/HARQ schemes are used, fast moving ground vehicles still suffer high loss rates (20% or more) with high variance when they are communicating to each other [3,4]. Another novel emerging technology is Google’s Project Loon [5], which aims to provide Internet access to rural and remote areas. It uses balloons placed in the stratosphere to create an aerial wireless network. As the weather can substantially impair the high altitude wireless communication [6,7], data delivery in Loon project may be subject to high loss rate. Traditional wireless technologies, such as satellite Internet communications, also suffer from high loss rate. Lee et al. [6] and Ma et al. [7] have revealed that, because of frequent physical channel errors, the loss rate is relatively high in satellite networks, sometimes even exceeding 50% caused by rain attenuation. In order to improve the low data transmission efficiency and low user experience in such lossy environments, which are usually bursty, Multipath TCP (MPTCP) [8] offers a solution to more efficient and robust data delivery. As an enhancement of regular TCP, MPTCP leverages multiple paths for the transmission of a single data flow, which makes it possible to pool the resources of disparate paths, effectively increasing the goodput of applications. Moreover, MPTCP allows the sender to seamlessly hand over the

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158 Table 1 Probabilities that the packet loss rate is higher than 20% or 30% for different handoff times on HSRs [1]. Packet loss rate

> 20%

> 30%

Handoff times: 6–10 Handoff times: 11–16

29% 53%

22% 42%

traffic from one path to another, leading to better resilience towards network failures. The wireless use case is one of the prominent use case for MPTCP. To create multipath transmission in such case, MPTCP usually needs to connect to multiple wireless networks, which is fully accepted by MPTCP community [9]. Li et al. [10] have studied the performance of MPTCP with various cellular carriers in the high speed train, and showed that MPTCP was much more robust than TCP. Moreover, MPTCP can also employ different kinds of wireless networks at the same time. For example, MPTCP can connect to one satellite network by a BGAN Terminal [11] and one cellular network, then two paths are built in these two networks respectively. As long as one path is available to use, the MPTCP connection would not stop sending data. In MPTCP, a scheduler is used to distribute segments among all paths [8,12] and has a great influence on the transport performance, especially when paths are heterogeneous. It is worth noting that the scheduler is different from the congestion control algorithm. While a congestion control algorithm decides all subflows’ congestion window sizes, thus guaranteeing the fairness with other flows, a scheduler is responsible for the assignment of segments among all subflows. To understand the performance of MPTCP schedulers in highly lossy networks, we conduct testbed experiments for both CBR and bulk traffic. In our test, we regard CBR traffic as the simplest video traffic to simplify the analysis of results. MPTCP implementation (MPTCP v0.90 based on Linux v3.18) [13] includes three schedulers by now: Lowest-RTT-First (LowRTT) [12], Redundant (RDDT) [14,15] and Round-robin (RR). LowRTT sends segments on the subflow with lowest Round Trip Time (RTT) estimation. RDDT redundantly sends packets over all subflows. RR scheduler chooses one subflow after the other circularly. The detailed test experiments and results will be introduced in Section 2, and we only show the conclusions here. In CBR traffic experiments, we find that RDDT achieves best application latency because of the redundant transmission, while LowRTT can be worse than RR when there exists low RTT and high packet loss rate path. In bulk traffic experiments, compared to LowRTT and RR, RDDT uses the bandwidth from only one path for its repeated scheduling. In conclusion, existing schedulers do not work well in the networks with moderate to high losses. Since a scheduler only working in some specific scenarios is not efficient and useful, we propose three design goals for our scheduler: • In highly lossy networks, especially the networks with bursty losses, compared to the default scheduler, new scheduler should improve the performance for video streaming traffic, such as Dynamic Adaptive Streaming over HTTP (DASH) traffic or CBR traffic, and not decrease the performance for other traffic. • In highly lossy networks, especially the networks with bursty losses, compared to RDDT, new scheduler should achieve similar application delay and significantly reduce the extra bandwidth consumption. • New scheduler should degrade to default scheduler when packet loss rate is low. In this paper, we propose Loss-Aware MPTCP Scheduler (LAMPS), a novel scheduler for MPTCP. We define transfer time as

147

the time-interval between sending and receiving a segment on an individual path. LAMPS takes both packet loss rate and delay into account when computing transfer time for all subflows, and selects the subflow with the lowest transfer time to send segment. Moreover, our new scheduler chooses segments based on the subflow’s state. If a subflow finds the path in a low performance state, LAMPS will assign redundant segments on it. In case of the normal state, LAMPS will distribute normal segments. Subflow state shifting condition is implemented in a reconfigurable manner. Users can set the state changing rules based on their networks situation. We have implemented LAMPS in Linux kernel v3.18 based on the modular scheduler framework in MPTCP v0.90 [13]. We employ real-world experiments, testbed experiments and emulations to evaluate the performance of LAMPS extensively. Since the aforementioned scenarios are all lossy and mostly in a bursty fashion, we test LAMPS’ performance in emulated bursty networks. Besides CBR traffic, we also run real video streaming application generating Dynamic Adaptive Streaming over HTTP (DASH) traffic to evaluate LAMPS’ performance for video traffic. The comprehensive experiment results show that LAMPS meets all its design goals. Our code is available online [16]. The rest of our paper is structured as follows. Section 2 shows the performance of MPTCP’s schedulers in lossy networks. Section 3 presents LAMPS’ design and implementation. Evaluations are shown in Section 4. Section 6 introduces related work and Section 7 concludes the paper. 2. MPTCP’S schedulers in lossy networks During the MPTCP architecture design, Raiciu et al. [8] and Paasch et al. [12] assume one MPTCP flow has several subflows and each subflow runs on a different path. MPTCP employs a scheduler module to decide which subflow should be used to transmit [12]. The scheduler can access the information of all subflows, and finish packet distribution based on these information. Each scheduled packet is ideally sent on one subflow, unless the subflow experiences timeout. After a timeout, the packets sent but not received on that subflow may be reinjected into other subflows. Moreover, when a scheduler is working, the number of assigned packets on one subflow is limited by the subflow’s congestion window. At the receiver, all the subflows of the same MPTCP connection share a unique receive window. Since the characteristics of paths may differ, this can lead to out-of-order delivery at the receiver and this situation is called head-of-line blocking [17]. Head-of-line blocking is harmful to the video streaming, which increases application latency and memory consumption, resulting in poor user experience. A recent study [18] shows that in North America more than 53% of the downstream traffic is video streaming and another report [19] also predicts that the downstream traffic would increase in the future. MPTCP implementation (MPTCP v0.90 based on Linux v3.18) [13] includes three schedulers by now. LowRTT: LowestRTT-First [12,20] is MPTCP’s default scheduler which sends segments on the subflow with lowest RTT estimation. If the subflow with the lowest RTT has a full window data to send, the scheduler will select the subflow with the second lower RTT to transmit the subsequent segments.1 RR: Round-robin scheduler picks one subflow after the other circularly. RDDT: Redundant scheduler [14,15] redundantly sends packets over all subflows.

1 LowRTT also includes a mechanism of opportunistic retransmission and penalisation to optimizes its buffer [12]. But we do not focus on this mechanism, because we find that buffer size does not limit MPTCP transmission when loss events happen in this paper’s experiments.

148

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

Fig. 1. The testbed topology and emulation topology for most experiments in our paper.

In the following subsections, we validate the performance of MPTCP in lossy networks. We consider the following communication scenario: A sender and a receiver connect to each other in two networks and MPTCP creates two subflows (S1 and S2) running on the two networks respectively. We build a testbed using four personal computers. These PCs are physically interconnected through a switch and logically constitute the network topology shown in Fig. 1(a). We employ Linux tc (traffic control) program [21] to generate and configure two bottleneck links through two different paths respectively. The packet loss generated by tc is random in this paper. We use the default MPTCP congestion control algorithm, LIA [22] to conduct experiments in this paper, unless otherwise stated. 2.1. CBR multipath transport in lossy networks To make our experiments more easily understood and following some existing MPTCP scheduling studies [12,23], in this subsection, we consider CBR video traffic to assess different schedulers’ influence on the application delay. We create a similar dummy application, which sends a fixed-size data block every 800 ms and records the sending and receiving time of each block. We regard the difference in values between the sending and receiving time as application delay. One MPTCP flow is established and runs two subflows through two bottleneck links respectively. The link capacities of both links are set to 10 Mbps, which is big enough and does not limit the transportation in this subsection. We set up link 1 with 200 ms delay and 2% packet loss rate2 and link 2 with 100 ms, while changing the loss rate of link 2 from 5% to 30% for 6 different cases. We use 30% as the maximum value of packet loss rate in this part, because in urban areas 30% is a quite common value shown in Table 1. 30 data blocks of 8 KB are sent on the MPTCP flow. The size of data blocks is the same as that in [12] and the experiment for each case runs 10 times. The mean application delay of all blocks is shown in Fig. 2(a). RDDT achieves best application latency because of the redundant transmission and LowRTT is even worse than RR when the packet loss rate of S2 increases. This is because LowRTT considers RTT as the only value to judge paths, and ignores an important fact that a loss packet needs at least one RTT to recover, so if the default scheduler selects a path with low RTT but high loss rate, the application still suffers high delay. Fig. 2(b) shows the ratio between the reinjected packets and all packets sent by the application. As aforementioned, reinjected packets are generated from MPTCP’s RTO retransmission or RDDT redundantly transmission. Ideally, each packet is sent only once, so the amount of reinjected packets can be used as the metric indicating extra bandwidth consumption. Compared to RR and LowRTT, RDDT greatly increases reinjected packets, which means occupying much more bandwidth

2

Fig. 2. Results of CBR traffic experiments. Link parameters of S1 keep same (200 ms RTT, 2% packet loss rate). The RTT of S2 remains 100 ms and loss rate varies from 5% to 30%.

than other schedulers. Fig. 2(b) also shows RDDT’s ratio is decreasing while the loss rate of S2 is increasing, because the congestion window of S2 managed by congestion control algorithm shrinks when the packet loss rate increases. 2.2. Bulk multipath transport in lossy networks We run iPerf for 50s to generate bulk flow. Each link capacity is configured to 10 Mbps. The packet loss rate and RTT of each link are shown in Table 2. We use 20% as the packet loss rate value as it is common in HSPA+ networks on HSRs [1]. We repeat the experiment for 10 times and gather the output from iPerf as results. Table 2 Link parameters in Section 2.2. Link 1 Time(s) RTT(ms) Loss rate(%)

The packet loss rate is bidirectional in this paper.

0–50 20 0

Link 2 0–10

10–20

0

20

20–30 15 0

30–40

40–50

20

0

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

149

Fig. 4. Workflow of LAMPS.

Fig. 3. The mean goodput of MPTCP connection for existing schedulers. Link parameters are changed every 10 s. S2 suffers bursty packet losses in 10–20 s and 30–40 s. Each value represents the mean over all the runs. The error bars show the standard deviations around the means.

Fig. 3 shows the mean goodput that the bulk flow gets for different schedulers. Compared to LowRTT and RR, RDDT uses the bandwidth from only one path for its repeated scheduling and can not leverage multipath bandwidth. Comparing LowRTT and RR, we can not find significant difference. The reason is that, in bulk data transport, where the windows are always filled, the LowRTT and RR’s scheduling is controlled by the arrival of acknowledgements instead of scheduler, which is called ack-clocked. Therefore, it is hard for schedulers to benefit bulk traffic. 3. LAMPS: a loss aware scheduler for multipath TCP 3.1. Overview Our aim in this paper is to produce a scheduler, which should inherit the advantages of existing schedulers and abandon the shortcomings at the meantime. The new scheduler should work well in wide range of scenarios especially over lossy networks. Specifically, we propose three design goals for our scheduler: • In highly lossy networks, especially the networks with bursty losses, compared to the default scheduler, new scheduler should improve the performance for CBR traffic and not decrease the performance for other traffic. • In highly lossy networks, especially the networks with bursty losses, compared to RDDT, new scheduler should achieve similar application delay and significantly reduce the extra bandwidth consumption. • New scheduler should degrade to default scheduler when packet loss rate is low. We start our design from analyzing why existing schedulers have poor performance in Section 2. First, since LowRTT can be worse than RR, we think it is not a good idea to use RTT as the only value to choose subflow and the packet loss rate should also be considered in selecting process. Second, redundantly sending packets is helpful for application delay but its usage should be limited to extreme situations since it also leads to high extra bandwidth consumption. These two ideas constitute our basic design of LAMPS. Its workflow is shown in Fig. 4. MPTCP’s scheduler is triggered when it is ready to send data. The common trigger events include application has added new data to the sending queue and acknowledgement from one subflow has freed some space in the related congestion window. LAMPS is designed based on MPTCP’s architecture, so it has the same trigger events as other MPTCP schedulers. LAMPS first chooses a subflow based on the result from path quality assessment module. Path

quality assessment module computes the transfer time for all subflows and LAMPS selects the subflow with lowest transfer time. In Section 3.2, we will introduce the path quality assessment module in detail. LAMPS defines two different subflow states, NORMAL and REDUNDANT, and sets the subflow’s state according to the path properties. The subflows in REDUNDANT state send the packets satisfying the following three conditions: First, the packet is in MPTCP’s send queue, which means it has not been acknowledged yet. Second, the packet has never been sent before on this subflow. Third, the packet has the smallest sequence number among the packets satisfing the first two conditions. The subflow in REDUNDANT state does not care if a packet is sent by the other subflows when it is picking packet, so the chosen packet may be sent on the other subflows before and it is redundantly sent on the REDUNDANT subflow again. If all available paths are in REDUNDANT state, all subflows follow the three conditions to send packets. In fact, at this moment, LAMPS acts just like RDDT. On the other hand, the subflow in NORMAL state selects the packet with the smallest sequence number that has never been sent by any NORMAL subflow. The NORMAL-to-REDUNDANT and REDUNDANT-to-NORMAL rules are configurable in LAMPS. Users can set the state changing rules on the basis of their networks’ feature. For example, if the network suffers random burst high losses, LAMPS can be configured to change a subflow’s state from NORMAL to REDUNDANT if the subflow’s packet loss rate is larger than some configurable constant value. In addition, if the packet loss rate decreases and is smaller than the value, subflow’s state will change back.3 3.2. Path quality assessment in lossy networks LowRTT, MPTCP’s default scheduler, utilizes half of RTT to estimate the time-interval between sending and receiving a segment on an individual path [12] and we define the time-interval as transfer time. The path with lower RTT gets higher priority. However, we argue that in lossy networks, RTT is not proper to represent transfer time, because high loss rate incurs significant retransmissions, thus increasing transfer time of many packets. The transfer time is very important in scheduler design of MPTCP, so our idea is to model it in lossy networks, where loss is not mainly caused by congestion, and use the modeled transfer time to assess path quality. 3.2.1. Transfer Time in Theory Subflows of MPTCP have inherited many behaviors from regular TCP. In MPTCP, after one segment scheduled on a subflow, the chosen subflow will be responsible for delivering the segment, which

3 LAMPS also inherits some other mechanisms from the default scheduler, such as selecting packets in reinjection queue first and opportunistic retransmission with penalisation, etc. We only introduce the changed parts.

150

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

Fig. 5. The transmission process of a segment on a MPTCP subflow.

behaves like a TCP flow. If it finds out a segment is lost, it will retransmit the segment. Fig. 5 shows the transmission process of a segment on a subflow. We denote the delay for a segment transmitting from the sender to the receiver and the delay for a acknowledgement transmitting from the receiver to the sender by d1 and d2 respectively. Both directions have high loss rate and we denote the loss rate of direction from sender to receiver by p. Moreover, we denote by c the number of sending times, which include both transmission and retransmission. For any positive constant number j, the probability that c = j is approximately equal to the probability that the previous j − 1 transmissions are failed before a segment is received4 , and can be calculated as follows:

P [c = j] = p j−1 (1 − p),

j = 1, 2, 3 · · · · · ·

(1)

where p is the loss rate. Then we can get the mean of c:

E[c] =

∞ 

p j−1 (1 − p) j =

j=1

1 , 1− p

i = 1, 2, 3 · · · · · ·

(2)

After a segment is sent, sender will wait for a period of time to know whether the segment is lost. We denote that time interval by t. Then with the mean of t E[t], the transfer time T can be estimated as

T = (E[c] − 1 ) × E[t] + d1

(3)

It is difficult to estimate the accurate value of t or E[t]. There are two events to determine a segment is lost: arrival of several duplicate acknowledgements or expiration of the retransmit timer. On one hand, if the first loss is detected by the former event, it will retransmit the lost segment, and the sender enters the fast recovery state. In this state, if the retransmitted segment is still lost, the sender recognizes the loss event by RTO expiring then enters loss state. After the RTO retransmission, the value of RTO will be exponentially increased if the acknowledgment does not arrive at the sender within an RTO. On the other hand, if the first loss is detected by timeout, sender will also retransmit the lost segment but enter loss state directly. It is worth noting that in loss state, only timeout helps to infer a segment is still lost. According to the analysis in this paragraph, we summarize t can be estimated by RTT, RTO or exponentially increased RTO. 3.2.2. Transfer time in practice It is difficult for the sender of MPTCP’s subflow to obtain the accurate E[t], p and d1 . We will show our way to dynamically estimate them. Based on what we analysed above, the mean of time interval to judge whether a packet is received, or E[t], has three candidates: 4 ACK loss events after the delivered segment may lead to more retransmission, but it is hard to speculate whether a ACK is delivered or lost because both TCP and MPTCP do not acknowledge ACK packets.

RTT, RTO and exponentially increased RTO. In Linux kernel, RTO and exponentially increased RTO are implemented as one variable. We redo the experiment in Section 2.1 and print out RTO variable. Then we find that RTO is quite sensitive to the losses and, because of the exponentially increases, RTO is usually much larger than RTT, even when the packet loss rate is low and at that time E[t] ≈ RTT for most packets. This happens much more frequently when subflow’s RTT is less than RT Omin , which is 200 ms in Linux kernel. On the other hand, RTT is quite stable and does not have minimum value limitation. As a result, we use RTT to estimate E[t] in LAMPS. Packet loss rate is always a variable value in a process of transportation. We denote by n1 the number of packets that were successfully delivered between two loss events, and by n2 the number of packets that are successfully delivered after the last loss event. Then packet loss rate can be calculated by the following equation:

p=

1 max(n1 , n2 )

(4)

Inspired by the standard method used to estimate RTT [24], we utilize Eq. (5) to update n1 when loss happens:

n1 = (1 − α ) × n1 + α × sample_n1

(5)

where sample_n1 is the number of packets that were successfully delivered between the last two loss events, and 0 ≤ α ≤ 1 is an empirical parameter, which means the weight of the new sample against the past estimation. The value of α affects the choice of state variable n1 . A higher α means n1 should be updated more frequently to accommodate the high variation in loss rate, and vice versa. Our evaluation in Section 4.3.4 confirms these discussions. α should be implemented as a configurable parameter and we choose 1 as its default value based on two reasons: First, all the use cases we are concerned with are lossy, but mostly in a bursty fashion, which means packet loss rate changes quickly. Using the most recent sample makes n1 more sensitive to the variation. Second, experiment results in Section 4.3.4 confirm our viewpoint. With packet loss rate rapidly changing, the estimated value is the most accurate when α = 1. Many researchers have proposed schemes to estimate one-way delay in TCP [25,26], which can be used to estimate d1 in Eq. (3). Unfortunately, one-way delay estimation is not available in current Linux kernel. In our implementation, d1 is substituted by rtt 2 . From the above, we substitute Eqs. (2) and (4), E[t] = rtt, d1 = rtt 2 into Eq. (3) and use the result to estimate subflow’s transfer time in LAMPS:

T =

max(n1 , n2 ) + 1 × rtt 2(max(n1 , n2 ) − 1 )

(6)

3.3. Implementation We have implemented LAMPS in Linux kernel v3.18 based on the modular scheduler framework in MPTCP v0.90 [13] and our code is available online [16]. The pseudo-code of LAMPS is given by Algorithm 1. Algorithm 2 demonstrates the TRANSFER_TIME() function, which is called in Algorithm 1. Function LAMPS_Main() is invoked when MPTCP connection is ready to send new data. LAMPS selects the best subflow based on the transfer time, which is estimated by Eq. (6) for the subflows have experienced packet loss or rtt 2 for the subflows have never experienced packet loss. Then LAMPS picks packet according to the different subflow’s state. Subflows record the last sent packet’s sequence number and use it to choose packets when themselves are in REDUNDANT state. The estimate of each subflow’s packet loss rate is updated when a loss event happens based on Eqs. (4) and (5). The estimation

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

Algorithm 1 The pseudo-code of LAMPS. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37:

function Subflow_Initialization(snd_una) mark1 ⇐ snd_una mark2 ⇐ snd_una n⇐0 last _ pkt _seq = 0 state = NORMAL end function function New_Loss_Happen(snd_una, new_cc_state) mark1 ⇐ mark2 mark2 ⇐ snd_una sample_n ⇐ mark2 − mark1 + 1 n ⇐ sample_n × α + n × (1 − α ) return end function function LAMPS_main(available_subflow_set) best _sub f low ⇐ null t mp_t ime ⇐ ∞ for each subflow s ∈ avail abl e_sub f l ow_set do if T RANSF ER_T IME (s ) < t mp_t ime then t mp_t ime ⇐ T RANSF ER_T IME (s ) best _sub f low ⇐ s end if end for if Match NORMAL-to-REDUNDANT condition then best _sub f low.state ⇐ REDUN DAN T else best _sub f low.state ⇐ NORMAL end if if best _sub f low.state = REDUN DAN T then pkt ⇐ the pkt behind best _sub f l ow.l ast _ pkt _seq else pkt ⇐ the next pkt never sent by other NORMAL sub f lows in send queue end if best _sub f l ow.l ast _ pkt _seq ⇐ pkt.seq return best _sub f low, pkt end function

Algorithm 2 Function: TRANSFER_TIME(s). 1: 2: 3: 4: 5: 6: 7: 8:

function TRANSFER_TIME(s) if no pkt lost on subflow s before then return rtt 2 else s.n ⇐ max(s.n, s.snd_una − s.mark2 ) +1 return 2(s.n × rtt s.n−1 ) end if end function

precess is shown in Function New_Loss_Happen(), which is called when a loss event happens on any subflow. Recent versions of Linux adapt the duplicate ack threshold based on the observed reordering. If the stack thinks a connection is in loss state, it will call our function, which means we leave the judgement that whether loss event happens to the stack itself. Several variables are used to estimate the loss rate of each subflow: variable mark1 and mark2 are the penultimate and the last sequence number of the lost segments respectively. n is regarded as the inverse of the loss rate. Variable snd_una saves the sequence number of the first segment which subflow needs an ack for. For brevity, the unit of the sequence number is segment in our pseudo-code. Smoothing parameter α is implemented as a configurable parameter, which is set to 1 based on the discussion in Section 3.2.

151

4. Evaluation We conduct real-world, testbed and emulation experiments to evaluate the performance of LAMPS. In real-world experiments, MPTCP with multiple schedulers is evaluated in high speed rails case. Since it is difficult to obtain path properties (e.g. handoff events) in real-world experiments, we also perform testbed and emulation experiments in order to evaluate LAMPS in more controllable networks. Testbed experiments are for CBR traffic and bulk traffic, while emulation experiments include more cases such as DASH traffic experiments, more path experiments, etc. We employ the default MPTCP congestion control algorithm, LIA [22] to conduct experiments in this paper, unless otherwise stated. Some measurement-based studies [27] have shown that packet loss rate is usually less than 5%. We use this empirical value to distinguish normal congestion losses and burst random losses, then we configure the NORMAL-to-REDUNDANT condition by packet loss rate exceeding 5% in all experiments. In the testbed and emulation experiments, we use 20% as the packet loss rate or the maximum of the packet loss rate, because it is a common value for high packet loss rate cases such as HSPA+ networks on HSRs [1]. Table 1 has already shown that such high packet loss rate is unneglectable in urban areas. 4.1. Real-world experiments We use a laptop as the client on a high speed train, and the laptop has two USB-cellular modems to access two distinct 4G cellular networks. The server is located in our campus network. Both the client and server run Ubuntu 16.04 with MPTCP v0.90 [13]. Our real-world experiments are conducted when the train speed is more than 250 km/h. We employ the same application in Section 2.1 to generate CBR traffic, and each MPTCP connection has two subflows using the two cellular networks respectively. In our experments, we run four application instances at the same time on the laptop, and each application instance builds a MPTCP connection with a different scheduler. All the connection last for 15 min. According to the oberservations in a measurement study [10], the handoff frequency of a cellular carrier can be more than 5 per minute with a probability of 80%, and the packet loss rate can surpass 70%, when a handoff event happens. Fig. 6 depicts the CDF of the application delay for all data blocks. RDDT achieves the lowest

Fig. 6. CDF of the application delay for all data blocks in real-world CBR traffic experiments.

152

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158 Table 5 Reinjected packet ratio in CBR traffic experiment.

Table 3 Reinjected packet ratio in real-world CBR traffic experiment. Scheduler

LowRTT

RR

RDDT

LAMPS

Scheduler

LowRTT

RR

RDDT

LAMPS

Reinjected packet ratio

0.03

0.02

0.78

0.05

Reinjected packet ratio

0.02

0.03

0.75

0.04

application delay, and LAMPS has a similar performance as RDDT. Both RDDT and LAMPS defeat LowRTT and RR. Table 3 shows the reinjected packet ratio. LAMPS reduces about ~15x reinjected packets than RDDT. Ideally, each packet is sent only once, but because of MPTCP’s RTO retransmission and RDDT redundantly transmission, some packets are reinjected consuming more bandwidth. From our results, we conclude that LAMPS reduces extra bandwidth consumption largely while keeps near the best application delay. We replace the dummy application with iPerf to evaluate LAMPS’ performance for bulk traffic. The other settings are the same with the former CBR traffic experiments. LowRTT, RR and LAMPS achieves similar mean goodputs, 16.4 Mbps, 15.8 Mbps, 16.2 Mbps, respectively, while RDDT only obtains 8.4 Mbps. With LAMPS, MPTCP still can leverage multiple paths’ bandwidth, but MPTCP with RDDT can only utilize one path’s bandwidth. 4.2. Testbed experiments We use the same testbed in Section 2 and the topology is shown in Fig. 1(a). 4.2.1. CBR Traffic We use the same application in Section 2.1 to generate CBR traffic and there are two subflows (S1 and S2) established between S and D. Both paths’ bandwidths are 10 Mbps. S1’s RTT, packet loss rate and S2’s RTT are fixed. S2’s packet loss rate is changed every 10 s. The link properties are shown in Table 4. We repeat this experiment 10 times for each scheduler. Fig. 7 shows

Table 4 Link parameters of CBR traffic experiment. Link 1 Time(s) RTT(ms) Loss Rate(%)

0-50 200 0

Link 2 0-10

10-20

0

20

20-30 150 0

30-40

40-50

20

0

Fig. 7. Mean application delay every 10 s. The parameters of subflow 1 keep same (200 ms RTT, 0% packet loss rate). The RTT of S2 remains 150 ms, while loss rate varies every 10 s according to Table 4. Each value represents the mean over all the runs. The error bars show the standard deviations around the means.

Table 6 Reinjected packet ratio in bulk traffic experiment. Scheduler

LowRTT

RR

RDDT

LAMPS

Reinjected packet ratio

0.02

0.01

0.25

0.02

the mean application delay and Table 5 shows mean reinjected packet ratio. LAMPS achieves a similar performance as RDDT, while reducing ~18x reinjected packet amount than RDDT. Ideally, each packet is sent only once, but because of MPTCP’s RTO retransmission and RDDT redundantly transmission, some packets are reinjected consuming more bandwidth. From our results, we conclude that LAMPS reduces extra bandwidth consumption largely. Compared to the default scheduler, LAMPS improves the application delay by ~31% and ~27% for different lossy phases, and achieves more stable delay. 4.2.2. Bulk traffic Then we evaluate LAMPS’ performance for bulk traffic. The experiment settings are the same with those in Section 2.2. Fig. 3 shows that using LAMPS will not decrease the goodput for bulk traffic. With LAMPS, MPTCP still has the ability to leverage multiple paths’ bandwidth. However, MPTCP with RDDT can only utilize one path’s bandwidth. We also calculate the mean reinjected packet ratio shown in Table 6. While LowRTT, RR and LAMPS have similar reinjected packet ratio, RDDT increases more than 12x extra bandwidth consumption without fully using paths’ bandwidth. 4.3. Emulation experiments To emulate different network environments, we adopt Mininet [28] to conduct evaluations, which allows us to generate a virtual network with Mininet hosts running MPTCP v0.90 [13]. In this subsection, experiments use the topology shown in Fig. 1(b) except Section 4.3.2. 4.3.1. Real video streaming traffic Since the majority of video streaming today is HTTP adaptive bitrate streaming, we evaluate LAMPS’ performance with real video application in this part. We set up an Apache2 server on the sender to provide Dynamic Adaptive Streaming over HTTP (DASH), which is the standard of video streaming [29]. A video named Big Buck Bunny from a public dataset [30] is employed in our experiment, and the receiver runs MP4Client [31] to stream the video. The link capacities are set to 10 Mbps and we set the first link’s RTT to 200 ms without packet loss rate. The second link’s RTT is set to 150 ms and it has periodic 20% packet loss rate, which occurs once every 20 s and continues 10 s each time. MPTCP generates two subflows running on the two links respectively. We analyze the log file from MP4Client. Table 7 shows memory consumption of the video client. LAMPS and Redundant are much better than LowRTT and RR. Moreover, LAMPS achieves a similar performance as RDDT. For LowRTT and RR, wrong packet scheduling leads to more out-of-order deliveries resulting in Head-of-Line blocking problem, which increases memory usage. Table 7 also shows the reinjected packet ratio for all schedulers. Compared to LowRTT and RR, LAMPS increases extra bandwidth consumption (reinjected packet ratio) slightly while guarantees much lower memory consumption. Compared to RDDT, LAMPS decreases 25%

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

153

Fig. 8. CDF of Delay Increase Ratio for All Parameter Settings for Each Topology.

Table 7 Memory consumption and Reinjected Packet Ratio in video streaming experiment.

LowRTT RR Redundant LAMPS

Max mem (MB)

Mean mem (MB)

Reinjected Packet ratio

71.5 71.6 24.1 22.1

48.9 49.1 19.8 19.5

0.25 0.25 0.40 0.30

reinjected packet ratio, because LAMPS only redundantly sends packets when a subflow is in REDUNDANT state. Additionally, we find that in our experiment, the FPS (frames per second) is quite stable for all schedulers. The reason is that the first link’s capacity is large enough for streaming, which means even if some packets are lost on the second path, they can be reinjected to the first one by MPTCP after RTO. 4.3.2. More paths and cover-all parameters In this part, we would like to know the LAMPS’ general performance for CBR traffic. We design an experiment framework first. In each experiment, each path has a fixed RTT and after the experiment starts s seconds, the packet loss rate surges to l and maintains t seconds for each path. Each experiment establishes one MPTCP flow, which continues 30s and produces CBR traffic. The domains of all parameters in experiment framework are shown in Table 8. As Paasch et al. [20] suggests, we adopt WSP algorithm [32], which is a space-filling algorithm to generate parameter sets to do experiments. We not only conduct 2-path experiments, but also add 3 and 4-path experiments. The topologies for 3 and 4-path experiments are similar to Fig. 1(b), except that there are 3 and 4 paths between the source and the destination. With the WSP algorithm, we generate 90, 67, and 57 parameter sets for 2–4 path experiments respectively. We repeat all experiment settings 5 times. To compare the results, we first find the minimum application delay in each setting, and employ the ratio of the average application delay of a scheduler to the minimum as the result of the scheduler in a specific parameter setting. We call the ratio delay increase ratio. For example, in all experiments configured by the same parameter setting, if the minimum application delay for all

Table 8 Domains of parameters for cover-all experiments. Parameter

Min

Max

RTT (ms) Loss rate l(%) Start time s(s) Continuous period t(s)

50 0 0 0

10 0 0 20 20 10

Table 9 Mean delay increase ratio for all parameter settings.

2-path 3-path 4-path

LowRTT

RR

RDDT

LAMPS

1764 1058 3773

2256 3556 14,591

1678 1101 2459

1596 1008 2565

Table 10 Mean reinjected packet ratio.

2-path 3-path 4-path

LowRTT

RR

RDDT

LAMPS

0.021 0.022 0.025

0.025 0.041 0.069

0.775 1.13 1.59

0.095 0.12 0.18

schedulers is 50 ms and the mean application delay of LowRTT is 75 ms, the delay increase ratio of LowRTT in that parameter setting is 1.5. Fig. 8 shows the CDF of delay increase ratio for all parameter settings for each topology. As we can see in Fig. 8, it is not easy to compare the CDF of LowRTT, RDDT and LAMPS in the cover-all experiments, because the packet loss rate is not limited to very high value. Comparing Fig. 8(a), (b) and (c), we observe the delay becomes higher with more paths, and RR performs worse with more paths. Table 9 shows the mean delay increase ratio for all parameter settings. LAMPS achieves 15%–22% mean application delay reduction compared to LowRTT. The results of reinjected packet ratio in Table 10 show that LAMPS reduces ~10x extra banwidth occupation that RDDT needs for all three group experiments. 4.3.3. More recent MPTCP schedulers In recent years, more MPTCP schedulers have been proposed. In order to have a complete evaluation of LAMPS, we introduce two new MPTCP schedulers, BLEST [23] and ECF [33], to make a comparison. They have been implemented in Linux kernel by their authors, but they are not included in the MPTCP implementation [13] now. BLEST minimizes application delay by reducing the number of spurious retransmissions and ECF more efficiently leverages the faster paths by determining if employing a slow path will make faster paths become idle. As shown in [23] and [13], BLEST and ECF outperform LowRTT, but they have not been evaluated over highly lossy networks. We reconduct the two experiments5 in Section 4.2 using Mininet [28]. As can be observed in Fig. 9, ECF and BLEST achieve similar performance as LowRTT for CBR traffic. The reasons are two-fold. First, since ECF and BLEST do not take packet loss rate into account when they opt for subflow, they still prefer to use the same 5

We use the implementations of BLEST and ECF published in [23] and [33].

154

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158 Table 12 Link parameters. Link 1 RTT(ms) Loss rate(%)

0-105s 200 10

Link 2 0-15s 100 5

15-30s 100 10

30-45s 100 20

Link 2 RTT(ms) Loss rate(%)

Fig. 9. Mean application delay every 10 s. The parameters of subflow 1 keep same (200 ms RTT, 0% packet loss rate). The RTT of S2 remains 150 ms, while loss rate varies every 10 s according to Table 4. Each value represents the mean over all the runs. The error bars show the standard deviations around the means.

Fig. 10. The goodput of MPTCP connection. Link parameters are changed every 10 s. S2 suffers bursty packet losses in 10–20 s and 30–40 s. Each value represents the mean over all the runs. The error bars show the standard deviations around the means. Table 11 Reinjected packet ratio in CBR and bulk traffic experiments.

CBR Bulk

LowRTT

BLEST

ECF

LAMPS

0.022 0.023

0.031 0.032

0.025 0.031

0.040 0.041

subflow that LowRTT employs. Second, they do not leverage redundantly sending to improve the transmission. Fig. 10 shows that the goodput is similar for all schedulers and the result conforms our opinion in Section 2.2 that it is hard for schedulers to benefit bulk traffic. Table 11 shows the comparison of reinjected packet ratio. As we can see, LAMPS sends a little more reinjected packets than other schedulers. LAMPS achieves better performance because of its novel method to select subflow and its redundantly sending component. 4.3.4. Packet loss rate estimation In this part, we focus on the influence of different smoothing parameter alpha on packet loss rate estimation. First of all, we add the code to estimate packet loss rate into the default scheduler, LowRTT [12] and we use the modified LowRTT in this subsection. We only add packet loss rate estimation code and do not touch the scheduling function of LowRTT. We generate both bulk and CBR traffic in this part. Application runs 105 s in each experiment and we repeat emulation 10

45-60s 100 30

60-75s 100 20

75-90s 100 10

90-105s 100 5

times for each case. Path properties are shown in Table 12. The packet loss rate of path 2 changes its value every 15 s. In addition, the bandwidths are large enough and cannot be the bottleneck of transmission. We use printk fucntion to obtain logs from running Linux kernel. Figs. 11 and 12 show the results. Comparing the results from different types of traffic with same alpha value, we observe that types of flows have no effect on packet loss rate estimation as long as there are enough packets sent into the networks. As we can see in Figs. 11 and 12, higher alpha makes the estimation of packet loss rate more sensitive to fast changes and we think al pha = 1 is suitable for networks with bursty losses. 4.3.5. Burst time scale Then, we focus on the LAMPS’ behavior with different burst durations. We set up the following experiment. The two paths’ RTTs are 500 ms and 400 ms. At the start, both paths are not configured packet loss rate. Loss event happens at 7th second and after that, the packet loss rate of path 2 will increases from 0 to 20%. The application generates CBR traffic. We vary the loss time duration from 0.2 s to 10 s. As can be observed in Fig. 13(a), we find that LAMPS’ performance is the same as LowRTT’s when burst duration is less than one RTT. The reason is that LAMPS still uses ACKs to gather path information, so it cannot find loss event happened less than one RTT. However, LAMPS’s performance is not worse than LowRTT’s in Fig. 13(a). Comparing Fig. 13(a) and (b), we observe that LAMPS improves application delay when the burst duration increases and LAMPS’ performance is no worse than LowRTT’s in all cases. 4.3.6. Congestion control algorithms Since most MPTCP congestion control algorithms react to packet loss events, it is necessary to show whether LAMPS performs well under different congestion control algorithms. We reconduct the first two experiments in Section 4.2 using Mininet [28]. However, this time, we focus our attention on the LAMPS’ performance with all MPTCP congestion control algorithms implemented in Linux kernel [13]: LIA [22], OLIA [34], BALIA [35] and WVEGAS [36].6 Fig. 14 shows that although different congestion control algorithms are used, LAMPS still reduces application delay compared to LowRTT in lossy phases and achieves the most similar behavior with RDDT for CBR traffic. Fig. 15 indicates that for bulk traffic, LAMPS still has the ability to leverage multiple paths’ bandwidths with different congestion control algorithms. We also calculate the mean reinjected packet ratio for both traffic. LAMPS reduces more than 10x extra bandwidth consumption in both bulk traffic and CBR Traffic for all congestion control algorithms. Table 13

6 In our experiments, the parameters of wVegas (initial α , total α and γ ) are set to their default values.

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

155

Fig. 11. The influence of different alpha on packet loss rate estimation for CBR flows. Dashed lines are exact values of loss rate.

Fig. 12. The influence of different alpha on packet loss rate estimation for bulk flows. Dashed lines are exact values of loss rate.

Table 13 Mean Reinjected packet ratio in CBR traffic experiment. Reinjected packet ratio

LowRTT

RR

RDDT

LAMPS

OLIA BALIA WVEGAS

0.02 0.03 0.02

0.03 0.02 0.02

0.83 0.75 0.76

0.06 0.06 0.03

Fig. 13. LAMPS’ performance with different burst lossty durations.

Fig. 14. Different congestion control algorithms with different schedulers for CBR traffic. The experiment settings are the same as those in Section 4.2. Each value represents the mean over all runs. The error bars show the standard deviations around the means.

Fig. 15. Different congestion control algorithms with different schedulers for bulk traffic. The experiment settings are the same as those in Section 4.2. Each value represents the mean over all runs. The error bars show the standard deviations around the means.

156

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158 Table 14 Reinjected packet ratio in CBR and bulk traffic experiments.

shows the reinjected packet ratio for CBR traffic and the result for bulk traffic is similar. We conclude that LAMPS works well with different congestion control algorithms for different traffic patterns. 4.3.7. Packets losses following Gilbert–Elliott loss model In previous testbed and emulation evaluations, we use tc [21] to create random packet losses. In this subsection, we generate packet losses following the Gilbert–Elliott loss model [37,38]. The Gilbert– Elliot model is usually employed to simulate bursty loss patterns. The tc program also supports Gilbert–Elliot model to add packet losses. We reconduct the two experiments in Section 4.2 with Mininet [28]. When we need to add bursty packet losses, we set the Gilbert–Elliot model with p = 50%, r = 50%, 1 − h = 20%, and 1 − k = 0%, because we find the average packet loss rate in such setting is near to 20% in our emulations. Figs. 16, 17 and Table 14 show the results. We find they are quite like the results in Section 4.2. Fig. 16 shows that although the Gilbert–Elliott loss model is used, LAMPS still reduces application delay compared to LowRTT in bursty loss phases, and achieves the most similar behavior with RDDT for CBR traffic. Fig. 17 depicts that for bulk traffic, LAMPS keeps similar performance as

CBR Bulk

LowRTT

RR

RDDT

LAMPS

0.025 0.024

0.032 0.012

0.78 0.31

0.041 0.042

in the testbed experiments in Section 4.2, and still can leverage multiple paths’ bandwidths. We also compute the mean reinjected packet ratio for both traffic. LAMPS reduces more than 10x extra bandwidth consumption in both bulk traffic and CBR Traffic. Table 14 shows the reinjected packet ratio for CBR traffic, and the result for bulk traffic is similar. We conclude that LAMPS works well with packet losses following the Gilbert–Elliott loss model. 5. Analysis We complement our experimental results by analyzing LAMPS in this section. Our evaluation shows that LAMPS performs better than LowRTT in lots of scenarios and they also yield similar results sometimes. In order to understand the reasons why LAMPS achieves better performance over highly lossy networks, we consider a typical multipath communication scenario and model the complementation time of a data block. Our analysis explains the two design ideas of LAMPS, considering both packet loss rate and RTT to choose subflow and conditional redundant sending, make LAMPS outperform LowRTT. LowRTT selects the subflow with lowest RTT among the subflows which have available congestion window space. Let S be the set of all subflows with free congestion window space. LowRTT uses Eq. (7) to choose the subflow to be used and LAMPS employs Eq. (8).

arg min rtti

(7)

i∈S

arg min i∈S

Fig. 16. Mean application delay every 10 s for CBR traffic. The parameters of subflow 1 keep same (200 ms RTT, 0% packet loss rate). The RTT of S2 remains 150 ms, while loss rate varies every 10 s. S2 suffers bursty packet losses following Gilbert– Elliott loss model in 10–20 s and 30–40 s. Each value represents the mean over all the runs. The error bars show the standard deviations around the means.

pi + 1 rtti 2 ( 1 − pi )

(8)

Consider the typical scenario shown in Fig. 18, where two paths are between the source and destination. Two subflows run on the two paths respectively. The selected subflows can be different for different schedulers. If the rtts and packet loss rates meet the conditions described in the shadow regions of Fig. 19, LAMPS and LowRTT would employ different subflows. LAMPS performs better than LowRTT in lots of scenarios partly because they choose to use diverse subflows. Moreover, if the redundant sending is not triggered, LAMPS’ different subflow choice is the only reason why LAMPS is better than LowRTT. Next, we consider the situation where one subflow in typical scenario is in REDUNDANT state. Without loss of generality, we assume subflow 2 is in REDUNDANT state. p +1 We first study the case that rtt1 > rtt2 and 2(11−p ) rtt1 < p2 +1 rtt2 , 2(1−p2 )

1

which implies p1 < p2 . In such case, the preferred sub-

flows are different for LowRTT and LAMPS. LowRTT chooses to use

Fig. 17. The goodput of MPTCP connection for bulk traffic. Link parameters are changed every 10 s. S2 suffers bursty packet losses following Gilbert–Elliott loss model in 10–20 s and 30–40 s. Each value represents the mean over all the runs. The error bars show the standard deviations around the means.

Fig. 18. A typical scenario for multipath communication.

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

157

6. Related work

Fig. 19. Different path properties make the scheduler use different subflows. LAMPS and LowRTT use different subflows in shadow regions and use the same subflow in non-shadow ones.

subflow 1 while LAMPS chooses to employ subflow 2. For a data block consisting of m segments, the lower bound of completion time for LowRTT can be modeled as



CTlowrtt > max

m−a

p2 + 1 a p1 + 1 rtt2 , rtt1 2 ( 1 − p ) 2 ( 1 − p1 )   2 cwnd cwnd 2 1



(9)

 is the average congestion window size in this period where cwnd and a segments are assigned to subflow 1. We use the transfer time defined in Section 3.2 to estimate the delivery time of any window of the data. For LAMPS, the upper bound can be modeled as

CTlamps <

p1 + 1 rtt1 2 ( 1 − p1 )  cwnd 1 m

(10)

Subflow 2 is in REDUNDANT state, so it would help to make CTlamps less than the case where subflow 1 is the only subflow in use. For CBR traffic, the windows of two subflows are usually not full. In such case, LowRTT uses the first subflow to send most data, so m − a ≈ 0 and a ≈ m. Then we have

m−a

p2 + 1 a p1 + 1 rtt2 < rtt1 2 ( 1 − p ) 2 (   2 cwnd2 cwnd1 1 − p1 )

(11)

m a p1 + 1 p1 + 1 rtt1 ≈ rtt1 2 ( 1 − p ) 2 (   1 cwnd1 cwnd1 1 − p1 )

(12)

From Eqs. (9)–(12), we draw a conclusion:

C Tlamps < C Tlowrtt

(13)

For bulk traffic, it is hard to compare m and m − a, so we cannot draw the same conclusion from Eqs. (9)–(12). Then, we study other cases. For the case that rtt1 > rtt2 and p1 +1 p +1 rtt1 > 2(12−p ) rtt2 , path 2 is worse than path 1, then sub2(1−p ) 1

Various studies have explored MPTCP’s scheduling problem. But these researches have neither been based on MPTCPâs implementation nor discussed highly lossy networks. Singh et al. [39] compare several early schedulers of MPTCP, where heterogeneous paths are considered. However, none of schedulers discussed in [39] takes loss into account and the experiments are based on MCTCP [40], which is different from MPTCP in many aspects. Kuhn et al. [41] focus on multiple heterogeneous wireless network access and try to reduce the receiver’s buffer occupancy. The scheme considers delay and congestion window sizes and does not consider the packet loss rate. Moreover, the scheme is implemented in ns2 simulator. Paasch et al. [20] show that compared to Round-Robin, LowRTT has better performance in CBR traffic in their experiments. However the authors only consider WLAN and 3G environments, where the packet loss rate is low and caused by congestion in general. Ferlin et al. [23] propose a send-window blocking estimation scheduler, named BLEST, to minimize application delay by reducing the number of spurious retransmissions. The heterogeneous networks in the paper are 3G+WLAN networks, which are different from the lossy networks discussed in our paper. The authors have implemented BLEST in Linux kernel, but now it is not included in [13]. Corbillion et al. [42] present an MPTCP scheduler to improve video streaming performance. The scheme is not implemented in real system and it is evaluated via simulation. In addition, their approach needs to modify the video application and does not consider highly lossy networks. Nikravesh et al. [43] conduct a measurement study based on a campus network, and propose MPFLEX, which is a new architecture for multipath communication. MPFLEX is not compatible with MPTCP and decouples the scheduling algorithm and OS protocol implementation. MPFLEX is not designed for highly lossy networks. Han et al. [44] present MP-DASH, a framework to schedule video traffic over MPTCP. MP-DASH can significantly reduce cellular usage and energy consumption with negligible degradation of QoE. However, their scheme does not consider highly lossy networks and requires modifications for video applications and MPTCP. Lim et al. [33] find that web browsing and video streaming usually create traffic composed of multiple uploads or downloads. They also observe that the LowRTT cannot use some paths with such a traffic pattern efficiently. ECF employs the information of RTT estimates, congestion windows and the amount of data queued in the send buffer to determine if employing a slow path will make faster paths become idle. Their evaluation shows that ECF uses all paths more efficiently and outperforms LowRTT for Web and video traffic, however, they do not evaluate ECF over highly lossy networks. The approach is implemented in Linux kernel, but now it is not included in [13].

7. Conclusion

2

flow 1 would be also in REDUNDANT state. LAMPS turns into RDDT, which achieves best performance for CBR traffic. For the case that p +1 p +1 rtt1 < rtt2 and 2(11−p ) rtt1 > 2(12−p ) rtt2 , subflow 1 is also in RE1

2

DUNDANT state in our evaluation. Then LAMPS also turns into p +1 p +1 RDDT. For the case that rtt1 < rtt2 and 2(11−p ) rtt1 < 2(12−p ) rtt2 , 1 2 both LAMPS and LowRTT prefer subflow 1 to send segments. In such case, for LowRTT, subflow 2 slows down the whole transmission. LAMPS would not encounter the same problem because it assigns all segments to subflow 1 and only schedule duplicate segments to subflow 2.

A variety of wireless communication links today are suffering high loss rates. In such lossy environments, MPTCP offers a solution for more robust data transmissions than regular TCP. However, we find that existing schedulers do not work well in lossy networks for both video and bulk traffic. Then we design and implement a scheduler, named LAMPS for MPTCP. Our experiment results show that LAMPS preserves application latency, keeps low memory consumption, and significantly reduces extra bandwidth consumption at the mean time in the presence of high packet loss rate.

158

E. Dong, M. Xu and X. Fu et al. / Computer Networks 157 (2019) 146–158

Conflict of interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgment The research is supported by the National Natural Science Foundation of China under Grant (61625203 and 61811530336) and the Science & Technology Program of Beijing (Z17110 0 0 052170 01). References [1] L. Li, K. Xu, D. Wang, C. Peng, Q. Xiao, R. Mijumbi, A measurement study on the TCP behaviors in HSPA+ networks on high-speed rails, in: IEEE Infocom, 2015, pp. 2731–2739. [2] F. Tso, J. Teng, W. Jia, D. Xuan, Mobility: a double-edged sword for HSPA networks: a large-scale test on Hong Kong mobile HSPA networks, in: ACM Mobihoc, 2010, pp. 81–90. [3] D.N. Cottingham, I.J. Wassell, R.K. Harle, Performance of IEEE 802.11a in vehicular contexts, in: IEEE VTC, 2007, pp. 854–858. [4] C. Steger, P. Radosavljevic, J.P. Frantz, Performance of IEEE 802.11b wireless LAN in an emulated mobile channel, in: Semiannual Vehicular Technology Conference, vol. 2, 2003, pp. 1479–1483. [5] Project Loon, 2019. http://www.google.com/loon/. [6] Y. Lee, S. Winkler, Effects of rain attenuation on satellite video transmission, in: IEEE VTC, 2011, pp. 1–5. [7] T. Ma, Y. Lee, S. Winkler, M. Ma, QoS provisioning by power control for video communication via satellite links, Int. J. Satell. Commun. Netw. 33 (3) (2014) 259–275. [8] A. Ford, C. Raiciu, M. Handley, O. Bonaventure, TCP Extensions for Multipath Operation with Multiple Addresses, 2013. (RFC 6824). [9] O. Bonaventure, C. Paasch, G. Detal, Use Cases and Operational Experience with Multipath TCP, 2017. (RFC 8041). [10] L. Li, K. Xu, T. Li, K. Zheng, C. Peng, D. Wang, X. Wang, M. Shen, R. Mijumbi, A measurement study on multi-path TCP with multiple cellular carriers on high speed rails, in: ACM SIGCOMM, 2018, pp. 161–175. [11] Thrane Thrane Explorer BGAN Terminals, 2017. http://explorersatellite.com/ BGAN/thrane_thrane_bgan.html. [12] C. Paasch, S. Ferlin, O. Alay, O. Bonaventure, Experimental evaluation of multipath schedulers, in: ACM SIGCOMM Workshop on CSWS, 2014, pp. 27–32. [13] C. Paasch, S. Barre, et al., Multipath TCP in the linux kernel, 2019. http://www. multipath-tcp.org. [14] A. Frömmgen, T. Erbshäußer, T. Zimmermann, K. Wehle, A. Buchmann, Remp TCP: Low Latency Multipath TCP, in: IEEE ICC, 2016, pp. 1–7. [15] I. Lopez, M. Aguado, C. Pinedo, E. Jacob, SCADA systems in the railway domain: enhancing reliability through Redundant multipath TCP, in: 2015 IEEE 18th International Conference on Intelligent Transportation Systems, 2015, pp. 2305–2310. [16] The Implementation and Experiment of Lamps, 2019. https://github.com/ lampsmptcp/lamps. [17] M. Scharf, S. Kiesel, Head-of-line blocking in TCP and SCTP: analysis and measurements, in: GLOBECOM, 2006, pp. 1–5. [18] Global Internet Phenomena Report, 2017. https://www.sandvine.com/ downloads/general/global- internet- phenomena. [19] White paper: Cisco VNI forecast and methodology, 2015–2020, 2017. [20] C. Paasch, R. Khalili, O. Bonaventure, On the benefits of applying experimental design to improve multipath TCP, in: ACM CoNEXT, 2013, pp. 393–398. [21] Linux tc Utility Program, 2019. https://linux.die.net/man/8/tc. [22] C. Raiciu, M. Handly, D. Wischik, Coupled Congestion Control for Multipath Transport Protocols, 2011. [23] S. Ferlin, O. Alay, O. Mehani, R. Boreli, BLEST: blocking estimation-based MPTCP scheduler for heterogeneous networks, in: IFIP Networking, 2016, pp. 431–439. (RFC 2988). [24] V. Paxson, M. Allman, Computing TCP’s Retransmission Timer, 20 0 0. [25] J. Choi, C. Yoo, One-way delay estimation and its application, Comput. Commun. 28 (7) (2005) 819–828. [26] A. Hernandez, E. Magana, One-way delay measurement and characterization, in: IEEE ICNS, 2007, p. 114. [27] Y.-C. Chen, Y. Lim, R.J. Gibbens, E.M. Nahum, R. Khalili, D. Towsley, A Measurement-based Study of multipath TCP performance over wireless networks, in: ACM IMC, 2013, pp. 455–468. [28] N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, N. McKeown, Reproducible network experiments using container-based emulation, in: ACM CoNEXT, 2012, pp. 253–264. [29] I. Sodagar, The MPEG-DASH standard for multimedia streaming over the internet, IEEE Multimed. 18 (4) (2011) 62–67. [30] S. Lederer, C. Müller, C. Timmerer, Dynamic adaptive streaming over HTTP dataset, in: ACM MMSys, 2012, pp. 89–94. [31] Gpac Dash Video Player, 2017. https://gpac.wp.mines-telecom.fr/.

[32] J. Santiago, M.Claeys-Bruno, M.Sergent, Construction of space-filling designs using WSP algorithm for high dimensional spaces, Chemom. Intell. Lab. Syst. 113 (2012) 26–31. [33] Y. Lim, E. Nahum, D. Towsley, R. Gibbens, ECF: an path scheduler to manage heterogeneous paths, in: ACM CoNEXT, 2017, pp. 33–34. [34] R. Khalili, N. Gast, M. Popovic, J.-Y.L. Boudec, MPTCP is not pareto-optimal: performance issues and a possible solution, IEEE/ACM TON 21 (5) (2013) 1651–1665. [35] Q. Peng, A. Walid, S.H. Low, Multipath TCP algorithms: theory and design, in: ACM SIGMETRICS, 2013, pp. 305–316. [36] Y. Cao, M. Xu, X. Fu, Delay-based congestion control for multipath TCP, in: ICNP, 2012, pp. 1–10. [37] E.N. Gilbert, Capacity of a burst-noise channel, Bell Syst. Tech. J. (1960) 1253–1265. [38] E.O. Elliott, Estimates of error rates for codes on burst-noise channels, Bell Syst. Tech. J. (1963) 1977–1997. [39] A. Singh, C. Goerg, A. Timm-Giel, M. Scharf, T.-R. Banniza, Performance comparison of scheduling algorithms for multipath transfer, in: IEEE GLOBECOM, 2012, pp. 2653–2658. [40] M. Scharf, T. Banniza, MCTCP: a multipath transport shim layer, in: IEEE GLOBECOM, 2011, pp. 1–5. [41] N. Kuhn, E. Lochin, A. Mifdaoui, G. Sarwar, O. Mehani, R. Boreli, DAPS: intelligent delay-aware packet scheduling for multipath transport, in: IEEE ICC, 2014, pp. 1222–1227. [42] X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, G. Simon, Cross-layer scheduler for video streaming over MPTCP, in: ACM MMSys, 2016, p. 7. [43] A. Nikravesh, Y. Guo, F. Qian, Z. Mao, S.S. Subhabrata, An in-depth understanding of multipath TCP on mobile devices: measurement and system design, in: ACM MobiCom, 2016, pp. 189–201. [44] B. Han, F. Qian, L. Ji, V. Gopalakrishnan, MP-DASH: adaptive video streaming over preference-aware multipath, in: ACM CoNEXT, 2016, pp. 129–143. Enhuan Dong received the B.E. degree from Harbin Institute of Technology. He is now a Ph.D candidate in the Department of Computer Science & Technology at Tsinghua University. He was a visiting Ph.D. student at the University of Goettingen in 2016–2017. His major research interests include multipath TCP and datacenter networks.

Mingwei Xu received the B.Sc. degree and the Ph.D. degree from Tsinghua University. He is a full professor in Department of Computer Science at Tsinghua University. His research interest includes computer network architecture, high-speed router architecture and network security. He is a member of the IEEE.

Xiaoming Fu received the Ph.D. degree in computer science from Tsinghua University, China. He is a full professor in the University of GÃüttingen. His research interests are architectures, protocols, and applications for networked systems, including information dissemination, mobile networking, cloud computing, and social networks. He is a member of the IEEE.

Yu Cao received the Ph.D. degree from Tsinghua University. He is a lecturer in National Digital Switching System Engineering & Technological Research Center, China. His research interests are multipath data transfer, data center networks and network security.