- Email: [email protected]
Performance improvements for TCP in mobile networks with high packet delay variations
ITC 17 / J. Moreira de Souza, N.L.S. da Fonseca and E.A. de Souza e Silva (Editors) 9 2001 Elsevier Science B.V. All rights reserved.
517
Performance Improvements for TCP in Mobile Networks with high Packet Delay Variations J. Schiller a, S. Gruhl b, T. Schwabe a and M. SchweigeP ~Dresden University of Technology, Chair for Telecommunications, D-01062 Dresden, Germany bLucent Technologies Niirnberg, Bell Labs Innovations, Thurn- und Taxis-Strat]e 10, D-90411 Niirnberg, Germany Previous TCP performance investigations for wireless systems focus on TCP throughput degradation due to packet loss on the error-prone wireless link. Since cellular mobile networks like GPRS and UMTS include a strong link layer error protection, using adaptive Forward Error Correction and Automatic Repeat Request schemes, a bad quality of the wireless link does not mainly result in IP packet loss but rather in an additional packet delay. Furthermore TCP congestion control has to deal with hand-offs and temporary link layer disconnections in cellular mobile networks leading to a significant packet delay variation and packet disordering. This paper analyzes the performance of TCP in cellular mobile networks, focusing on the packet delay and disordering problem. We find spurious'retransmissions and TCP timeouts being a dominant cause for throughput degradation. Finally we propose an algorithm to filter spurious TCP retransmissions at the Base Station to increase TCP performance and increase overall wireless link capacity. 1. I N T R O D U C T I O N
AND RELATED WORK
The Transmission Control Protocol (TCP) is one of the most important standards for best effort, reliable data transmission. Today's Internet traffic uses predominantly TCP, as for applications like HTTP for Web Browsing, FTP for File Download or SMTP for Electronic Mail Transfer. Any Internet compatible end system supports the TCP/IP protocol suite, usually as part of the operating system. TCP is a protocol of the OSI transport layer and provides reliable end-to-end transport and end system based traffic control functions [1,2]. Reliability is achieved using backward error correction based on a "selective repeat" retransmission strategy with positive acknowledgments. Today's most common TCP implementations support timer based packet loss detection (Retransmission Timeout) as well as acknowledgment indicated loss detection (Fast Retransmit).
518 To avoid network congestion and end system overload TCP performs traffic control. The TCP sender adapts its data transmission rate using a window mechanism. The current window size indicates the number of packets, which are injected into the network per round trip cycle. It is calculated as the minimum of the receiver's window and the so-called congestion window. The receiver window size indicates the instantaneous available buffer space at the TCP receiver and is announced in the returning TCP acknowledgments. The TCP sender adapts its data transmission rate to the current network situation to avoid network congestion using the congestion window. A congestion situation is always assumed when TCP detects a packet loss, either through a Retransmission Timeout or the arrival of 3 duplicate acknowledgments (Fast Retransmit). The TCP sender reacts by reducing the congestion window to its half (Fast Retransmit) or to 1 Maximum Segment Size (Retransmission Timeout). If no packet loss is detected TCP continuously increases the congestion window to maximize the throughput. There are 2 ways to open the congestion window: Slow Start with an exponential and Congestion Avoidance with a linear increase per round trip cycle. Initially, at the beginning of a connection, TCP starts with a congestion window of 1 MSS and uses Slow Start as increase rule. Figure 1 shows the behavior of the TCP congestion window after connection establishment and after detecting packet losses. More detailed information on the TCP window mechanism can be found in [1-3].
,15
~_ 40 "6 E " z .E ._ o0 o
35
PacketError J
ao 25 20
._~ @
,3
10
s
ob0
0
5
10
15
20
25
30
35
40
Round Trip Time
Figure 1. Dynamics of the TCP congestion window TCP was historically designed and optimized for a wire-line network environment. Due to the low bit error rates on wire-line links the primary cause for packet losses is network congestion. Consequently, the congestion control algorithm of TCP directly reacts on packet losses. 1.1. R e l a t e d W o r k Wireless links typically do not offer a low transmission error rate as found in wire-line links. This leads to packet losses which are not caused by network congestion. Investigations with early wireless systems have shown that TCP reacts regularly with an un-
519 necessary reduction of its sending data rate after every loss. During the following slowly increase of the congestion window TCP may not fully utilize the potential available bandwidth. The result is an unnecessary reduction of the end-to-end throughput, and hence, sub-optimal performance [4-8]. Furthermore hand-offs and temporary link layer disconnections in mobile communication systems may lead to packet disordering and sporadic additional delays. TCP reacts with retransmission timeouts, spurious packet transfer and unnecessary reduction of its congestion window [9]. Recently several schemes have been proposed in the literature to tackle the problem of high frame error rates on the wireless link. We classify them into approaches trying to hide the TCP sender from non-congestion losses and into approaches making the TCP sender aware of the existence of the wireless link [4]: 9 Hiding the wireless link with Radio Link Layer schemes -
-
-
(reliable) Link Layer proposals [10,11] TCP aware Link Layer Protocols like the Berkeley Snoop Protocol [12] appropriate ARQ schemes [13]
9 Hiding the wireless link with TCP split connection approaches - indirect TCP [14] Remote Socket architecture [15] - WAP concept -
9 Extending TCP with add ons - TCP Selective Acknowledgments [16] Explicit Loss Notification - TCP Eifel algorithm [9] -
The proposals have different disadvantages. TCP non-aware link layer solutions have no information on the particular TCP state and hence do not take TCP congestion control into account. The Berkeley Snoop protocol as an example of a TCP aware solution relies on the accessibility of IP-content and thus does not work with laver 3 encryption, like IP Sec, nor IP tunneling. TCP split connection approaches break the end-to-end semantic. This is not desirable, as it circumvents the reliable end-to-end data transmission associated with TCP. The major disadvantage of the TCP add-on proposals is that the T C P / I P protocol suite is usually part of the operating system. Proposing modifications to the TCP protocol requires non-practical software updates for all involved systems. This makes interesting proposals like the TCP Eifel algorithm less feasible, although it covers the problem of spurious TCP packet retransmissions. 2. T C P P E R F O R M A N C E
IN MOBILE NETWORKS
2.1. P a c k e t Delay V a r i a t i o n Modern cellular mobile networks, like General Packet Radio Service (GPRS) or Universal Mobile Telecommunications System (UMTS) already include a very strong link layer error protection on the wireless link, using Forward Error Correction (FEC) and Automatic Repeat Request (ARQ) schemes. Due to the repeated transmission of link
520
layer PDUs, a bad quality of the wireless link does not mainly result in IP packet errors but rather in an additional packet delay. E.g. a reliable link layer ARQ scheme could completely hide the IP layer from any packet losses on the wireless link. Consequently, varying channel conditions of the mobile link lead to a packet delay variation seen by the upper layers. Supplementary, the presence of hand-offs and link layer disconnections in cellular mobile networks lead to spontaneous, unusual packet delays. The transport protocol has to deal with this significant packet delay variation. If the delay exceeds a certain value the Retransmission Timer elapses and TCP assumes a packet loss due to a major congestion situation within the network (Spurious Retransmission Timeout). The congestion control reacts with a rapid decrease of the congestion window to 1 Maximum Segment Size. The problem is, that in mobile networks these unusual long packet delays may occur due to local ARQ retransmissions, hand-offs or link laver disconnections and not because of network congestion. Therefore TCP wrongly reacts with packet retransmission and congestion window decrease and its performance drops. The timer based loss detection of TCP is based on the experienced Round Trip Time [3]. Due to the cumulative nature of the Retransmission Timeout cah:ulation it is hard to derive a timeout probability analytically. We show in simulations [17] with a two-state packet delay model that spurious TCP Retransmission Timeouts frequently occur, when the duration of the bad state (the channel is completely blocked during this time) reaches the magnitude of the TCP-RTO granularity (see Figure 2 right). In this example spontaneous packet delays longer than 100 ms lead to spurious Retransmission Timeouts. Note that TCP performance depends on the network scenario and the capacity of the wireless link. For small round trip times the effect is less noticeable, since TCP is able to recover its congestion window faster than for the case of long round trip times (see Figure 2 left). 1ooo Network
Network Network
Delay Delay Delay
= 200 m s - - - - + - = 400 m s - - ~ - - = 800 m s .... * ....
5-. /" ,,,. /' ,,
900 800
0.5
700 -~
"~
0.4
~
0.3
~
0.2
Fast Retransmlsslons Rctransmisslon T|meouts
-.. -- ,,
~ ----x---
? /
600
" " - ~ - . . . . . ~ . . . . . . . . . . . ~--'"~
,/
500 400
/
,/
300
/'
200 /'
lOO o 1o
1oo meanhnkbltn:hng
lOOO durauon[ms]
lOOOO
,
,
.
- ....
t
9
10
.
. : ....
,
9
.
. ~ ....
100 mean
hnk blocking
,
9
1000 duratlon
~
9
~... 10000
[ms]
Figure 2. TCP Performance and Number of spurious Retransmission Timeouts 2.2. Analysis of Hand-off Scenario's As a second important cause for spurious TCP packets we identify hand-off scenarios. Mobile cellular networks comprise a structure as depicted in figure 3. User mobility is the prime reason for the network to reroute the user from one serving Base Station A
521 to another B. For the sake of simplicity we aggregate common network elements from standards as the Radio Access Network and components of the wireless backbone (SGSN, GGSN) into the term Gateway. The Gateway supports the hand-off by routing incoming packets from the corresponding host to the new Base Station B. Naturally there will be packets queued at Base Station A at hand-off. Various strategies are possible, starting from full or partly packet drops of queued data at Base Station A to packet forwarding from Base Station A to B. As there is parallel incoming traffic from the Corresponding Host to the forwarded packets, the resulting IP packet reordering at the receiver does also yield TCP problems. The effects on TCP performance will also differ for the various TCP versions (Tahoe, Reno, New-Reno, SACK). In the following we motivate by example, that we find all solutions producing significant amount of spurious retransmissions. We used the network simulator n s [18] and its Mobile IPv4 implementation. The TCP parameters followed the recommendations of n s . Our TCP traces are recorded at the receiver. The figures show incoming TCP sequence number over time.
Figure 3. Wireless System Model for Multi-Cell Support Figure 4 shows the behavior of TCP Tahoe after the loss of 5 packets during a hand-off, where no packet forwarding between the Base Stations were performed. The queued handoff packets from the Gateway (SeqNr. 1499-1517) take the new path, while the packets stored at Base Station A (SeqNr. 1494-1498) are dropped. All packets arriving via the new path acknowledge the last valid in sequence packet (SeqNr. 1493) to the receiver. Thus the TCP sender performs a Fast Retransmit for the next packet (SeqNr. 1494). This packet is queued at the Gateway and does not reach the Mobile until the Gateway buffer is served. When the corresponding acknowledgment for packet (SeqNr. 1494) reaches the
522 kk A
1530 ~A k
1525 -
=
AA A =A~
1520 "
AA~
,_.:,
~, normal
1515"
8
--~-lost
..%
=== 1 5 1 0 "
---'.----retransmit .... duplicate
,:,'b ..%
== 1 5 0 5 -
::.',
i..
1495
:::. gateway queue
::.< :::.'>
1500 -
ji
AAA~
1 4 9 0 -, 836
841
846
861
856
851 time/S
Figure 4. TCP Tahoe with 5 hand-off packet losses and no packet forwarding TCP sender it increases its congestion window using Slow Start and sends the next two missing packets (SeqNr. 1495 and 1496). As the TCP sender is unaware of the actual packet loss, it retransmits all packets until the last required acknowledgment (SeqNr. 1517). Packets with SeqNr. 1499-1502 are spurious retransmissions. 22220 22215
f
/
.~::
22210
//
22205
22200 22195
/ /
/
22190
/
]]
22175
22165
..4"" .... ":'
838
...... o l d A c k
2"
...........
837
..... retransmit ..... duplicate
22185 22180
9 normal .... lost
.... ,:.-.
839
840
I
841
842
843
time Is
Figure 5. TCP SACK with multiple hand-off packet losses and no packet forwarding Figure 5 shows the behavior of TCP SACK [16] for massive packet loss (SeqNr. 2217222208). This situation happens where no packet forwarding from Base Station A takes place, e.g. for a hand-off from between two heterogenous wireless systems like UMTS and IEEE 802.11 wireless LAN. There is a 2 second Retransmission Timeout between the packet retransmissions (SeqNr. 22172-22173) and the following Slow Start phase. Some packets (SeqNr. 22209-22215) were not lost during the hand-off and thus they are transmitted twice leading to a waste of capacity. If packet forwarding between the Base Stations is performed TCP, faces packet disordering from the mix of old and new TCP packets arriving at the Base Station queue. Figure 6 illustrates this situation for TCP Tahoe. The unordered packet arrival makes the
523
.j
1750 1745
= 1740
~
~:,:::::::::" "::"~r ~r
1735
9
.,:o'::":"
~ 1730
u
1725
]
<"
...... retransmit -=-duplicate
_~
l, .:--.--:i":""
1720
normal --~--lost
4"
..:.:.:.
~ forward
1715 1710 1.130
1.140
1.150 time
1.160
1.170
Is
Figure 6. TCP Tahoe with 5 hand-off packet losses and packet forwarding TCP receiver acknowledging always earlier packets and the TCP sender assumes packet losses performing packet retransmissions. Several packets are spurious retransmissions (SeqNr. 1725-1728, 1734-1742). We summarize that for all investigated TCP variants and various hand-off strategies spurious packet retransmissions occur. Even Selective Acknowledgment (SACK), which is supposed to provide the best performance under complex loss scenarios, suffers from Retransmission Timeout and spurious packet transfer. 3. F I L T E R I N G S P U R I O U S T C P P A C K E T S We have motivated that in mobile networks duplicate TCP packets may occur. In this section we propose a filter mechanism for duplicate TCP packets from redundant retransmissions in order to save the wireless link from spurious packet transfer and thus improve overall network performance. The description of our proposal is based on the Berkeley Snoop protocol [12] using a similar architecture operating on IP packets at the Base Station only. Note that the term Base Station applies primarily to wireless networks, but may be extended to any hop before a bottleneck link with variable loss and throughput character. Regarding to the previously introduced network elements, the term Base Station may apply to either the Base Station Sub System or the SGSN. The upper branch of figure 7 depicts the typical behavior of a TCP receiver, when it receives a duplicate packet due to TCP retransmission. It discards the packet after it has traveled the air link. Below we show the principle of our solution "Spurious Packet Filter" (SPF), where a duplicate packet is recognized at the Base Station and discarded. It is not transmitted over the scarce air link resources. Removing redundant data at the Base station is beneficial from various points of view. The processed TCP flow experiences faster service for its remaining IP packets speeding up the whole data transfer. Second, the overall offered load to the network is reduced, thus competing traffic will receive faster service. Given that enough data transfer is requested by subscribers from the operator, i.e. the amount of transferred volume is limited by
524 network resources, this additional traffic will thus increase revenue for the operator. Discarding duplicate packets at the Receiver Sender
] ---[ Data
"i'BaseStation
I
-----[ Data ~
Mobile [ Receiver receive E ~ duplicate?
Discarding duplicate packets at the Base Station
Agent I .... i
....
---[ Data ~
Mobile I Receiver
receive [ - - ~ duplicate?
Figure 7. Comparision of packet discarding schemes for duplicate packets Our proposal is based on the following assumptions: 9 TCP-retransmissions happen. We have motivated this assumption for various mobile network scenarios in the previous section. The redundancy of IP packets can be clearly indicated to the algorithm. We assume that all IP traffic during the whole lifetime of a TCP flow can be monitored at one central instance. Given appropriate monitoring of all IP packets one can safely detect IP packets associated to the well-defined TCP-protocol and build full state maintenance for each TCP flow traversing this node. Due to the standardized nature of the TCP protocol the indication of retransmissions is possible. By monitoring the acknowledgments on the backward channel, we learn which TCP packets are in-fact successfully received. Thus we assume, that we can safely predict which IP packets can be deleted for down-link traffic. IP packets are not segmented during their transmission. If they were and this leads to the problem of identifying TCP sequence numbers with each IP packet, the IP packets may first be reassembled before they are processed as described in this document. 9 We rely on a time-out algorithm at the Base Station to recover from loss on the wireless link, as in [1]. Filtering duplicate packets can be solved in a straight forward fashion. Duplicate T C P packets are filtered at the Base Station. This makes filtering only a matter of finding identical TCP-IP packets in the Base Station queue, which may be implemented as the following. Upon arrival of an IP packet the queue is searched for packets, which have the same IP-source and IP-destination address, the same source and destination port
525 number and the same TCP-sequence number. Optionally one may decide to check also for identical TCP packet length or even TCP packet content. When such a packet is found, the packet in the queue is replaced by the newly arrived packet. No further action is performed. The formerly introduced mobile networks infrastructure typically contains several entities tied by some internal flow control algorithm and thus distributes packets over the network elements. Thus restricting a filtering algorithm to a local view only may not allow elimination of all duplicates. In our following proposal we show how packets can be identified as duplicates after they have left a local queue of stored IP packets. This allows to place the filtering algorithm at a central network element (Gateway) with no access to the Base Station queue. Our proposal recognizes and discards incoming duplicates, which are already sent to the mobile station. Doing so is optimistic and assumes no actual packet loss on the air link. Our solution disables original TCP retransmissions and thus has to provide its own recovery mechanism for loss using its own time-out mechanism. The principle of installing a TCP aware agent at/before the Base Station is found in the Berkeley TCP-Snoop algorithm [12]. Here each TCP state is monitored at the Base Station. To ease our description we concentrate on the filtering function, which can be implemented stand alone or applied as extension to the TCP-snoop framework. Technical details as e.g. the detection of processing of ACKs, the implementation of RTT-estimation and the management of local timeouts are therefore not described. The core of our algorithm is the altered retransmission branch denoted as "sender rexmission" in figure 9 compared to the original flow-chart in figure 8 [12].
1. Forward Packet 2. Reset Local Rexmit Counter Sender Rexmission Yes
No
1. Mark as Cong. Loss 2. Forward Packet. Congestion Loss
1. Cache Packet 2. Forward Packet Common Case
Figure 8. Original proposed Snoop algorithm for IP data packet arrivals Where the TCP-snoop simply forwards the packet, we consider the local cache first. If the packet is already acknowledged by the receiver, we can safely discard the packet. If it is still in the cache, we replace the packet in the cache. The reason for the replacement is the following. The retransmission may be due to a former bit error in the old packet when traveling the backbone to the snoop agent. This is extremely rare, but might happen. The replacement in the cache will take care that the replaced packet will be retransmitted
526
No
Discard Packet Sender
Rexmission Yes
No
I
Packet already acknowledged
1. Mark as Cong. Loss [ I Replace old Packet 1 2. Forward Packet. ] in Cache with new | One [ Congestion
Loss
Packet not yet acknowledged
1. Cache Packet 2. Forward Packet Common Case
Figure 9. Our proposed algorithm for discarding duplicate packets in the future instead of the potentially faulty previous one. The mechanism prevents any original TCP retransmissions to pass to the receiver and achieves our performance gain. Now we have to ensure that local retransmission work. This is naturally implemented by understanding duplicate ACKs sent from the mobile. As these may get lost we additionally rely on timers as the original TCP sender and the TCP-snoop implementation do too. In our case every action that reads "Forward Packet" also starts a local retransmission timer. When the timer expires, the packet will be sent agaiN. A retransmission counter, initialized with 1 at the time of the first transmission, is increased by one. If this value is smaller than a fixed system parameter MAX_RETRANSMISSION_ATTEMPTS, e.g. with the value 5, the timer is reset again. Besides the advantage of re-implementing the reliable end-to-end connection originally provided by TCP, there is the advantage that timers may be adapted to be better suited to the wireless environment. This potential additional improvement is not covered here.
3.1. Application To analyze dynamic application behavior over a wireless link we have implemented a testbed application [19]. The testbed incorporates a UMTS specific link error model and protocol stack. External applications are routed through this simulation in real-time providing the application with a virtual wireless link. We have used this framework to incorporate the spurious packet filter and test its stability and performance. Figure 10 illustrates the overall performance gain of approximately 10%. TCP transfer volume was 1 MByte over a wireless link with 64 kbit/s data rate. Wireless errors are suppressed in this measurement, to avoid that randomly severe ARQ delays mix with our simulation parameter "mean link blocking". We opened the wireless channel for five seconds, allowing the TCP connection to recover. We then completely block the link for 1,4,5,7 and 9 seconds. Blocking may be due to link layer disconnections or handoffs with no IP-loss nor reordering. Another reason for delay may be QoS scheduling, when precedence is granted to priority traffic, like real-time traffic. For every blocking
527 Fzle
of
a
iMbyte
Fzle
over
.•
4O
~0 ~:
Transfer
64kblt/s
wzth wzthout
Llnk
SPF SPF---~---
25
20
15
J i
2
i
,
3 mean
i
4 iznk
5 blockzng
i
i
6 duratzon
i
7 zn
8
9
$
Figure 10. SPF performance gain over link blocking duration duration we find the filter to effectively increase TCP throughput. The experienced absolute performance gain of approximately 4 kbit/s relates to the fact that TCP packets are timing out sequentially. Extending the duration of the link's ON-period does not influence the absolute performance gain, but lowers the relative system gain. Please note that we have only modelled the influence of a blocked link here. Future wireless networks will require a large number of small high capacity radio cells to supply the promised high data rates. In such environment hand-offs will be an important and frequent cause for link blocking, causing retransmissions by timeout as well as due to packet loss and reordering. Higher relative performance gains are possible with the spurious packet filter, but systematic measurements are difficult, as the number of retransmission and thus the performance gain is dependent on the TCP state at hand-off time. The limitations of our simulation tool currently prevent for systematic and reproducible measurements. Furthermore the important influence of TCP versions and different hand-off-types require its dedicated investigation beyond the scope of this paper. 4. C O N C L U S I O N In this paper we investigate the performance of reliable data transmission in mobile networks using the TCP transport protocol. We show for recent cellular mobile networks that packet loss on the air link are less of a problem than packet delay variations and hand-off related effects like packet reordering. We explain in detail various examples of how these effects lead to spurious TCP retransmissions and thus performance degradation. Spurious packet transfer is wasting scarce radio resources. Previous research focused only on the effect and the quality of the adaption mechanism on the sender side. In deployments where the last hop is e.g. a corporate wireless LAN the cost for a transmission of a duplicate IP packet is low and thus was not considered for optimization. In public cellular wireless networks capacity is scarce and the network operator has a strong interest in saving transmission bandwidth over the air-link in order to provide the resources to a competing user. We therefore introduce a new paradigm for TCP performance optimization and propose
528 a solution to filter spurious duplicate T C P packets in the wireless network prior to the transmission on the air link. Thus we effectively improve T C P performance and wireless system utilization. REFERENCES 1. 2. 3. 4.
5. 6. 7. 8. 9. 10. 11. 12.
13.
14. 15.
16. 17. 18. 19.
J. Postel, "Transmission Control Protocol," RFC 793, IETF, September 1981. W. Stevens, TCP//IP Illustrated, Volume 1. Addison Wesley, 1994. M. Allman, V. Paxson, and W. Stevens, "TCP congestion control," RFC 2581, IETF, 1999. H. Balakrishnan, V. Padmanabhan, S. Seshan, and R. Katz, "A comparision of mechansisms for improving TCP performance over wireless links," IEEE/A CM Transactions on Networking, vol. 5, pp. 756-769, December 1997. F. Lefevre and G. Vivier, "Understanding tcps behavior over wireless links," tech. rep., Motorola Labs Paris, 2000. C. Barakat, E. Altman, and W. Dabbous, "On TCP performance in a heterogeneous network: A survey," IEEE Communications Magazine, January 2000. J. Pan, J. Mark, and X. Shen, "TCP performance and its improvement over lossy wireless links," in Proceedings of IEEE Globecom 2000, pp. 62-66, November 2000. G. Xylomenos and G. Polyzos, "Internet protocol performance over networks with wireless links," IEEE Network, July/August 1999. R. Ludwig and R. Katz, "The eifel algorithm: Making TCP robust against spurious retransmissions," A CM SIGCOMM, vol. 30, January 2000. H. Chaskar, T. Lakshman, and M. U., "TCP over wireless with link level error control: Analysis and design methodology," IEEE//A CM Transactions on Networking, October 1999. G. Xylomenos and G. Polyzos, "Link layer support for quality of service on wireless internet links," IEEE Personal Communications, October 1999. H. Balakrishnan, S. Seshan, E. Amir, and R. Katz, "Improving T C P / I P performance over wireless networks," in Proceedings of the 1st A CM Conference on Mobile Computing and Networking, November 1995. G. Carle, F. Fitzek, and A. Wolisz, "Combining transport layer and link layer mechanisms for transparent QoS support of IP based applications," in Proceedings of IP Quality of Service for Wireless and Mobile Networks (IQWiM99) Workshop, April 1999. A. Bakre and B. Badrinath, "I-TCP: Indirect TCP for mobile hosts," in Proceedings of the 15th International Conference on Distributed Computing Systems (ICDCS), May 1995. M. Schl~iger, B. Rathke, S. Bodenstein, and A. Wolisz, "Advocating a remote socket achitecture for internet access using wireless LANs," Special Issue of MONET on Wireless Internet and Intranet Access, 2001. M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow, "TCP selective acknowledgement options," RFC 1323, IETF, 1996. J. Schiller and S. Gruhl, "Investigations on TCP traffic in mobile networks," in Proceedings. ITG Workshop, Bremen, Germany, January 2001. K. Fall and K. Varadhan, "ns notes and documentation," http://www.isi.edu/nsnam/ns, February 2000. M. Link, G. S., and B. M., "A realtime testbed for adaptive multimedia applications over UMTS," in Proceedings. IEEE 3G Wireless, 2001.
517
Performance Improvements for TCP in Mobile Networks with high Packet Delay Variations J. Schiller a, S. Gruhl b, T. Schwabe a and M. SchweigeP ~Dresden University of Technology, Chair for Telecommunications, D-01062 Dresden, Germany bLucent Technologies Niirnberg, Bell Labs Innovations, Thurn- und Taxis-Strat]e 10, D-90411 Niirnberg, Germany Previous TCP performance investigations for wireless systems focus on TCP throughput degradation due to packet loss on the error-prone wireless link. Since cellular mobile networks like GPRS and UMTS include a strong link layer error protection, using adaptive Forward Error Correction and Automatic Repeat Request schemes, a bad quality of the wireless link does not mainly result in IP packet loss but rather in an additional packet delay. Furthermore TCP congestion control has to deal with hand-offs and temporary link layer disconnections in cellular mobile networks leading to a significant packet delay variation and packet disordering. This paper analyzes the performance of TCP in cellular mobile networks, focusing on the packet delay and disordering problem. We find spurious'retransmissions and TCP timeouts being a dominant cause for throughput degradation. Finally we propose an algorithm to filter spurious TCP retransmissions at the Base Station to increase TCP performance and increase overall wireless link capacity. 1. I N T R O D U C T I O N
AND RELATED WORK
The Transmission Control Protocol (TCP) is one of the most important standards for best effort, reliable data transmission. Today's Internet traffic uses predominantly TCP, as for applications like HTTP for Web Browsing, FTP for File Download or SMTP for Electronic Mail Transfer. Any Internet compatible end system supports the TCP/IP protocol suite, usually as part of the operating system. TCP is a protocol of the OSI transport layer and provides reliable end-to-end transport and end system based traffic control functions [1,2]. Reliability is achieved using backward error correction based on a "selective repeat" retransmission strategy with positive acknowledgments. Today's most common TCP implementations support timer based packet loss detection (Retransmission Timeout) as well as acknowledgment indicated loss detection (Fast Retransmit).
518 To avoid network congestion and end system overload TCP performs traffic control. The TCP sender adapts its data transmission rate using a window mechanism. The current window size indicates the number of packets, which are injected into the network per round trip cycle. It is calculated as the minimum of the receiver's window and the so-called congestion window. The receiver window size indicates the instantaneous available buffer space at the TCP receiver and is announced in the returning TCP acknowledgments. The TCP sender adapts its data transmission rate to the current network situation to avoid network congestion using the congestion window. A congestion situation is always assumed when TCP detects a packet loss, either through a Retransmission Timeout or the arrival of 3 duplicate acknowledgments (Fast Retransmit). The TCP sender reacts by reducing the congestion window to its half (Fast Retransmit) or to 1 Maximum Segment Size (Retransmission Timeout). If no packet loss is detected TCP continuously increases the congestion window to maximize the throughput. There are 2 ways to open the congestion window: Slow Start with an exponential and Congestion Avoidance with a linear increase per round trip cycle. Initially, at the beginning of a connection, TCP starts with a congestion window of 1 MSS and uses Slow Start as increase rule. Figure 1 shows the behavior of the TCP congestion window after connection establishment and after detecting packet losses. More detailed information on the TCP window mechanism can be found in [1-3].
,15
~_ 40 "6 E " z .E ._ o0 o
35
PacketError J
ao 25 20
._~ @
,3
10
s
ob0
0
5
10
15
20
25
30
35
40
Round Trip Time
Figure 1. Dynamics of the TCP congestion window TCP was historically designed and optimized for a wire-line network environment. Due to the low bit error rates on wire-line links the primary cause for packet losses is network congestion. Consequently, the congestion control algorithm of TCP directly reacts on packet losses. 1.1. R e l a t e d W o r k Wireless links typically do not offer a low transmission error rate as found in wire-line links. This leads to packet losses which are not caused by network congestion. Investigations with early wireless systems have shown that TCP reacts regularly with an un-
519 necessary reduction of its sending data rate after every loss. During the following slowly increase of the congestion window TCP may not fully utilize the potential available bandwidth. The result is an unnecessary reduction of the end-to-end throughput, and hence, sub-optimal performance [4-8]. Furthermore hand-offs and temporary link layer disconnections in mobile communication systems may lead to packet disordering and sporadic additional delays. TCP reacts with retransmission timeouts, spurious packet transfer and unnecessary reduction of its congestion window [9]. Recently several schemes have been proposed in the literature to tackle the problem of high frame error rates on the wireless link. We classify them into approaches trying to hide the TCP sender from non-congestion losses and into approaches making the TCP sender aware of the existence of the wireless link [4]: 9 Hiding the wireless link with Radio Link Layer schemes -
-
-
(reliable) Link Layer proposals [10,11] TCP aware Link Layer Protocols like the Berkeley Snoop Protocol [12] appropriate ARQ schemes [13]
9 Hiding the wireless link with TCP split connection approaches - indirect TCP [14] Remote Socket architecture [15] - WAP concept -
9 Extending TCP with add ons - TCP Selective Acknowledgments [16] Explicit Loss Notification - TCP Eifel algorithm [9] -
The proposals have different disadvantages. TCP non-aware link layer solutions have no information on the particular TCP state and hence do not take TCP congestion control into account. The Berkeley Snoop protocol as an example of a TCP aware solution relies on the accessibility of IP-content and thus does not work with laver 3 encryption, like IP Sec, nor IP tunneling. TCP split connection approaches break the end-to-end semantic. This is not desirable, as it circumvents the reliable end-to-end data transmission associated with TCP. The major disadvantage of the TCP add-on proposals is that the T C P / I P protocol suite is usually part of the operating system. Proposing modifications to the TCP protocol requires non-practical software updates for all involved systems. This makes interesting proposals like the TCP Eifel algorithm less feasible, although it covers the problem of spurious TCP packet retransmissions. 2. T C P P E R F O R M A N C E
IN MOBILE NETWORKS
2.1. P a c k e t Delay V a r i a t i o n Modern cellular mobile networks, like General Packet Radio Service (GPRS) or Universal Mobile Telecommunications System (UMTS) already include a very strong link layer error protection on the wireless link, using Forward Error Correction (FEC) and Automatic Repeat Request (ARQ) schemes. Due to the repeated transmission of link
520
layer PDUs, a bad quality of the wireless link does not mainly result in IP packet errors but rather in an additional packet delay. E.g. a reliable link layer ARQ scheme could completely hide the IP layer from any packet losses on the wireless link. Consequently, varying channel conditions of the mobile link lead to a packet delay variation seen by the upper layers. Supplementary, the presence of hand-offs and link layer disconnections in cellular mobile networks lead to spontaneous, unusual packet delays. The transport protocol has to deal with this significant packet delay variation. If the delay exceeds a certain value the Retransmission Timer elapses and TCP assumes a packet loss due to a major congestion situation within the network (Spurious Retransmission Timeout). The congestion control reacts with a rapid decrease of the congestion window to 1 Maximum Segment Size. The problem is, that in mobile networks these unusual long packet delays may occur due to local ARQ retransmissions, hand-offs or link laver disconnections and not because of network congestion. Therefore TCP wrongly reacts with packet retransmission and congestion window decrease and its performance drops. The timer based loss detection of TCP is based on the experienced Round Trip Time [3]. Due to the cumulative nature of the Retransmission Timeout cah:ulation it is hard to derive a timeout probability analytically. We show in simulations [17] with a two-state packet delay model that spurious TCP Retransmission Timeouts frequently occur, when the duration of the bad state (the channel is completely blocked during this time) reaches the magnitude of the TCP-RTO granularity (see Figure 2 right). In this example spontaneous packet delays longer than 100 ms lead to spurious Retransmission Timeouts. Note that TCP performance depends on the network scenario and the capacity of the wireless link. For small round trip times the effect is less noticeable, since TCP is able to recover its congestion window faster than for the case of long round trip times (see Figure 2 left). 1ooo Network
Network Network
Delay Delay Delay
= 200 m s - - - - + - = 400 m s - - ~ - - = 800 m s .... * ....
5-. /" ,,,. /' ,,
900 800
0.5
700 -~
"~
0.4
~
0.3
~
0.2
Fast Retransmlsslons Rctransmisslon T|meouts
-.. -- ,,
~ ----x---
? /
600
" " - ~ - . . . . . ~ . . . . . . . . . . . ~--'"~
,/
500 400
/
,/
300
/'
200 /'
lOO o 1o
1oo meanhnkbltn:hng
lOOO durauon[ms]
lOOOO
,
,
.
- ....
t
9
10
.
. : ....
,
9
.
. ~ ....
100 mean
hnk blocking
,
9
1000 duratlon
~
9
~... 10000
[ms]
Figure 2. TCP Performance and Number of spurious Retransmission Timeouts 2.2. Analysis of Hand-off Scenario's As a second important cause for spurious TCP packets we identify hand-off scenarios. Mobile cellular networks comprise a structure as depicted in figure 3. User mobility is the prime reason for the network to reroute the user from one serving Base Station A
521 to another B. For the sake of simplicity we aggregate common network elements from standards as the Radio Access Network and components of the wireless backbone (SGSN, GGSN) into the term Gateway. The Gateway supports the hand-off by routing incoming packets from the corresponding host to the new Base Station B. Naturally there will be packets queued at Base Station A at hand-off. Various strategies are possible, starting from full or partly packet drops of queued data at Base Station A to packet forwarding from Base Station A to B. As there is parallel incoming traffic from the Corresponding Host to the forwarded packets, the resulting IP packet reordering at the receiver does also yield TCP problems. The effects on TCP performance will also differ for the various TCP versions (Tahoe, Reno, New-Reno, SACK). In the following we motivate by example, that we find all solutions producing significant amount of spurious retransmissions. We used the network simulator n s [18] and its Mobile IPv4 implementation. The TCP parameters followed the recommendations of n s . Our TCP traces are recorded at the receiver. The figures show incoming TCP sequence number over time.
Figure 3. Wireless System Model for Multi-Cell Support Figure 4 shows the behavior of TCP Tahoe after the loss of 5 packets during a hand-off, where no packet forwarding between the Base Stations were performed. The queued handoff packets from the Gateway (SeqNr. 1499-1517) take the new path, while the packets stored at Base Station A (SeqNr. 1494-1498) are dropped. All packets arriving via the new path acknowledge the last valid in sequence packet (SeqNr. 1493) to the receiver. Thus the TCP sender performs a Fast Retransmit for the next packet (SeqNr. 1494). This packet is queued at the Gateway and does not reach the Mobile until the Gateway buffer is served. When the corresponding acknowledgment for packet (SeqNr. 1494) reaches the
522 kk A
1530 ~A k
1525 -
=
AA A =A~
1520 "
AA~
,_.:,
~, normal
1515"
8
--~-lost
..%
=== 1 5 1 0 "
---'.----retransmit .... duplicate
,:,'b ..%
== 1 5 0 5 -
::.',
i..
1495
:::. gateway queue
::.< :::.'>
1500 -
ji
AAA~
1 4 9 0 -, 836
841
846
861
856
851 time/S
Figure 4. TCP Tahoe with 5 hand-off packet losses and no packet forwarding TCP sender it increases its congestion window using Slow Start and sends the next two missing packets (SeqNr. 1495 and 1496). As the TCP sender is unaware of the actual packet loss, it retransmits all packets until the last required acknowledgment (SeqNr. 1517). Packets with SeqNr. 1499-1502 are spurious retransmissions. 22220 22215
f
/
.~::
22210
//
22205
22200 22195
/ /
/
22190
/
]]
22175
22165
..4"" .... ":'
838
...... o l d A c k
2"
...........
837
..... retransmit ..... duplicate
22185 22180
9 normal .... lost
.... ,:.-.
839
840
I
841
842
843
time Is
Figure 5. TCP SACK with multiple hand-off packet losses and no packet forwarding Figure 5 shows the behavior of TCP SACK [16] for massive packet loss (SeqNr. 2217222208). This situation happens where no packet forwarding from Base Station A takes place, e.g. for a hand-off from between two heterogenous wireless systems like UMTS and IEEE 802.11 wireless LAN. There is a 2 second Retransmission Timeout between the packet retransmissions (SeqNr. 22172-22173) and the following Slow Start phase. Some packets (SeqNr. 22209-22215) were not lost during the hand-off and thus they are transmitted twice leading to a waste of capacity. If packet forwarding between the Base Stations is performed TCP, faces packet disordering from the mix of old and new TCP packets arriving at the Base Station queue. Figure 6 illustrates this situation for TCP Tahoe. The unordered packet arrival makes the
523
.j
1750 1745
= 1740
~
~:,:::::::::" "::"~r ~r
1735
9
.,:o'::":"
~ 1730
u
1725
]
<"
...... retransmit -=-duplicate
_~
l, .:--.--:i":""
1720
normal --~--lost
4"
..:.:.:.
~ forward
1715 1710 1.130
1.140
1.150 time
1.160
1.170
Is
Figure 6. TCP Tahoe with 5 hand-off packet losses and packet forwarding TCP receiver acknowledging always earlier packets and the TCP sender assumes packet losses performing packet retransmissions. Several packets are spurious retransmissions (SeqNr. 1725-1728, 1734-1742). We summarize that for all investigated TCP variants and various hand-off strategies spurious packet retransmissions occur. Even Selective Acknowledgment (SACK), which is supposed to provide the best performance under complex loss scenarios, suffers from Retransmission Timeout and spurious packet transfer. 3. F I L T E R I N G S P U R I O U S T C P P A C K E T S We have motivated that in mobile networks duplicate TCP packets may occur. In this section we propose a filter mechanism for duplicate TCP packets from redundant retransmissions in order to save the wireless link from spurious packet transfer and thus improve overall network performance. The description of our proposal is based on the Berkeley Snoop protocol [12] using a similar architecture operating on IP packets at the Base Station only. Note that the term Base Station applies primarily to wireless networks, but may be extended to any hop before a bottleneck link with variable loss and throughput character. Regarding to the previously introduced network elements, the term Base Station may apply to either the Base Station Sub System or the SGSN. The upper branch of figure 7 depicts the typical behavior of a TCP receiver, when it receives a duplicate packet due to TCP retransmission. It discards the packet after it has traveled the air link. Below we show the principle of our solution "Spurious Packet Filter" (SPF), where a duplicate packet is recognized at the Base Station and discarded. It is not transmitted over the scarce air link resources. Removing redundant data at the Base station is beneficial from various points of view. The processed TCP flow experiences faster service for its remaining IP packets speeding up the whole data transfer. Second, the overall offered load to the network is reduced, thus competing traffic will receive faster service. Given that enough data transfer is requested by subscribers from the operator, i.e. the amount of transferred volume is limited by
524 network resources, this additional traffic will thus increase revenue for the operator. Discarding duplicate packets at the Receiver Sender
] ---[ Data
"i'BaseStation
I
-----[ Data ~
Mobile [ Receiver receive E ~ duplicate?
Discarding duplicate packets at the Base Station
Agent I .... i
....
---[ Data ~
Mobile I Receiver
receive [ - - ~ duplicate?
Figure 7. Comparision of packet discarding schemes for duplicate packets Our proposal is based on the following assumptions: 9 TCP-retransmissions happen. We have motivated this assumption for various mobile network scenarios in the previous section. The redundancy of IP packets can be clearly indicated to the algorithm. We assume that all IP traffic during the whole lifetime of a TCP flow can be monitored at one central instance. Given appropriate monitoring of all IP packets one can safely detect IP packets associated to the well-defined TCP-protocol and build full state maintenance for each TCP flow traversing this node. Due to the standardized nature of the TCP protocol the indication of retransmissions is possible. By monitoring the acknowledgments on the backward channel, we learn which TCP packets are in-fact successfully received. Thus we assume, that we can safely predict which IP packets can be deleted for down-link traffic. IP packets are not segmented during their transmission. If they were and this leads to the problem of identifying TCP sequence numbers with each IP packet, the IP packets may first be reassembled before they are processed as described in this document. 9 We rely on a time-out algorithm at the Base Station to recover from loss on the wireless link, as in [1]. Filtering duplicate packets can be solved in a straight forward fashion. Duplicate T C P packets are filtered at the Base Station. This makes filtering only a matter of finding identical TCP-IP packets in the Base Station queue, which may be implemented as the following. Upon arrival of an IP packet the queue is searched for packets, which have the same IP-source and IP-destination address, the same source and destination port
525 number and the same TCP-sequence number. Optionally one may decide to check also for identical TCP packet length or even TCP packet content. When such a packet is found, the packet in the queue is replaced by the newly arrived packet. No further action is performed. The formerly introduced mobile networks infrastructure typically contains several entities tied by some internal flow control algorithm and thus distributes packets over the network elements. Thus restricting a filtering algorithm to a local view only may not allow elimination of all duplicates. In our following proposal we show how packets can be identified as duplicates after they have left a local queue of stored IP packets. This allows to place the filtering algorithm at a central network element (Gateway) with no access to the Base Station queue. Our proposal recognizes and discards incoming duplicates, which are already sent to the mobile station. Doing so is optimistic and assumes no actual packet loss on the air link. Our solution disables original TCP retransmissions and thus has to provide its own recovery mechanism for loss using its own time-out mechanism. The principle of installing a TCP aware agent at/before the Base Station is found in the Berkeley TCP-Snoop algorithm [12]. Here each TCP state is monitored at the Base Station. To ease our description we concentrate on the filtering function, which can be implemented stand alone or applied as extension to the TCP-snoop framework. Technical details as e.g. the detection of processing of ACKs, the implementation of RTT-estimation and the management of local timeouts are therefore not described. The core of our algorithm is the altered retransmission branch denoted as "sender rexmission" in figure 9 compared to the original flow-chart in figure 8 [12].
1. Forward Packet 2. Reset Local Rexmit Counter Sender Rexmission Yes
No
1. Mark as Cong. Loss 2. Forward Packet. Congestion Loss
1. Cache Packet 2. Forward Packet Common Case
Figure 8. Original proposed Snoop algorithm for IP data packet arrivals Where the TCP-snoop simply forwards the packet, we consider the local cache first. If the packet is already acknowledged by the receiver, we can safely discard the packet. If it is still in the cache, we replace the packet in the cache. The reason for the replacement is the following. The retransmission may be due to a former bit error in the old packet when traveling the backbone to the snoop agent. This is extremely rare, but might happen. The replacement in the cache will take care that the replaced packet will be retransmitted
526
No
Discard Packet Sender
Rexmission Yes
No
I
Packet already acknowledged
1. Mark as Cong. Loss [ I Replace old Packet 1 2. Forward Packet. ] in Cache with new | One [ Congestion
Loss
Packet not yet acknowledged
1. Cache Packet 2. Forward Packet Common Case
Figure 9. Our proposed algorithm for discarding duplicate packets in the future instead of the potentially faulty previous one. The mechanism prevents any original TCP retransmissions to pass to the receiver and achieves our performance gain. Now we have to ensure that local retransmission work. This is naturally implemented by understanding duplicate ACKs sent from the mobile. As these may get lost we additionally rely on timers as the original TCP sender and the TCP-snoop implementation do too. In our case every action that reads "Forward Packet" also starts a local retransmission timer. When the timer expires, the packet will be sent agaiN. A retransmission counter, initialized with 1 at the time of the first transmission, is increased by one. If this value is smaller than a fixed system parameter MAX_RETRANSMISSION_ATTEMPTS, e.g. with the value 5, the timer is reset again. Besides the advantage of re-implementing the reliable end-to-end connection originally provided by TCP, there is the advantage that timers may be adapted to be better suited to the wireless environment. This potential additional improvement is not covered here.
3.1. Application To analyze dynamic application behavior over a wireless link we have implemented a testbed application [19]. The testbed incorporates a UMTS specific link error model and protocol stack. External applications are routed through this simulation in real-time providing the application with a virtual wireless link. We have used this framework to incorporate the spurious packet filter and test its stability and performance. Figure 10 illustrates the overall performance gain of approximately 10%. TCP transfer volume was 1 MByte over a wireless link with 64 kbit/s data rate. Wireless errors are suppressed in this measurement, to avoid that randomly severe ARQ delays mix with our simulation parameter "mean link blocking". We opened the wireless channel for five seconds, allowing the TCP connection to recover. We then completely block the link for 1,4,5,7 and 9 seconds. Blocking may be due to link layer disconnections or handoffs with no IP-loss nor reordering. Another reason for delay may be QoS scheduling, when precedence is granted to priority traffic, like real-time traffic. For every blocking
527 Fzle
of
a
iMbyte
Fzle
over
.•
4O
~0 ~:
Transfer
64kblt/s
wzth wzthout
Llnk
SPF SPF---~---
25
20
15
J i
2
i
,
3 mean
i
4 iznk
5 blockzng
i
i
6 duratzon
i
7 zn
8
9
$
Figure 10. SPF performance gain over link blocking duration duration we find the filter to effectively increase TCP throughput. The experienced absolute performance gain of approximately 4 kbit/s relates to the fact that TCP packets are timing out sequentially. Extending the duration of the link's ON-period does not influence the absolute performance gain, but lowers the relative system gain. Please note that we have only modelled the influence of a blocked link here. Future wireless networks will require a large number of small high capacity radio cells to supply the promised high data rates. In such environment hand-offs will be an important and frequent cause for link blocking, causing retransmissions by timeout as well as due to packet loss and reordering. Higher relative performance gains are possible with the spurious packet filter, but systematic measurements are difficult, as the number of retransmission and thus the performance gain is dependent on the TCP state at hand-off time. The limitations of our simulation tool currently prevent for systematic and reproducible measurements. Furthermore the important influence of TCP versions and different hand-off-types require its dedicated investigation beyond the scope of this paper. 4. C O N C L U S I O N In this paper we investigate the performance of reliable data transmission in mobile networks using the TCP transport protocol. We show for recent cellular mobile networks that packet loss on the air link are less of a problem than packet delay variations and hand-off related effects like packet reordering. We explain in detail various examples of how these effects lead to spurious TCP retransmissions and thus performance degradation. Spurious packet transfer is wasting scarce radio resources. Previous research focused only on the effect and the quality of the adaption mechanism on the sender side. In deployments where the last hop is e.g. a corporate wireless LAN the cost for a transmission of a duplicate IP packet is low and thus was not considered for optimization. In public cellular wireless networks capacity is scarce and the network operator has a strong interest in saving transmission bandwidth over the air-link in order to provide the resources to a competing user. We therefore introduce a new paradigm for TCP performance optimization and propose
528 a solution to filter spurious duplicate T C P packets in the wireless network prior to the transmission on the air link. Thus we effectively improve T C P performance and wireless system utilization. REFERENCES 1. 2. 3. 4.
5. 6. 7. 8. 9. 10. 11. 12.
13.
14. 15.
16. 17. 18. 19.
J. Postel, "Transmission Control Protocol," RFC 793, IETF, September 1981. W. Stevens, TCP//IP Illustrated, Volume 1. Addison Wesley, 1994. M. Allman, V. Paxson, and W. Stevens, "TCP congestion control," RFC 2581, IETF, 1999. H. Balakrishnan, V. Padmanabhan, S. Seshan, and R. Katz, "A comparision of mechansisms for improving TCP performance over wireless links," IEEE/A CM Transactions on Networking, vol. 5, pp. 756-769, December 1997. F. Lefevre and G. Vivier, "Understanding tcps behavior over wireless links," tech. rep., Motorola Labs Paris, 2000. C. Barakat, E. Altman, and W. Dabbous, "On TCP performance in a heterogeneous network: A survey," IEEE Communications Magazine, January 2000. J. Pan, J. Mark, and X. Shen, "TCP performance and its improvement over lossy wireless links," in Proceedings of IEEE Globecom 2000, pp. 62-66, November 2000. G. Xylomenos and G. Polyzos, "Internet protocol performance over networks with wireless links," IEEE Network, July/August 1999. R. Ludwig and R. Katz, "The eifel algorithm: Making TCP robust against spurious retransmissions," A CM SIGCOMM, vol. 30, January 2000. H. Chaskar, T. Lakshman, and M. U., "TCP over wireless with link level error control: Analysis and design methodology," IEEE//A CM Transactions on Networking, October 1999. G. Xylomenos and G. Polyzos, "Link layer support for quality of service on wireless internet links," IEEE Personal Communications, October 1999. H. Balakrishnan, S. Seshan, E. Amir, and R. Katz, "Improving T C P / I P performance over wireless networks," in Proceedings of the 1st A CM Conference on Mobile Computing and Networking, November 1995. G. Carle, F. Fitzek, and A. Wolisz, "Combining transport layer and link layer mechanisms for transparent QoS support of IP based applications," in Proceedings of IP Quality of Service for Wireless and Mobile Networks (IQWiM99) Workshop, April 1999. A. Bakre and B. Badrinath, "I-TCP: Indirect TCP for mobile hosts," in Proceedings of the 15th International Conference on Distributed Computing Systems (ICDCS), May 1995. M. Schl~iger, B. Rathke, S. Bodenstein, and A. Wolisz, "Advocating a remote socket achitecture for internet access using wireless LANs," Special Issue of MONET on Wireless Internet and Intranet Access, 2001. M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow, "TCP selective acknowledgement options," RFC 1323, IETF, 1996. J. Schiller and S. Gruhl, "Investigations on TCP traffic in mobile networks," in Proceedings. ITG Workshop, Bremen, Germany, January 2001. K. Fall and K. Varadhan, "ns notes and documentation," http://www.isi.edu/nsnam/ns, February 2000. M. Link, G. S., and B. M., "A realtime testbed for adaptive multimedia applications over UMTS," in Proceedings. IEEE 3G Wireless, 2001.