- Email: [email protected]
Agent-based rate coordination between TCP and ABR congestion control algorithms
Computer Communications 24 (2001) 850±859
www.elsevier.com/locate/comcom
Agent-based rate coordination between TCP and ABR congestion control algorithms q K. Djemame*, M. Kara ATM-Multimedia Group 1, School of Computer Studies, University of Leeds, Leeds LS2 9JT, UK Received 6 November 2000; accepted 6 November 2000
Abstract In this paper we study the extent of negative interactions between Transmission Control Protocol (TCP) and Asynchronous Transfer Mode (ATM) congestion control through the application of an approach based on coordination via explicit cooperation. To do so, we implement an agent that bridges the gap between the ATM layer and the TCP layer in the protocol stack at the sender end to coordinate the congestion control algorithms of the TCP transport protocol and the ATM cell-oriented switching architecture. The novel idea uses ATM rate-based ¯ow control to adjust the credit-based window size of TCP via the implementation of the agent. The effects of running TCP over Available Bit Rate (ABR) are studied with the help of two simulation models (LAN and WAN). Firstly, we show that TCP window (CWND) and ABR Actual Cell Rate (ACR) are weakly correlated. Secondly, we derive an Agent-based algorithm to ensure that even when the ATM switch queues are indicating no congestion, it is the switch (and not the source) which should control the increment in the TCP source rates. The introduction of the agent at the edge of the network coordinates the TCP and ATM sending transmission and clearly demonstrates a strong correlation coef®cient between CWND and ACR. The simulation results show that coordination at the edge of the network makes better use of available bandwidth and improves TCP's performance over ATM for a range of performance metrics such as Cell Loss Ratio, Packet Retransmission Ratio and Throughput. q 2001 Elsevier Science B.V. All rights reserved. Keywords: Agent-based rate coordination; Transmission control protocol; Asynchronous transfer mode; Available bit rate; Performance evaluation
1. Introduction The Transmission Control Protocol (TCP) has emerged in the last few years as the standard in data communication. It is a reliable transport protocol that uses a window-based ¯ow and error control. Asynchronous Transfer Mode (ATM) is expected to be the technology of choice for high-speed computer networks. It was developed as part of the work on broadband Integrated Service Digital Network (ISDN) (but is still used in non-ISDN environments) where very high data rates are required. Available Bit Rate (ABR) is a service category de®ned by the ATM Forum [4] which supports applications with vague requirements for throughput and delay, for example data traf®c. ABR uses a rate-based congestion control based on q This work is funded in part by the Engineering and Physical Science Research Council (EPSRC) under project GR/L86937. * Corresponding author. Tel.: 144-113-233-6590; fax: 144-113-233-5468. E-mail addresses: [email protected] (K. Djemame), [email protected] (M. Kara). 1 http://www.scs.leeds.ac.uk/atm-mm.
network's feedback. Feedback information about network congestion is conveyed from network elements to end systems in order to dynamically adapt the cell emission process of the ABR sources to the current load of the network. However, a transport protocol such as TCP encounters problems when running over ABR due to its window-based congestion control interferences with ABR's rate-based. The goal of this paper is to propose an approach to cooperation between TCP and ABR congestion control algorithms to improve overall performance. 1.1. TCP control TCP is a very popular transport protocol, and is the most widely used for today's Internet data applications. It combines logic for routing through an Internet with endto-end control. TCP is actually a collection of algorithms that send packets into the network without a reservation and then react to observable events to occur. Such algorithms include congestion control and recovery from loss of packets [2]. Each sender maintains two windows: the window RWND
0140-3664/01/$ - see front matter q 2001 Elsevier Science B.V. All rights reserved. PII: S 0140-366 4(00)00357-1
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Nomenclature AAL5 ATM adaptation layer 5 ABR Available bit rate ACK Acknowledgement ACR Allowed cell rate ATM Asynchronous transfer mode BRM Cell Backward resource management cell CI Congestion indication CLR Cell loss ratio CWND Congestion window ECN Explicit congestion noti®cation EFCI Explicit forward congestion indication EPD Early packet discard ER Explicit rate ERICA Explicit rate indication for congestion avoidance FRM cell Forward resource management cell LAN Local area network LLC Logical link control IP Internet protocol ISDN Integrated service digital network MCR Minimum cell rate MSS Maximum segment size NI No increase PCR Peak cell rate PPD Partial packet discard PRR Packet retransmission ratio QoS Quality of service RDF Rate decrease factor RIF Rate increase factor RTO Retransmission time out RTT Round-trip time SSTHRESH Slow start threshold TCP Transmission control protocol TTCA Time to transmit CWND at ACR VBR Variable bit rate VC Virtual circuit WAN Wide area network
the receiver has granted and a second window CWND called the congestion window. The minimum of the congestion window and the receiver's window is used when sending packets. The congestion control mechanism has two distinct phases. Slow Start and Congestion Avoidance. Upon starting a connection or restarting after a packet loss, the congestion window size is set to one packet, and then doubled once every Round-Trip Time (RTT), i.e. upon the receipt of an acknowledgement (ACK). Congestion avoidance takes care of violation which would occur if the retransmit time is too short. During this phase, CWND is linearly increased as opposed to exponential increase during slow start phase. A third parameter SSTHRESH is used to start the congestion phase and is usually initially set to
851
64 kB (kilobytes). Upon noti®cation of network congestion, i.e. a timeout, SSTHRESH is set to half the current CWND and CWND is set to one and the slow start phase restarts once again. There are several ways to detect a packet loss: (1) a timer: when the timer is greater than the Retransmission Time Out (RTO), a packet is assumed to be lost; and (2) when a packet has been acknowledged more than three times, the next packet is assumed to be lost. Earlier implementations of TCP were distributed in releases corresponding to the original implementation of Jacobson [10]. However, over the years, there have been some ®netuning, improvements and additions to the original TCP algorithms. Some of the proposed changes have been widely adopted and are part of TCP implementations today, e.g. fast retransmit and recovery [2]. TCP-Tahoe and TCP Reno are the (most) deployed current versions. Other revisions for TCP have been proposed and include TCP Vegas [6] and Selective ACKs [17]. 1.2. ABR mechanism The ABR control unit adjusts the rate of ATM cells into the network dynamically according to the congestion status of the network. This rate is called the Actual Cell Rate (ACR). The source is not allowed to send cells faster than ACR to avoid network congestion. The rate-based control scheme (explicit rate or relative rate marking (EFCI) uses special Resource Management (RM) cells [4]. In the Explicit Rate (ER) scheme, the source sends an RM cell once in every Nrm (default Nrm 32 cells or in Trm (default Trm 100 ms time units along the Virtual Circuit (VC) to the destination. Each cell contains three ®elds that provide feedback to the source: a Congestion Indication (CI) bit, a No Increase (NI) bit, and an ER ®eld. The source sets the RM cell's ER ®eld to the rate at which it likes to send and transmits the Forward RM (FRM) Cell. As each FRM cell is received by the destination, it is turned around and transmitted back to the source as a Backward RM (BRM) Cell. Any of the CI, NI and ER ®elds may be changed by an ATM switch along the VC or the destination before the corresponding BRM cell returns to the source. ACR always varies between the limits of Minimum Cell Rate (MCR) and Peak Cell Rate (PCR). Several rules are de®ned in Ref. [4] for modifying ACR according to the received information. The major ones are: NI bit 0 and CI bit 0; ACR Ã max[MCR, min[ER, PCR, ACR 1 RIF £ PCR]] NI bit 1 and CI bit 0; ACR Ã max[MCR, min[ER, ACR]] CI bit 1; ACR Ã max[MCR, min[ER, ACR(1 2 RDF)]] Rate Increase Factor (RIF) and Rate Decrease Factor (RDF) are de®ned in Ref. [4] (default value: 1/16).
852
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
1.3. Problems and motivation of the work Integrating ATM technology has revealed poor throughput performance of TCP over ATM [20]. This poor performance is mainly due to packet fragmentation which occurs when a TCP packet ¯ows into an ATM VC though the ATM Adaptation Layer 5 (AAL5). AAL5 is responsible for dividing TCP's packets into sets of 48 byte data units called cells. Fragmentation at the AAL5 is necessary since the typical size of a TCP packet is much larger than that of a cell. The loss of even a single cell in any of the ATM network's switches results in the corruption of the entire packet to which that cell belongs, and consequently causes its retransmission. Techniques to improve cell discard behaviour such as Partial Packet Discard (PPD) and Early Packet Discard (EPD) are aimed at reducing the transmission of useless cells [20]. Various issues have been investigated for the performance evaluation of TCP over ATM. These issues may be divided into three groups: those addressing the dynamics of TCP, those addressing the behaviour of ATM and those which address the interaction between TCP's window-based and ATM layer congestion control mechanisms [14]. Although performance evaluation of TCP over ATM has been extensively studied, progress has been made only in identifying localised solutions to particular problems such as buffering requirements [12]. There is much to be done on the overall effect of these schemes when interacting with each other in order to focus the attention towards coherent solutions. On a TCP connection over ATM implementing the ABR service, the problems come from the interaction between both congestion control methods. TCP and ABR use two different ways to detect congestion in the network: TCP uses packet losses to regulate its rate (window) whereas ABR uses the feedback information transported by RM cells to control its ACR. It is worth mentioning that ABR does not wait for a packet loss to reduce its rate. Also, in case of congestion, the ABR control loop is designed to slow down the ABR source. With current implementations of TCP, ATM networks are limited to packet drops as the only viable mechanism to inform TCP sources of congestion. Thus, noti®cation is implicit and ABR is unable to signal to the real (TCP) traf®c sources the rate at which they should transmit (although the mechanism to use Explicit Congestion Noti®cation (ECN) has been proposed [9,19], it has not been implemented yet). Our aim in this paper is to study the extent of negative interactions between TCP and ATM congestion control through the application of an approach based on coordination via explicit cooperation. To do so, we implement an agent that bridges the gap between the ATM layer and the TCP layer in the protocol stack at the sender end to coordinate the congestion control algorithms of the TCP transport protocol and the ATM cell-oriented switching architecture. Agents are an important area of research and development in various domains such as arti®cial intelligence, high-
performance computing and communication networks [1]. Our proposal is driven from our interest in proposing a framework for a coherent approach for better coordination between TCP and ATM congestion control algorithms [7]. The effects of running TCP over ABR are studied with the help of a simulation model in a LAN and WAN environment. The TCP model is designed to simulate data transfer between n hosts over the ATM network. On a typical backbone link, the TCP queues are at the edge router and not inside the ATM network. Firstly, we show that TCP window (CWND) and ABR ACR are weakly correlated. Secondly, we derive an Agent-based algorithm to ensure that even when the ATM switch queues are indicating no congestion, it is the switch (and not the source) which should control the increment in the TCP source rates. The introduction of the agent at the edge of the network coordinates the TCP and ATM sending transmission and clearly demonstrates a strong correlation coef®cient between CWND and ACR. The simulation results show that coordination at the ATM network's edge makes better use of available bandwidth and a consistent improvement of TCP's performance over ATM for performance metrics such as the Cell Loss Ratio (CLR), the Packet Retransmission Ratio (PRR) and the Throughput. 1.4. Related work The closest work relative to this paper was proposed by Satyavolu et al. [21], NarvaÂez and Siu [18], Koike [15] and Kalampoukas et al. [11]. These researchers independently suggested the idea of overwriting the ACK ®elds and pacing the ACKs (called acknowledgement bucket by these authors). Satyavolu et al. suggest a rate-to-window translation scheme based upon the Explicit Rate Indication for Congestion Avoidance (ERICA) ATM ABR algorithm [21]. NarvaÂez and Siu suggest the implementation of somewhat complex TCP emulation engine at the rate-controller and an acknowledgement bucket [18]. Koike also suggests the use of an acknowledgement bucket [15]. Further, these authors explored the idea of overwriting the ACK ®elds and pacing the ACKs from the point of view of TCP/IP-ATM internetworking, i.e. extending the ATM ABR type rate control from ATM edges to TCP end systems. Kalampoukas et al. propose an approach based on buffer occupancy [11]. Their work shows that explicit window control is more effective than buffer management techniques due to its inherence congestion (and loss) avoidance approach. Unlike the approaches in Refs. [15,18], our idea is based on regulating the circulation of packets rather than that of ACKs. It also differs from the work in Refs. [11,21] in that we take an existing ABR rate control scheme and do a rate to (TCP) window translation. Our approach uses a more direct translation of the ABR feedback into an ideal TCP window that can lead to burstiness avoidance in network traf®c.
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
853
Fig. 1. (a) Coordinating TCP and ABR sources. (b) Role of the agent.
1.5. Paper organisation We begin Section 2 by presenting our proposal for a new congestion control scheme. The simulation environment is presented in Section 3. Performance results over the simulation model are discussed in Section 4. Some concluding remarks and perspectives of the research for future developments are given in Section 5. A list of acronyms is included as nomenclature. 2. Proposal for a new congestion control scheme In TCP, the maximum traf®c is controlled by the window CWND, while in ABR the traf®c is controlled by MCR, PCR and ACR. A key factor that impacts user and network performance for TCP over ABR is how well the traf®c management mechanism used in TCP end system, ATM end system and switch mesh together in providing good end-to-end performance. At the edge of the ATM network, the possibility of using ER indication instead of backward congestion noti®cation is an interesting enhancement to TCP, especially as different studies have shown that using the ER indication mode in ATM results in much better utilisation and fairness results compared to backward congestion noti®cation schemes [22]. This resemblance will allow better integration of ATM and TCP as both schemes would react in a similar way at the same congestion information. Also, handling congestion in advance reduces the amount of buffer required in the intermediate switches and the delay caused by retransmissions. 2.1. Assessment of coordination The association between the TCP sender window and the ACR may be analysed using the correlation coef®cient between CWND and ACR over a set of observations as follows: E Ci 2 C i Ai 2 A i p Corr C; A p E Ci 2 C i 2 E Ai 2 A i 2 where Ci and Ai are the ith CWND and corresponding ACR
observations. Corr(C,A) measures the association between CWND and ACR. If Corr(C,A) is equal to zero, CWNDi and ACRi are independent. However, if the value of Corr(C,A) is close to 21 or 1 then there is a strong association between them. Change in ACR is made at the ABR source when receiving a BRM cell containing feedback provided by the network, mainly a CI bit, a NI bit and the ER. Thus, the source keeps transmitting at the same rate during the time elapsing between the transmission of the FRM Cell and the reception of the BRM Cell. If the rate-based approach uses a negative polarity of feedback, RM cells are sent only to decrease the rate. By using positive polarity RM cells are required to increase the rate. TCP sender updates CWND either at RTT when receiving an ACK (increase of CWND) or when a timeout occurs (decrease of CWND). The CWND±ACR correlation may be used as a tool to compare the effectiveness of: (1) buffer allocation schemes; (2) different cell dropping policies; and (3) different number of TCP sources. 2.2. Coordination at ATM network's boundaries Our contribution involves calculating the rate feedback in the ABR switch algorithm and translate it into a window value to give feedback to TCP. This method exploits the fact that TCP rate is controlled by the ABR rate. CWND and ACR are both instantaneous indicators of the window size and the cells transmission speed at their respective layers. The (normalised) ratio CWND/ACR gives the time TTCA it takes a CWND window (bytes) to be transmitted at ACR (cells/s). However, TTCA can readily be obtained from the ratio of the normalised CWND (bytes) to normalised ACR (bytes/s): 8 > < CWND ; ACR ± 0 TTCA ACR p 53 > : not defined; otherwise The notion of TTCA is by de®nition nonexistent in situations where ACR becomes nil. The number 53 in the above equation is derived from the (payload 1 header) size of ATM cells. In non-zero ACR situations, TTCA is a positive
854
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Because of the dynamic nature of the available bandwidth for ABR service, the ER (and consequently ACR) command to the source is expected to change constantly. Thus, variations in ABR's ACR occur at a very high rate (small time scales) while TCP's window adjustments operate over longer time scales. In order to avoid an oscillation of the protocol, a scheme is implemented which treats the advertised ACR(s) as an average allocation over the ¯ow's RTT:
Fig. 2. LAN simulation model.
number: CWND is within the range [1¼MAXCWND] and ACR within the range [MCR¼PCR]. The goal behind considering the parameter TTCA is to show that the congestion control algorithms implemented at the ATM switch successfully controls the TCP sources, in other words (window-based) TCP congestion control algorithms become rate limited as well. A value of TTCA close to the ratio MAXCWND/ (PCR p 53) is a desirable property: we would like TCP sender and ABR source to have a maximum transmission window and a maximum cell rate respectively, assuming the buffering requirements for zero cell loss at the ATM switches are ful®lled. In the worst scenarios, and assuming ACR remains constant, an increase of CWND will lead to packet drops whereas a decrease of CWND will lead to a waste of bandwidth. Assuming CWND remains constant, an increase of ACR will lead to a waste of bandwidth whereas a decrease of ACR will lead to packet drops. Both the CWND±ACR correlation coef®cient and TTCA indicators provide information relating to the coordination between the congestion measures at TCP transport and ATM physical layers. 2.2.1. Role of the agent The role of the coordinator agent at the edge of the ATM network is to coordinate TCP and ABR sources (Fig. 1(a)) by bridging the gap between the ATM layer and the TCP layer in the protocol stack at the source at each RTT. It inputs the values of ACR and PCR from the ABR source and MAXCWND from TCP sender, respectively, and outputs CWNDACR value to explicitly indicating the TCP sender rate (Fig. 1(b)). CWNDACR is a new parameter derived from TTCA and de®ned as follows: MAX_TTCA
MAXCWND ; PCR p 53
CWNDACR MAX_TTCA p ACR p 53 MAXCWND p
ACR : PCR
reset counter i to prepare for measurement on next RTT; at reception of BRM celli calculate ACRi; at each RTT: calculate the average ACRavg ACR1 1 ACR2 1 ¼ 1 ACRn =n; set CWNDACR to MAXCWND p ACRavg =PCR: 2.2.2. ABR source On receiving a BRM cell, the ABR source calculates the new ACR using the values of ER, CI and NI set in the cell. It then allows ACR and PCR information to be sent to the agent after processing the BRM cell so that the agent can compute the new CWNDACR. 2.2.3. TCP sender Given that the variations in CWND is partially controlled by the agent, the objective of the protocol here is to partially disable TCP congestion control algorithm: at each RTT we make TCP sender window limit enforceable by the ATM network to control the TCP rate. CWND is set to CWNDACR provided by the agent whatever the actual TCP sender window is upon receiving an ACK by the TCP source. Packet losses are still indicated by the basic TCP's original congestion mechanisms which are maintained. If a transmission timer expires then the usual retransmission and window reduction actions are taken: SSTHRESH (slow start threshold) is halved, CWND is set to one segment and slow start is entered. In case three duplicate ACKs are received, the TCP sender sets the value of CWND to half of its current size (in case of TCP Reno) to restart slow start (fast recovery and fast retransmit). However, the source leaves the slow start mode as soon as an ACK packet is received and new actions are performed. 3. Experimental design 3.1. Simulation experiments and objectives We describe in this section the network con®gurations and parameters used in the stimulation. The objective of the experiments is twofold. Firstly, we analyse the association between TCP sender window and the ABR ACR using the correlation coef®cient between CWND and ACR. The
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
855
Group Src.3 to Destination 3: two-hop traf®c Group Src.4 to Destination 4: one-hop traf®c 3.3. Simulation tool
Fig. 3. WAN simulation model.
analysis is done over a set of observations when running simulations with TCP Reno and the Agent-based algorithm discussed is Section 2.2. Secondly, we show that the Agentbased algorithm under various scenarios exhibits a better performance in the evaluation of the simulation tests. We consider the simulation experiments performed with the help of LAN and WAN simulation models. 3.2. Topologies LAN model: Fig. 2 illustrates our LAN simulation con®guration consisting of N TCP sources, two ATM switches and N TCP destinations. TCP sources send data whenever allowed by the TCP window. All traf®c is unidirectional (discounting the TCP ACKs and the BRM cells in the reverse direction). WAN model: The WAN simulation model is a multistage wide area network with four switches SWi i 0¼3 (Fig. 3). The sources are divided into four groups. The traf®c ¯ow is shown by dashed lines. The purpose of choosing such a complex model is to make it as realistic as possible. In this model the TCP sources each establish a single connection with the similarly numbered destination across the ATM switches and the bottleneck links: Group Src.1 to Destination 1: end-to-end traf®c Group Src.2 to Destination 2: two-hop traf®c Table 1 CWND±ACR correlation coef®cient Ð Scen.LAN Number of sources
TCP Reno
Agent-based
1 5 10 20 40 60 80 100
0.893 0.622 0.496 0.371 0.326 0.234 0.196 0.116
0.981 0.977 0.972 0.970 0.970 0.968 0.964 0.961
The YATS simulation package is used in the simulation study in Ref. [5]. YATS is a detailed simulator for TCP over ATM. Most of common features of TCP such as slow start, congestion detection, congestion avoidance, fast retransmit and fast recovery are included in the simulator. We extended the simulator in order to implement the coordinator agent and allow communication between ABR and TCP sources. The agent was developed from scratch while major changes were made to the TCP sender. Minor changes to the ABR source allowed ACR and PCR information to be sent to the agent after processing a BRM cell. Traf®c is generated by YATS speci®c objects using a TCP/IP/AAL5/ATM protocol stack. 3.4. Simulation scenario The TCP sources are greedy sources with in®nite supply of data and always have data ready to send as and when permitted by the TCP ¯ow control. A large bulk data transfer application runs on top of TCP sources. This application recognises the start±stop protocol and generates frames of 8000 bytes length. To study the congestion behaviours at ATM and TCP layers, we consider that another Variable Bit Rate (VBR) application is running in the background. VBR has higher priority than ABR, and is simulated by an ON/OFF source (1 ms ON time and 1 ms OFF time) which operates at the maximum amplitude during the ON time. We assume the ATM switch has a buffer shared by all VCs passing through. The scheduling policy at the buffer is FIFO. In order to induce variable congestion in the switches, the size of the shared buffer is varied for the simulation cases. The queues that model the switches' buffers have a unique size, which is ®xed to 512 kB (9892 cells). The reason for this choice is to offer a tight buffer constraint. The switches are con®gured with ERICA [13] with the following parameters Ð target utilisation: 95%; averaging interval length defaults: 1 ms. ERICA is a well-known algorithm which achieves the goals of fairness, high utilisation and low delay using rate-based control. TCP Maximum Segment Size (MSS) is set to 512 bytes. We take into account 20 bytes of TCP header, 20 bytes IP header, 8 bytes LLC, 8 bytes AAL5 header. TCP sources use a 64 kB window, and all links run at 155.52 Mbps. The internal delay in each ATM switch (dSWITCH) is set to 10 ms. We consider a LAN scenario (Scen.LAN) where the delay (3 p dLAN 1 2 p dSWITCH) is set to a typical value of 100 ms (Fig. 2). The number of sources N is set in the range [1¼100]. The values of N are arbitrarily selected, but nonetheless deemed suf®cient for producing traf®c ¯ow required in the simulation study of high-speed
856
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Table 2 CWND±ACR correlation coef®cient Ð Scen.WAN Number of sources
TCP Reno
Agent-based
1 5 10 20 40 60 80 100
0.664 0.647 0.530 0.303 0.289 0.186 0.084 0.073
0.988 0.979 0.959 0.955 0.924 0.919 0.825 0.814
networking. In the WAN scenario (Scen.WAN) the delay (dWAN p 5 1 dSWITCH p 4) is set to a typical value of 10 ms. Finally, the number of sources in each group is set in the range [20¼100]. 4. Results and evaluation 4.1. Coef®cient of correlation In this section we examine the interaction between TCP and ATM layer congestions using the CWND±ACR correlation coef®cient. This analysis for Scen.LAN and Scen.WAN (for group 2) by varying the number of sources are shown in Tables 1 and 2. For both environments, a weak CWND±ACR correlation is observed for all different number of sources for TCP Reno (except when the number of sources is equal to 1) whereas a strong association between CWND and ACR is observed for the Agentbased algorithm. In the case of TCP Reno, the sharing of common switch resources by all connections and the sharing of the bandwidth with the VBR application make the behaviour of CWND and ACR not related at all. To capture the TCP behaviour during simultaneous opening of multiple TCP connections and compare the actual TCP source window with CWNDACR, the observations are made from warm-up time under Scen.LAN.
Fig. 4(a)±(c) shows ABR ACR, TCP window CWND and CWNDACR, respectively. We recall that CWNDACR is the TCP sender congestion window provided by the agent. In Fig. 4(c) CWNDACR is used instead of CWND. For a connection running at 155.52 Mbps, the inter-RM cell time is 86.4 ms and determines the responsiveness of the system. This results in high-frequency rate variations caused by the ABR feedback. Consequently this results in ACR not remaining constant but quickly coming up and down in steps on the receipt of every BRM cell. In contrast, TCP sets CWND to one segment and is doubled once every RTT (on receipt of an ACK) either at the very beginning of the connection or when recovering from a packet loss. During the congestion avoidance phase CWND is linearly increased. The retransmission timers are in general set in the range [100¼500 ms] for fast and slow timeouts, respectively. This results in CWND slowly changing on the receipt of every ACK or timeout, and highlights synchronisation problems between TCP ACKs and ABR RM cells dynamics. For example, simulation showed that uCWND 2 CWNDACRu , 128 bytes counted for 0.35% only. 4.2. Performance results In this section the simulation results for the algorithm discussed in Section 2.2 (Agent-based) under scenarios Scen.LAN and Scen.WAN are presented and compared to those of TCP Reno. The focus is mainly on the assessment of congestion under various numbers of sources and their effect on the performance metrics used in the evaluation of simulation tests. It is desirable to observe performance at both TCP and ATM layers. For ABR, the performance of TCP over ATM is measured by important factors such as CLR, PRR and Throughput. PRR is de®ned as the ratio of the total number of TCP packets retransmitted to the total number of TCP packets sent by the TCP source. TCP throughput is measured at the destination TCP layers and is de®ned as the total number of bytes delivered to the destination application, divided by the total simulation time.
Fig. 4. Behaviour of the transmission rates: (a) actual ACR; (b) actual CWND; (c) CWNDACR (Scen.LAN, Source1).
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
857
Fig. 5. Performance metrics Ð LAN environment.
Throughput for each source as well as the average throughput is measured. All throughputs are based on the number of useful packets received by the destination endpoint (Goodput). Retransmitted packets are not considered. Similarly, CLR and PRR for each connection as well as their average ®gures for all connections are measured. The analysis is based on long-term congestion behaviour of the network. Each simulation was run divided into 10 batches each with an additional warmup time. Average of these 10 batches is considered. The con®dence of the results is 95%. The simulation run times are suf®ciently large to capture long term congestion behaviour. In both scenarios all the sources start transmitting at the same time. 4.2.1. LAN environment A comparison of CLR, PRR, and Throughput is shown in Fig. 5 for different number of sources. Agent-based exhibits very low CLR and PRR. For a number of sources N in the range [5¼60] Agent-based exhibits zero cell loss and packet retransmission. For N in the range [5¼20] Agentbased exhibits a consistent throughput improvement compared to TCP Reno (up to 25% for N 5 and 15% for N 10: However, the throughput is comparable for
both TCP Reno and the Agent-based algorithm for N $ 40 (5% improvement only). 4.2.2. WAN environment Analysis of the results in the four switch con®guration is more complex due to the number of switches and groups involved. The vast difference in characteristics of the groups makes comparisons between different groups complex. The performance results for group 2 are presented in Fig. 6. In general the following were observed: (1) the WAN scenario resulted in groups 1 and 2 having similar CLR and PRR. Group 3 encountered the highest losses and retransmissions while group 4 had the lowest; (2) throughput measurements were similar for groups 1 and 3. Groups 2 and 4 had higher values. This is reasonable because groups 1 and 3 experience bottlenecks at switch 2 and again at switch 3; (3) switch 0 experienced a low congestion whereas switch 2 experienced a high one (indicating that it is most likely a bottleneck). We deliberately increased the number of sources from 20 to 100 for each group to create a congestion situation under a small buffer size (512 kB) and understand the TCP behaviour in scenario Scen.WAN. Performance results for Scen.WAN are similar to those of Scen.LAN (CLR, PRR and Throughput) (Fig. 6). On the whole and
Fig. 6. Performance metrics Ð WAN environment.
858
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
for all groups, the Agent-based algorithm exhibits lower CLR and PRR than TCP Reno. It also shows a better throughput than TCP Reno for a number of TCP sources in the range [20¼100] (up to 10% for N 100: 4.2.3. Discussion On the whole, the Agent-based algorithm produced more favourable results for these LAN and WAN con®gurations. An overall comparison of the results indicates that the Agent-based algorithm performs better relative to TCP Reno. This is clearly seen from all CLR and PRR graphs and in throughput results for varying the number of sources. The same observations are made when setting the MSS to 9180 bytes [16]: a similar consistent pattern of improvement for the Agent-based algorithm over TCP Reno is noticeable. The good performance exhibited by the Agent-based algorithm is explained as follows. When the TCP sources start transmission they each send one segment of data (e.g. as part of their slow start phase). These data may arrive at the switches almost at the same time. However, the switch with small buffer size can serve only a few numbers of connections and the rest are refused. The affected connections thus timeout and cannot transmit for sometime. However, the successful connections, which are supposed to enjoy switch resource, do not increase their rate (as in the earlier implementation of TCP) but will decrease it instead because the network is experiencing congestion. The reason is that ABR sources decrease their rate and consequently TCP window size sources as well (a decrease in ACR results in a decrease of CWNDACR as well). When the originally affected sources come out from packet loss timeout, they are likely to face a switch relieved of congestion. Congestion control algorithms implemented at the ATM switch successfully control the TCP sources transmission rate. As long as the TCP sources respond to feedback from the ATM network, and by manipulating this feedback, the network can in¯uence the TCP sources to behave in its desired manner. This somehow illustrates a congestion prevention as the adverse performance effects due to packet losses is avoided. The relatively decrease in CLR and PRR in the Agent-based algorithm compared with TCP Reno suggests that full control of TCP sources by the ATM switch provides better performance: TCP source rate noti®cation must be given at a rate suf®cient to prevent packet loss due to buffer over¯ow and/or cause under-utilisation of the bottleneck links. Moreover, our rate-to-window translation scheme can be based on any rate-based ABR's algorithm. Our implementation uses ERICA's feedback and inherits its fairness, stability and robustness properties. Of course the maximum throughput achieved in all cases is smaller than the available capacity. Compared to the link speed of 155.52 Mbps, the loss of bandwidth is due to various protocol (TCP, IP, LLC, AAL5) headers and the VBR application running in the background.
5. Conclusions In this paper, we have examined a way of improving end-host congestion control algorithms for TCP over ATM-ABR. The correlation coef®cient between CWND and ACR was proposed to study the association between TCP layer and ATM layer. It was observed that both CWND and ACR do not re¯ect the association between congestion experienced at the two layers and that CWND and ACR are weakly correlated. The introduction of a new agent at the edge of the network coordinates the TCP sender and ATM source. The task of this agent is to keep the ratio CWND/(ACR p 53) close to the newly de®ned parameter MAXCWND/(PCR p 53), by conveying ABR rate feedback to TCP sources and translating it into a window value. Our idea is based on regulating the circulation of packets rather than that of ACKs thanks to the ABR rate feedback. Through the experiments, our goal was to show that the congestion control algorithms implemented at the ATM switch successfully controls the TCP sources, in other words TCP congestion control algorithms become rate limited. We have shown through experiments that it is desirable for TCP source to change its transmission rate according to a full ATM switch control in order to avoid oscillations inherent in TCP's windowing algorithm. Our key contribution was to demonstrate that rate-controlled TCP sources can achieve low PRR in addition to better throughput performance. Our method does not require changes to TCP header formats or the protocol, and can be implemented in ATM edge devices/switches. To provide the ATM platform for the future experimental set-up, the ATM on Linux distribution from EPFL (Ecole Polytechnique FeÂdeÂrale de Lausanne) will be used [3]. So far, we have been using the (explicit) coordinator agent to understand and assess coherence issues between TCP and ATM. The next step is to use active networks [8] to implement the functionality of the agent. In an (ATM) active network, nodes can perform computations on user data as it traverses the network, and applications are provided with mechanisms to select and program the computation that occurs. The challenge is to design new elaborated traf®c and congestion control algorithms (at TCP and ATM levels via active networking) to optimise network resources and to guarantee QoS requirements. The use of active networks can be either as a facilitator, i.e. an implicit agent when TCP runs over ATM or a new service with capabilities designed for TCP. References [1] Proceedings of the Third International Conference on Autonomous Agents (Agents'99), Seattle, Washington, May, 1999, ACM, http:// www.cs.washington.edu/research/agents99/. [2] M. Allman, V. Paxson, W. Stevens, TCP Congestion Control, RFC 2581, April 1999.
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859 [3] W. Almesberger, ATM on Linux, Institute for Computer Communications and Applications, Ecole Polytechnique FeÂdeÂrale de Lausanne, Switzerland, November, 1999, http://icawww1.ep¯.ch/linux-atm. [4] ATM Forum Technical Committee, Traf®c Management Speci®cation, Version 4.1, AF-TM-0121.000, March 1999. [5] M. Baumann, YATS User's Manual for Version 0.3, Communications Laboratory, Dresden University of Technology, Germany, January 1998, http://www.ifn.et.tu-dresden.de/tk/yats/yats.html. [6] L.S. Brakmo, S.W. O'Malley, L.L. Peterson, TCP Vegas: new techniques for congestion detection and avoidance, Proceedings of SIGCOMM'94, August 1994, ftp://ftp.cs.arizona.edu/xkernel/ papers/vegas.ps. [7] K. Djemame, M. Kara, Proposals for a coherent approach to cooperation between TCP and ATM congestion control algorithms, in: J.T. Bradley, N.J. Davies (Eds.), Proceedings of the 15th Annual UK Performance Engineering Workshop (UKPEW'99), Bristol, UK, July 1999, pp. 273±284, http://www.scs.leeds.ac.uk/atm-mm/papers/ ukpew99.ps.gz. [8] T. Faber, ACC: using active networking to enhance feedback congestion control mechanisms, IEEE Network 12 (3) (1998) 61±65 (http:// www.isi.edu/~faber/pubs/html/network98/). [9] S. Floyd, TCP and explicit congestion noti®cation, ACM Computer Communication Review 24 (5) (1994) 8±23. [10] V. Jacobson, Congestion avoidance and control, ACM Computer Communication Review 18 (1988) 314±329 (Proceedings of Sigcomm'88 Symposium, Stanford, CA). [11] L. Kalampoukas, A. Varma, K.K. Ramakrishnan, Explicit window adaptation: a method to enhance TCP performance, Proceedings of INFOCOM'98, San Francisco, CA, April 1998. [12] S. Kalyanaraman, R. Jain, S. Fahmy, R. Goyal, Performance and buffering requirements of internet protocols over ATM ABR and UBR services, IEEE Communications Magazine 36 (6) (1998) 152±157 (http://www.cis.ohio-state.edu/~jain/papers/ieee-mag.htm).
859
[13] S. Kalyanaraman, R. Jain, S. Fahmy, R. Goyal, B. Vandalore, The ERICA switch algorithm for ABR traf®c management in ATM networks, IEEE/ACM Transactions on Networking February (2000) 87±98, (http://www.cis.ohio-state.edu/~jain/papers/ erica.htm). [14] M. Kara, M. Rahin, Towards a framework for performance evaluation of TCP behaviour over ATM networks, Proceedings of the 13th IFIP International Conference on Computer Communication, ICCC'97, November 1997, pp. 49±60, ftp://agora.leeds.ac.uk/scs/doc/reports/ 1997/97_42.ps.Z. [15] A. Koike, TCP Flow Control with ACR Information, ATM Forum/970998, December 1997. [16] M. Laubach, J. Halpern, Classical IP and ARP over ATM, RFC 2225, April 1998. [17] M. Mathis, J. Madhavi, S. Floyd, A. Romanow, TCP Selective Acknowledgement Options, RFC 2018, October 1996, ftp:// ftp.ee.lbl.gov/papers/rfc2018.ps. [18] P. NarvaÂez, K.-Y. Siu, Acknowledgement bucket scheme for regulating TCP ¯ow over ATM, Computer Networks and ISDN 30 (19) (1998) 1775±1791, (http://perth.mit.edu/~pnarvaez/mypapers/ paper3 _tech.ps.gz). [19] K. Ramakrishnan, S. Floyd, A Proposal to Add Explicit Congestion Noti®cation (ECN) to IP, RFC 2481, January 1999. [20] A. Romanow, S. Floyd, Dynamics of TCP traf®c over ATM networks, IEEE Journal on Selected Areas in Communications 13 (4) (1995) 633±641. [21] R. Satyavolu, K. Duvedi, S. Kalyanaraman, Explicit Rate Control of TCP Applications, ATM_Forum/98-0152R1, February 1998, http:// www.ecse.rpi.edu/homepages/shivkuma. [22] D. Sisalem, H. Schulzrinne, End-to-end rate control in ABR, Proceedings of the First Workshop on ATM Traf®c Management (WATM'95), Paris, France, December 1995, 281±287, IFIP WG 6.2, http://www.cs.columbia.edu/~hgs/papers/sisa9512_end.ps.gz.
www.elsevier.com/locate/comcom
Agent-based rate coordination between TCP and ABR congestion control algorithms q K. Djemame*, M. Kara ATM-Multimedia Group 1, School of Computer Studies, University of Leeds, Leeds LS2 9JT, UK Received 6 November 2000; accepted 6 November 2000
Abstract In this paper we study the extent of negative interactions between Transmission Control Protocol (TCP) and Asynchronous Transfer Mode (ATM) congestion control through the application of an approach based on coordination via explicit cooperation. To do so, we implement an agent that bridges the gap between the ATM layer and the TCP layer in the protocol stack at the sender end to coordinate the congestion control algorithms of the TCP transport protocol and the ATM cell-oriented switching architecture. The novel idea uses ATM rate-based ¯ow control to adjust the credit-based window size of TCP via the implementation of the agent. The effects of running TCP over Available Bit Rate (ABR) are studied with the help of two simulation models (LAN and WAN). Firstly, we show that TCP window (CWND) and ABR Actual Cell Rate (ACR) are weakly correlated. Secondly, we derive an Agent-based algorithm to ensure that even when the ATM switch queues are indicating no congestion, it is the switch (and not the source) which should control the increment in the TCP source rates. The introduction of the agent at the edge of the network coordinates the TCP and ATM sending transmission and clearly demonstrates a strong correlation coef®cient between CWND and ACR. The simulation results show that coordination at the edge of the network makes better use of available bandwidth and improves TCP's performance over ATM for a range of performance metrics such as Cell Loss Ratio, Packet Retransmission Ratio and Throughput. q 2001 Elsevier Science B.V. All rights reserved. Keywords: Agent-based rate coordination; Transmission control protocol; Asynchronous transfer mode; Available bit rate; Performance evaluation
1. Introduction The Transmission Control Protocol (TCP) has emerged in the last few years as the standard in data communication. It is a reliable transport protocol that uses a window-based ¯ow and error control. Asynchronous Transfer Mode (ATM) is expected to be the technology of choice for high-speed computer networks. It was developed as part of the work on broadband Integrated Service Digital Network (ISDN) (but is still used in non-ISDN environments) where very high data rates are required. Available Bit Rate (ABR) is a service category de®ned by the ATM Forum [4] which supports applications with vague requirements for throughput and delay, for example data traf®c. ABR uses a rate-based congestion control based on q This work is funded in part by the Engineering and Physical Science Research Council (EPSRC) under project GR/L86937. * Corresponding author. Tel.: 144-113-233-6590; fax: 144-113-233-5468. E-mail addresses: [email protected] (K. Djemame), [email protected] (M. Kara). 1 http://www.scs.leeds.ac.uk/atm-mm.
network's feedback. Feedback information about network congestion is conveyed from network elements to end systems in order to dynamically adapt the cell emission process of the ABR sources to the current load of the network. However, a transport protocol such as TCP encounters problems when running over ABR due to its window-based congestion control interferences with ABR's rate-based. The goal of this paper is to propose an approach to cooperation between TCP and ABR congestion control algorithms to improve overall performance. 1.1. TCP control TCP is a very popular transport protocol, and is the most widely used for today's Internet data applications. It combines logic for routing through an Internet with endto-end control. TCP is actually a collection of algorithms that send packets into the network without a reservation and then react to observable events to occur. Such algorithms include congestion control and recovery from loss of packets [2]. Each sender maintains two windows: the window RWND
0140-3664/01/$ - see front matter q 2001 Elsevier Science B.V. All rights reserved. PII: S 0140-366 4(00)00357-1
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Nomenclature AAL5 ATM adaptation layer 5 ABR Available bit rate ACK Acknowledgement ACR Allowed cell rate ATM Asynchronous transfer mode BRM Cell Backward resource management cell CI Congestion indication CLR Cell loss ratio CWND Congestion window ECN Explicit congestion noti®cation EFCI Explicit forward congestion indication EPD Early packet discard ER Explicit rate ERICA Explicit rate indication for congestion avoidance FRM cell Forward resource management cell LAN Local area network LLC Logical link control IP Internet protocol ISDN Integrated service digital network MCR Minimum cell rate MSS Maximum segment size NI No increase PCR Peak cell rate PPD Partial packet discard PRR Packet retransmission ratio QoS Quality of service RDF Rate decrease factor RIF Rate increase factor RTO Retransmission time out RTT Round-trip time SSTHRESH Slow start threshold TCP Transmission control protocol TTCA Time to transmit CWND at ACR VBR Variable bit rate VC Virtual circuit WAN Wide area network
the receiver has granted and a second window CWND called the congestion window. The minimum of the congestion window and the receiver's window is used when sending packets. The congestion control mechanism has two distinct phases. Slow Start and Congestion Avoidance. Upon starting a connection or restarting after a packet loss, the congestion window size is set to one packet, and then doubled once every Round-Trip Time (RTT), i.e. upon the receipt of an acknowledgement (ACK). Congestion avoidance takes care of violation which would occur if the retransmit time is too short. During this phase, CWND is linearly increased as opposed to exponential increase during slow start phase. A third parameter SSTHRESH is used to start the congestion phase and is usually initially set to
851
64 kB (kilobytes). Upon noti®cation of network congestion, i.e. a timeout, SSTHRESH is set to half the current CWND and CWND is set to one and the slow start phase restarts once again. There are several ways to detect a packet loss: (1) a timer: when the timer is greater than the Retransmission Time Out (RTO), a packet is assumed to be lost; and (2) when a packet has been acknowledged more than three times, the next packet is assumed to be lost. Earlier implementations of TCP were distributed in releases corresponding to the original implementation of Jacobson [10]. However, over the years, there have been some ®netuning, improvements and additions to the original TCP algorithms. Some of the proposed changes have been widely adopted and are part of TCP implementations today, e.g. fast retransmit and recovery [2]. TCP-Tahoe and TCP Reno are the (most) deployed current versions. Other revisions for TCP have been proposed and include TCP Vegas [6] and Selective ACKs [17]. 1.2. ABR mechanism The ABR control unit adjusts the rate of ATM cells into the network dynamically according to the congestion status of the network. This rate is called the Actual Cell Rate (ACR). The source is not allowed to send cells faster than ACR to avoid network congestion. The rate-based control scheme (explicit rate or relative rate marking (EFCI) uses special Resource Management (RM) cells [4]. In the Explicit Rate (ER) scheme, the source sends an RM cell once in every Nrm (default Nrm 32 cells or in Trm (default Trm 100 ms time units along the Virtual Circuit (VC) to the destination. Each cell contains three ®elds that provide feedback to the source: a Congestion Indication (CI) bit, a No Increase (NI) bit, and an ER ®eld. The source sets the RM cell's ER ®eld to the rate at which it likes to send and transmits the Forward RM (FRM) Cell. As each FRM cell is received by the destination, it is turned around and transmitted back to the source as a Backward RM (BRM) Cell. Any of the CI, NI and ER ®elds may be changed by an ATM switch along the VC or the destination before the corresponding BRM cell returns to the source. ACR always varies between the limits of Minimum Cell Rate (MCR) and Peak Cell Rate (PCR). Several rules are de®ned in Ref. [4] for modifying ACR according to the received information. The major ones are: NI bit 0 and CI bit 0; ACR Ã max[MCR, min[ER, PCR, ACR 1 RIF £ PCR]] NI bit 1 and CI bit 0; ACR Ã max[MCR, min[ER, ACR]] CI bit 1; ACR Ã max[MCR, min[ER, ACR(1 2 RDF)]] Rate Increase Factor (RIF) and Rate Decrease Factor (RDF) are de®ned in Ref. [4] (default value: 1/16).
852
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
1.3. Problems and motivation of the work Integrating ATM technology has revealed poor throughput performance of TCP over ATM [20]. This poor performance is mainly due to packet fragmentation which occurs when a TCP packet ¯ows into an ATM VC though the ATM Adaptation Layer 5 (AAL5). AAL5 is responsible for dividing TCP's packets into sets of 48 byte data units called cells. Fragmentation at the AAL5 is necessary since the typical size of a TCP packet is much larger than that of a cell. The loss of even a single cell in any of the ATM network's switches results in the corruption of the entire packet to which that cell belongs, and consequently causes its retransmission. Techniques to improve cell discard behaviour such as Partial Packet Discard (PPD) and Early Packet Discard (EPD) are aimed at reducing the transmission of useless cells [20]. Various issues have been investigated for the performance evaluation of TCP over ATM. These issues may be divided into three groups: those addressing the dynamics of TCP, those addressing the behaviour of ATM and those which address the interaction between TCP's window-based and ATM layer congestion control mechanisms [14]. Although performance evaluation of TCP over ATM has been extensively studied, progress has been made only in identifying localised solutions to particular problems such as buffering requirements [12]. There is much to be done on the overall effect of these schemes when interacting with each other in order to focus the attention towards coherent solutions. On a TCP connection over ATM implementing the ABR service, the problems come from the interaction between both congestion control methods. TCP and ABR use two different ways to detect congestion in the network: TCP uses packet losses to regulate its rate (window) whereas ABR uses the feedback information transported by RM cells to control its ACR. It is worth mentioning that ABR does not wait for a packet loss to reduce its rate. Also, in case of congestion, the ABR control loop is designed to slow down the ABR source. With current implementations of TCP, ATM networks are limited to packet drops as the only viable mechanism to inform TCP sources of congestion. Thus, noti®cation is implicit and ABR is unable to signal to the real (TCP) traf®c sources the rate at which they should transmit (although the mechanism to use Explicit Congestion Noti®cation (ECN) has been proposed [9,19], it has not been implemented yet). Our aim in this paper is to study the extent of negative interactions between TCP and ATM congestion control through the application of an approach based on coordination via explicit cooperation. To do so, we implement an agent that bridges the gap between the ATM layer and the TCP layer in the protocol stack at the sender end to coordinate the congestion control algorithms of the TCP transport protocol and the ATM cell-oriented switching architecture. Agents are an important area of research and development in various domains such as arti®cial intelligence, high-
performance computing and communication networks [1]. Our proposal is driven from our interest in proposing a framework for a coherent approach for better coordination between TCP and ATM congestion control algorithms [7]. The effects of running TCP over ABR are studied with the help of a simulation model in a LAN and WAN environment. The TCP model is designed to simulate data transfer between n hosts over the ATM network. On a typical backbone link, the TCP queues are at the edge router and not inside the ATM network. Firstly, we show that TCP window (CWND) and ABR ACR are weakly correlated. Secondly, we derive an Agent-based algorithm to ensure that even when the ATM switch queues are indicating no congestion, it is the switch (and not the source) which should control the increment in the TCP source rates. The introduction of the agent at the edge of the network coordinates the TCP and ATM sending transmission and clearly demonstrates a strong correlation coef®cient between CWND and ACR. The simulation results show that coordination at the ATM network's edge makes better use of available bandwidth and a consistent improvement of TCP's performance over ATM for performance metrics such as the Cell Loss Ratio (CLR), the Packet Retransmission Ratio (PRR) and the Throughput. 1.4. Related work The closest work relative to this paper was proposed by Satyavolu et al. [21], NarvaÂez and Siu [18], Koike [15] and Kalampoukas et al. [11]. These researchers independently suggested the idea of overwriting the ACK ®elds and pacing the ACKs (called acknowledgement bucket by these authors). Satyavolu et al. suggest a rate-to-window translation scheme based upon the Explicit Rate Indication for Congestion Avoidance (ERICA) ATM ABR algorithm [21]. NarvaÂez and Siu suggest the implementation of somewhat complex TCP emulation engine at the rate-controller and an acknowledgement bucket [18]. Koike also suggests the use of an acknowledgement bucket [15]. Further, these authors explored the idea of overwriting the ACK ®elds and pacing the ACKs from the point of view of TCP/IP-ATM internetworking, i.e. extending the ATM ABR type rate control from ATM edges to TCP end systems. Kalampoukas et al. propose an approach based on buffer occupancy [11]. Their work shows that explicit window control is more effective than buffer management techniques due to its inherence congestion (and loss) avoidance approach. Unlike the approaches in Refs. [15,18], our idea is based on regulating the circulation of packets rather than that of ACKs. It also differs from the work in Refs. [11,21] in that we take an existing ABR rate control scheme and do a rate to (TCP) window translation. Our approach uses a more direct translation of the ABR feedback into an ideal TCP window that can lead to burstiness avoidance in network traf®c.
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
853
Fig. 1. (a) Coordinating TCP and ABR sources. (b) Role of the agent.
1.5. Paper organisation We begin Section 2 by presenting our proposal for a new congestion control scheme. The simulation environment is presented in Section 3. Performance results over the simulation model are discussed in Section 4. Some concluding remarks and perspectives of the research for future developments are given in Section 5. A list of acronyms is included as nomenclature. 2. Proposal for a new congestion control scheme In TCP, the maximum traf®c is controlled by the window CWND, while in ABR the traf®c is controlled by MCR, PCR and ACR. A key factor that impacts user and network performance for TCP over ABR is how well the traf®c management mechanism used in TCP end system, ATM end system and switch mesh together in providing good end-to-end performance. At the edge of the ATM network, the possibility of using ER indication instead of backward congestion noti®cation is an interesting enhancement to TCP, especially as different studies have shown that using the ER indication mode in ATM results in much better utilisation and fairness results compared to backward congestion noti®cation schemes [22]. This resemblance will allow better integration of ATM and TCP as both schemes would react in a similar way at the same congestion information. Also, handling congestion in advance reduces the amount of buffer required in the intermediate switches and the delay caused by retransmissions. 2.1. Assessment of coordination The association between the TCP sender window and the ACR may be analysed using the correlation coef®cient between CWND and ACR over a set of observations as follows: E Ci 2 C i Ai 2 A i p Corr C; A p E Ci 2 C i 2 E Ai 2 A i 2 where Ci and Ai are the ith CWND and corresponding ACR
observations. Corr(C,A) measures the association between CWND and ACR. If Corr(C,A) is equal to zero, CWNDi and ACRi are independent. However, if the value of Corr(C,A) is close to 21 or 1 then there is a strong association between them. Change in ACR is made at the ABR source when receiving a BRM cell containing feedback provided by the network, mainly a CI bit, a NI bit and the ER. Thus, the source keeps transmitting at the same rate during the time elapsing between the transmission of the FRM Cell and the reception of the BRM Cell. If the rate-based approach uses a negative polarity of feedback, RM cells are sent only to decrease the rate. By using positive polarity RM cells are required to increase the rate. TCP sender updates CWND either at RTT when receiving an ACK (increase of CWND) or when a timeout occurs (decrease of CWND). The CWND±ACR correlation may be used as a tool to compare the effectiveness of: (1) buffer allocation schemes; (2) different cell dropping policies; and (3) different number of TCP sources. 2.2. Coordination at ATM network's boundaries Our contribution involves calculating the rate feedback in the ABR switch algorithm and translate it into a window value to give feedback to TCP. This method exploits the fact that TCP rate is controlled by the ABR rate. CWND and ACR are both instantaneous indicators of the window size and the cells transmission speed at their respective layers. The (normalised) ratio CWND/ACR gives the time TTCA it takes a CWND window (bytes) to be transmitted at ACR (cells/s). However, TTCA can readily be obtained from the ratio of the normalised CWND (bytes) to normalised ACR (bytes/s): 8 > < CWND ; ACR ± 0 TTCA ACR p 53 > : not defined; otherwise The notion of TTCA is by de®nition nonexistent in situations where ACR becomes nil. The number 53 in the above equation is derived from the (payload 1 header) size of ATM cells. In non-zero ACR situations, TTCA is a positive
854
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Because of the dynamic nature of the available bandwidth for ABR service, the ER (and consequently ACR) command to the source is expected to change constantly. Thus, variations in ABR's ACR occur at a very high rate (small time scales) while TCP's window adjustments operate over longer time scales. In order to avoid an oscillation of the protocol, a scheme is implemented which treats the advertised ACR(s) as an average allocation over the ¯ow's RTT:
Fig. 2. LAN simulation model.
number: CWND is within the range [1¼MAXCWND] and ACR within the range [MCR¼PCR]. The goal behind considering the parameter TTCA is to show that the congestion control algorithms implemented at the ATM switch successfully controls the TCP sources, in other words (window-based) TCP congestion control algorithms become rate limited as well. A value of TTCA close to the ratio MAXCWND/ (PCR p 53) is a desirable property: we would like TCP sender and ABR source to have a maximum transmission window and a maximum cell rate respectively, assuming the buffering requirements for zero cell loss at the ATM switches are ful®lled. In the worst scenarios, and assuming ACR remains constant, an increase of CWND will lead to packet drops whereas a decrease of CWND will lead to a waste of bandwidth. Assuming CWND remains constant, an increase of ACR will lead to a waste of bandwidth whereas a decrease of ACR will lead to packet drops. Both the CWND±ACR correlation coef®cient and TTCA indicators provide information relating to the coordination between the congestion measures at TCP transport and ATM physical layers. 2.2.1. Role of the agent The role of the coordinator agent at the edge of the ATM network is to coordinate TCP and ABR sources (Fig. 1(a)) by bridging the gap between the ATM layer and the TCP layer in the protocol stack at the source at each RTT. It inputs the values of ACR and PCR from the ABR source and MAXCWND from TCP sender, respectively, and outputs CWNDACR value to explicitly indicating the TCP sender rate (Fig. 1(b)). CWNDACR is a new parameter derived from TTCA and de®ned as follows: MAX_TTCA
MAXCWND ; PCR p 53
CWNDACR MAX_TTCA p ACR p 53 MAXCWND p
ACR : PCR
reset counter i to prepare for measurement on next RTT; at reception of BRM celli calculate ACRi; at each RTT: calculate the average ACRavg ACR1 1 ACR2 1 ¼ 1 ACRn =n; set CWNDACR to MAXCWND p ACRavg =PCR: 2.2.2. ABR source On receiving a BRM cell, the ABR source calculates the new ACR using the values of ER, CI and NI set in the cell. It then allows ACR and PCR information to be sent to the agent after processing the BRM cell so that the agent can compute the new CWNDACR. 2.2.3. TCP sender Given that the variations in CWND is partially controlled by the agent, the objective of the protocol here is to partially disable TCP congestion control algorithm: at each RTT we make TCP sender window limit enforceable by the ATM network to control the TCP rate. CWND is set to CWNDACR provided by the agent whatever the actual TCP sender window is upon receiving an ACK by the TCP source. Packet losses are still indicated by the basic TCP's original congestion mechanisms which are maintained. If a transmission timer expires then the usual retransmission and window reduction actions are taken: SSTHRESH (slow start threshold) is halved, CWND is set to one segment and slow start is entered. In case three duplicate ACKs are received, the TCP sender sets the value of CWND to half of its current size (in case of TCP Reno) to restart slow start (fast recovery and fast retransmit). However, the source leaves the slow start mode as soon as an ACK packet is received and new actions are performed. 3. Experimental design 3.1. Simulation experiments and objectives We describe in this section the network con®gurations and parameters used in the stimulation. The objective of the experiments is twofold. Firstly, we analyse the association between TCP sender window and the ABR ACR using the correlation coef®cient between CWND and ACR. The
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
855
Group Src.3 to Destination 3: two-hop traf®c Group Src.4 to Destination 4: one-hop traf®c 3.3. Simulation tool
Fig. 3. WAN simulation model.
analysis is done over a set of observations when running simulations with TCP Reno and the Agent-based algorithm discussed is Section 2.2. Secondly, we show that the Agentbased algorithm under various scenarios exhibits a better performance in the evaluation of the simulation tests. We consider the simulation experiments performed with the help of LAN and WAN simulation models. 3.2. Topologies LAN model: Fig. 2 illustrates our LAN simulation con®guration consisting of N TCP sources, two ATM switches and N TCP destinations. TCP sources send data whenever allowed by the TCP window. All traf®c is unidirectional (discounting the TCP ACKs and the BRM cells in the reverse direction). WAN model: The WAN simulation model is a multistage wide area network with four switches SWi i 0¼3 (Fig. 3). The sources are divided into four groups. The traf®c ¯ow is shown by dashed lines. The purpose of choosing such a complex model is to make it as realistic as possible. In this model the TCP sources each establish a single connection with the similarly numbered destination across the ATM switches and the bottleneck links: Group Src.1 to Destination 1: end-to-end traf®c Group Src.2 to Destination 2: two-hop traf®c Table 1 CWND±ACR correlation coef®cient Ð Scen.LAN Number of sources
TCP Reno
Agent-based
1 5 10 20 40 60 80 100
0.893 0.622 0.496 0.371 0.326 0.234 0.196 0.116
0.981 0.977 0.972 0.970 0.970 0.968 0.964 0.961
The YATS simulation package is used in the simulation study in Ref. [5]. YATS is a detailed simulator for TCP over ATM. Most of common features of TCP such as slow start, congestion detection, congestion avoidance, fast retransmit and fast recovery are included in the simulator. We extended the simulator in order to implement the coordinator agent and allow communication between ABR and TCP sources. The agent was developed from scratch while major changes were made to the TCP sender. Minor changes to the ABR source allowed ACR and PCR information to be sent to the agent after processing a BRM cell. Traf®c is generated by YATS speci®c objects using a TCP/IP/AAL5/ATM protocol stack. 3.4. Simulation scenario The TCP sources are greedy sources with in®nite supply of data and always have data ready to send as and when permitted by the TCP ¯ow control. A large bulk data transfer application runs on top of TCP sources. This application recognises the start±stop protocol and generates frames of 8000 bytes length. To study the congestion behaviours at ATM and TCP layers, we consider that another Variable Bit Rate (VBR) application is running in the background. VBR has higher priority than ABR, and is simulated by an ON/OFF source (1 ms ON time and 1 ms OFF time) which operates at the maximum amplitude during the ON time. We assume the ATM switch has a buffer shared by all VCs passing through. The scheduling policy at the buffer is FIFO. In order to induce variable congestion in the switches, the size of the shared buffer is varied for the simulation cases. The queues that model the switches' buffers have a unique size, which is ®xed to 512 kB (9892 cells). The reason for this choice is to offer a tight buffer constraint. The switches are con®gured with ERICA [13] with the following parameters Ð target utilisation: 95%; averaging interval length defaults: 1 ms. ERICA is a well-known algorithm which achieves the goals of fairness, high utilisation and low delay using rate-based control. TCP Maximum Segment Size (MSS) is set to 512 bytes. We take into account 20 bytes of TCP header, 20 bytes IP header, 8 bytes LLC, 8 bytes AAL5 header. TCP sources use a 64 kB window, and all links run at 155.52 Mbps. The internal delay in each ATM switch (dSWITCH) is set to 10 ms. We consider a LAN scenario (Scen.LAN) where the delay (3 p dLAN 1 2 p dSWITCH) is set to a typical value of 100 ms (Fig. 2). The number of sources N is set in the range [1¼100]. The values of N are arbitrarily selected, but nonetheless deemed suf®cient for producing traf®c ¯ow required in the simulation study of high-speed
856
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
Table 2 CWND±ACR correlation coef®cient Ð Scen.WAN Number of sources
TCP Reno
Agent-based
1 5 10 20 40 60 80 100
0.664 0.647 0.530 0.303 0.289 0.186 0.084 0.073
0.988 0.979 0.959 0.955 0.924 0.919 0.825 0.814
networking. In the WAN scenario (Scen.WAN) the delay (dWAN p 5 1 dSWITCH p 4) is set to a typical value of 10 ms. Finally, the number of sources in each group is set in the range [20¼100]. 4. Results and evaluation 4.1. Coef®cient of correlation In this section we examine the interaction between TCP and ATM layer congestions using the CWND±ACR correlation coef®cient. This analysis for Scen.LAN and Scen.WAN (for group 2) by varying the number of sources are shown in Tables 1 and 2. For both environments, a weak CWND±ACR correlation is observed for all different number of sources for TCP Reno (except when the number of sources is equal to 1) whereas a strong association between CWND and ACR is observed for the Agentbased algorithm. In the case of TCP Reno, the sharing of common switch resources by all connections and the sharing of the bandwidth with the VBR application make the behaviour of CWND and ACR not related at all. To capture the TCP behaviour during simultaneous opening of multiple TCP connections and compare the actual TCP source window with CWNDACR, the observations are made from warm-up time under Scen.LAN.
Fig. 4(a)±(c) shows ABR ACR, TCP window CWND and CWNDACR, respectively. We recall that CWNDACR is the TCP sender congestion window provided by the agent. In Fig. 4(c) CWNDACR is used instead of CWND. For a connection running at 155.52 Mbps, the inter-RM cell time is 86.4 ms and determines the responsiveness of the system. This results in high-frequency rate variations caused by the ABR feedback. Consequently this results in ACR not remaining constant but quickly coming up and down in steps on the receipt of every BRM cell. In contrast, TCP sets CWND to one segment and is doubled once every RTT (on receipt of an ACK) either at the very beginning of the connection or when recovering from a packet loss. During the congestion avoidance phase CWND is linearly increased. The retransmission timers are in general set in the range [100¼500 ms] for fast and slow timeouts, respectively. This results in CWND slowly changing on the receipt of every ACK or timeout, and highlights synchronisation problems between TCP ACKs and ABR RM cells dynamics. For example, simulation showed that uCWND 2 CWNDACRu , 128 bytes counted for 0.35% only. 4.2. Performance results In this section the simulation results for the algorithm discussed in Section 2.2 (Agent-based) under scenarios Scen.LAN and Scen.WAN are presented and compared to those of TCP Reno. The focus is mainly on the assessment of congestion under various numbers of sources and their effect on the performance metrics used in the evaluation of simulation tests. It is desirable to observe performance at both TCP and ATM layers. For ABR, the performance of TCP over ATM is measured by important factors such as CLR, PRR and Throughput. PRR is de®ned as the ratio of the total number of TCP packets retransmitted to the total number of TCP packets sent by the TCP source. TCP throughput is measured at the destination TCP layers and is de®ned as the total number of bytes delivered to the destination application, divided by the total simulation time.
Fig. 4. Behaviour of the transmission rates: (a) actual ACR; (b) actual CWND; (c) CWNDACR (Scen.LAN, Source1).
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
857
Fig. 5. Performance metrics Ð LAN environment.
Throughput for each source as well as the average throughput is measured. All throughputs are based on the number of useful packets received by the destination endpoint (Goodput). Retransmitted packets are not considered. Similarly, CLR and PRR for each connection as well as their average ®gures for all connections are measured. The analysis is based on long-term congestion behaviour of the network. Each simulation was run divided into 10 batches each with an additional warmup time. Average of these 10 batches is considered. The con®dence of the results is 95%. The simulation run times are suf®ciently large to capture long term congestion behaviour. In both scenarios all the sources start transmitting at the same time. 4.2.1. LAN environment A comparison of CLR, PRR, and Throughput is shown in Fig. 5 for different number of sources. Agent-based exhibits very low CLR and PRR. For a number of sources N in the range [5¼60] Agent-based exhibits zero cell loss and packet retransmission. For N in the range [5¼20] Agentbased exhibits a consistent throughput improvement compared to TCP Reno (up to 25% for N 5 and 15% for N 10: However, the throughput is comparable for
both TCP Reno and the Agent-based algorithm for N $ 40 (5% improvement only). 4.2.2. WAN environment Analysis of the results in the four switch con®guration is more complex due to the number of switches and groups involved. The vast difference in characteristics of the groups makes comparisons between different groups complex. The performance results for group 2 are presented in Fig. 6. In general the following were observed: (1) the WAN scenario resulted in groups 1 and 2 having similar CLR and PRR. Group 3 encountered the highest losses and retransmissions while group 4 had the lowest; (2) throughput measurements were similar for groups 1 and 3. Groups 2 and 4 had higher values. This is reasonable because groups 1 and 3 experience bottlenecks at switch 2 and again at switch 3; (3) switch 0 experienced a low congestion whereas switch 2 experienced a high one (indicating that it is most likely a bottleneck). We deliberately increased the number of sources from 20 to 100 for each group to create a congestion situation under a small buffer size (512 kB) and understand the TCP behaviour in scenario Scen.WAN. Performance results for Scen.WAN are similar to those of Scen.LAN (CLR, PRR and Throughput) (Fig. 6). On the whole and
Fig. 6. Performance metrics Ð WAN environment.
858
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859
for all groups, the Agent-based algorithm exhibits lower CLR and PRR than TCP Reno. It also shows a better throughput than TCP Reno for a number of TCP sources in the range [20¼100] (up to 10% for N 100: 4.2.3. Discussion On the whole, the Agent-based algorithm produced more favourable results for these LAN and WAN con®gurations. An overall comparison of the results indicates that the Agent-based algorithm performs better relative to TCP Reno. This is clearly seen from all CLR and PRR graphs and in throughput results for varying the number of sources. The same observations are made when setting the MSS to 9180 bytes [16]: a similar consistent pattern of improvement for the Agent-based algorithm over TCP Reno is noticeable. The good performance exhibited by the Agent-based algorithm is explained as follows. When the TCP sources start transmission they each send one segment of data (e.g. as part of their slow start phase). These data may arrive at the switches almost at the same time. However, the switch with small buffer size can serve only a few numbers of connections and the rest are refused. The affected connections thus timeout and cannot transmit for sometime. However, the successful connections, which are supposed to enjoy switch resource, do not increase their rate (as in the earlier implementation of TCP) but will decrease it instead because the network is experiencing congestion. The reason is that ABR sources decrease their rate and consequently TCP window size sources as well (a decrease in ACR results in a decrease of CWNDACR as well). When the originally affected sources come out from packet loss timeout, they are likely to face a switch relieved of congestion. Congestion control algorithms implemented at the ATM switch successfully control the TCP sources transmission rate. As long as the TCP sources respond to feedback from the ATM network, and by manipulating this feedback, the network can in¯uence the TCP sources to behave in its desired manner. This somehow illustrates a congestion prevention as the adverse performance effects due to packet losses is avoided. The relatively decrease in CLR and PRR in the Agent-based algorithm compared with TCP Reno suggests that full control of TCP sources by the ATM switch provides better performance: TCP source rate noti®cation must be given at a rate suf®cient to prevent packet loss due to buffer over¯ow and/or cause under-utilisation of the bottleneck links. Moreover, our rate-to-window translation scheme can be based on any rate-based ABR's algorithm. Our implementation uses ERICA's feedback and inherits its fairness, stability and robustness properties. Of course the maximum throughput achieved in all cases is smaller than the available capacity. Compared to the link speed of 155.52 Mbps, the loss of bandwidth is due to various protocol (TCP, IP, LLC, AAL5) headers and the VBR application running in the background.
5. Conclusions In this paper, we have examined a way of improving end-host congestion control algorithms for TCP over ATM-ABR. The correlation coef®cient between CWND and ACR was proposed to study the association between TCP layer and ATM layer. It was observed that both CWND and ACR do not re¯ect the association between congestion experienced at the two layers and that CWND and ACR are weakly correlated. The introduction of a new agent at the edge of the network coordinates the TCP sender and ATM source. The task of this agent is to keep the ratio CWND/(ACR p 53) close to the newly de®ned parameter MAXCWND/(PCR p 53), by conveying ABR rate feedback to TCP sources and translating it into a window value. Our idea is based on regulating the circulation of packets rather than that of ACKs thanks to the ABR rate feedback. Through the experiments, our goal was to show that the congestion control algorithms implemented at the ATM switch successfully controls the TCP sources, in other words TCP congestion control algorithms become rate limited. We have shown through experiments that it is desirable for TCP source to change its transmission rate according to a full ATM switch control in order to avoid oscillations inherent in TCP's windowing algorithm. Our key contribution was to demonstrate that rate-controlled TCP sources can achieve low PRR in addition to better throughput performance. Our method does not require changes to TCP header formats or the protocol, and can be implemented in ATM edge devices/switches. To provide the ATM platform for the future experimental set-up, the ATM on Linux distribution from EPFL (Ecole Polytechnique FeÂdeÂrale de Lausanne) will be used [3]. So far, we have been using the (explicit) coordinator agent to understand and assess coherence issues between TCP and ATM. The next step is to use active networks [8] to implement the functionality of the agent. In an (ATM) active network, nodes can perform computations on user data as it traverses the network, and applications are provided with mechanisms to select and program the computation that occurs. The challenge is to design new elaborated traf®c and congestion control algorithms (at TCP and ATM levels via active networking) to optimise network resources and to guarantee QoS requirements. The use of active networks can be either as a facilitator, i.e. an implicit agent when TCP runs over ATM or a new service with capabilities designed for TCP. References [1] Proceedings of the Third International Conference on Autonomous Agents (Agents'99), Seattle, Washington, May, 1999, ACM, http:// www.cs.washington.edu/research/agents99/. [2] M. Allman, V. Paxson, W. Stevens, TCP Congestion Control, RFC 2581, April 1999.
K. Djemame, M. Kara / Computer Communications 24 (2001) 850±859 [3] W. Almesberger, ATM on Linux, Institute for Computer Communications and Applications, Ecole Polytechnique FeÂdeÂrale de Lausanne, Switzerland, November, 1999, http://icawww1.ep¯.ch/linux-atm. [4] ATM Forum Technical Committee, Traf®c Management Speci®cation, Version 4.1, AF-TM-0121.000, March 1999. [5] M. Baumann, YATS User's Manual for Version 0.3, Communications Laboratory, Dresden University of Technology, Germany, January 1998, http://www.ifn.et.tu-dresden.de/tk/yats/yats.html. [6] L.S. Brakmo, S.W. O'Malley, L.L. Peterson, TCP Vegas: new techniques for congestion detection and avoidance, Proceedings of SIGCOMM'94, August 1994, ftp://ftp.cs.arizona.edu/xkernel/ papers/vegas.ps. [7] K. Djemame, M. Kara, Proposals for a coherent approach to cooperation between TCP and ATM congestion control algorithms, in: J.T. Bradley, N.J. Davies (Eds.), Proceedings of the 15th Annual UK Performance Engineering Workshop (UKPEW'99), Bristol, UK, July 1999, pp. 273±284, http://www.scs.leeds.ac.uk/atm-mm/papers/ ukpew99.ps.gz. [8] T. Faber, ACC: using active networking to enhance feedback congestion control mechanisms, IEEE Network 12 (3) (1998) 61±65 (http:// www.isi.edu/~faber/pubs/html/network98/). [9] S. Floyd, TCP and explicit congestion noti®cation, ACM Computer Communication Review 24 (5) (1994) 8±23. [10] V. Jacobson, Congestion avoidance and control, ACM Computer Communication Review 18 (1988) 314±329 (Proceedings of Sigcomm'88 Symposium, Stanford, CA). [11] L. Kalampoukas, A. Varma, K.K. Ramakrishnan, Explicit window adaptation: a method to enhance TCP performance, Proceedings of INFOCOM'98, San Francisco, CA, April 1998. [12] S. Kalyanaraman, R. Jain, S. Fahmy, R. Goyal, Performance and buffering requirements of internet protocols over ATM ABR and UBR services, IEEE Communications Magazine 36 (6) (1998) 152±157 (http://www.cis.ohio-state.edu/~jain/papers/ieee-mag.htm).
859
[13] S. Kalyanaraman, R. Jain, S. Fahmy, R. Goyal, B. Vandalore, The ERICA switch algorithm for ABR traf®c management in ATM networks, IEEE/ACM Transactions on Networking February (2000) 87±98, (http://www.cis.ohio-state.edu/~jain/papers/ erica.htm). [14] M. Kara, M. Rahin, Towards a framework for performance evaluation of TCP behaviour over ATM networks, Proceedings of the 13th IFIP International Conference on Computer Communication, ICCC'97, November 1997, pp. 49±60, ftp://agora.leeds.ac.uk/scs/doc/reports/ 1997/97_42.ps.Z. [15] A. Koike, TCP Flow Control with ACR Information, ATM Forum/970998, December 1997. [16] M. Laubach, J. Halpern, Classical IP and ARP over ATM, RFC 2225, April 1998. [17] M. Mathis, J. Madhavi, S. Floyd, A. Romanow, TCP Selective Acknowledgement Options, RFC 2018, October 1996, ftp:// ftp.ee.lbl.gov/papers/rfc2018.ps. [18] P. NarvaÂez, K.-Y. Siu, Acknowledgement bucket scheme for regulating TCP ¯ow over ATM, Computer Networks and ISDN 30 (19) (1998) 1775±1791, (http://perth.mit.edu/~pnarvaez/mypapers/ paper3 _tech.ps.gz). [19] K. Ramakrishnan, S. Floyd, A Proposal to Add Explicit Congestion Noti®cation (ECN) to IP, RFC 2481, January 1999. [20] A. Romanow, S. Floyd, Dynamics of TCP traf®c over ATM networks, IEEE Journal on Selected Areas in Communications 13 (4) (1995) 633±641. [21] R. Satyavolu, K. Duvedi, S. Kalyanaraman, Explicit Rate Control of TCP Applications, ATM_Forum/98-0152R1, February 1998, http:// www.ecse.rpi.edu/homepages/shivkuma. [22] D. Sisalem, H. Schulzrinne, End-to-end rate control in ABR, Proceedings of the First Workshop on ATM Traf®c Management (WATM'95), Paris, France, December 1995, 281±287, IFIP WG 6.2, http://www.cs.columbia.edu/~hgs/papers/sisa9512_end.ps.gz.