Enabling High Performance Data Transfers

Posted by root in the Network section on Wed 21 Dec 2005 at 03:36

This (DRAFT) page is currently under active revision. Please send any suggestions, additions or corrections to us at nettune@ncne.org so we can keep the information here as up-to-date as possible.

• Introduction
• Tutorial
  • Bandwidth*Delay Products (BDP)
  • Buffers
  • Computing the BDP
• High Performance Networking Options
  • TCP Selective Acknowledgments (SACK, RFC2018)
  • Large Windows (RFC1323)
  • Maximum Buffer Sizes on the host
  • Application Buffers
  • Path MTU
• Using Web Based Network Diagnostic Servers
  • SYN Test Server
  • ORNL Bandwidth Tester
  • ANL Network Diagnostic Tool
• Support for Various Operating Systems
• Detailed Procedures for System tuning and Raising network limits for:
  • FreeBSD
  • on Linux
  • on Macintosh OS X
  • under NetBSD
  • on Microsoft Windows 2000 and Windows XP systems
  • or on other historical Operating Systems.

Introduction

Today, the majority of university users have physical network connections that are at least 100 megabits per second all the way through the Internet to every important data center in the world (as well as to every other university user). For many users, that connection might be 1 gigabit per second or faster. In some countries (e.g. Korea and Japan) the same statement applies to every home connection as well: 100 Mb/s from home to all important web servers, data centers and to each other.

To put these data rates into perspective, consider this: 100 Mb/s is more than 10 megabytes in one second, or 600 megabytes (an entire CD-R image) in one minute. Clearly very few people see these data rates. However, some experts can get very high data rates (for example see the Land Speed Records). Why? The biggest strength of the Internet is the way in which the TCP/IP "hourglass" hides the details of the network from the application and vice versa. An unfortunate but direct consequence of the hourglass is that it also hides all flaws everywhere. Network performance debugging (often euphemistically called "TCP tuning") is extremely difficult because nearly all flaws have exactly same symptom: reduced performance. For example insufficient TCP buffer space is indistinguishable from excess packet loss (silently repaired by TCP retransmissions) because both flaws just slow the application, without any specific identifying symptoms.

Flaws fall into three broad areas: the applications themselves, the computer system (including the operating system and TCP tuning) and the network path. Each of these areas requires a very different approach to performance debugging. This page is focused on helping users and system administrators optimize the TCP/IP on their computer systems.

• Applications sometimes perform poorly on long paths (even when the network is perfect) because they are not designed to fully overlap the speed of light delay to deliver the data with the processing at the end systems. It is quite difficult to write complicated applications that do this overlap properly, but it must be done for an application to perform well on a long network path. We are developing tools and documentation to help application programmers test and debug applications under these conditions. (Watch this space!)

  For example secure shell and secure copy (ssh and scp) implement internal flow control using an application level mechanism that severely limits the amount of data in the network, greatly reducing the performance all but the shortest paths. PSC is now supporting a patch to ssh and scp that updates the application flow control window from the kernel buffer size. With this patch, the TCP tuning directions on this page can alleviate the dominant bottlenecks in scp. In most environments scp will run at full link rate or the CPU limit for the chosen encryption.

• Network paths can be very hard to debug because TCP's ability to compensate for flaws is inversely proportional to the round trip time (RTT). So for example a flaw that will cause an application to take an extra second on a 1 millisecond path will generally cause the same application to take an extra 10 seconds on a 10 millisecond path. This "symptom scaling" effect arises because TCP's ability to compensate for flaws is metered in round trips: if a given flaw is compensated in 50 round trips (typical for losses on a medium speed link), then a single loss affects a 1 ms path for only 50 ms, and a 10 ms path for 500 ms. Symptom scaling makes diagnosis particularly difficult, because flaws that are complete show stoppers on long paths are often be undetectable on short paths.

  We have a new tool, pathdiag, which compensates for symptom scaling so that it can detect previously undetectable flaws in a local network. The basic approach is to measure the properties of a short section of the path, and extrapolate the results as though the path was extended to the full RTT with an ideal network. It then uses a TCP performance model to predict if the resulting path would perform well enough to meet the application requirements.

  Pathdiag also does a complete check of the TCP options described on this page since it requires a well-tuned TCP implementation at the far end of the path. If it is available to you it is both the easiest to use and the most accurate test available. Since pathdiag requires a very short RTT the closest pathdiag server might not be close enough to you, in which case the SYN testserver described below can also verify most TCP configuration options with one click.

The objectives of this page are to summarize all of the end system network tuning issues, provide easy configuration checks for non-experts, and maintain a repository of operating system specific advice and information about getting the best possible network performance on these platforms.

In the Tutorial we will briefly explain the issues and define some terms. Under High Performance Networking Options we describe each of the optional TCP features may have to configured without addressing the details of any specific operating system. The section, "Detailed Procedures", provides step-by-step directions on making the necessary changes for several operating systems.

Note that today most TCP implementations are pretty good. The primary flaws are default configurations that are ideal for Internet back roads: many millions of relatively low speed home users.

Acknowledgments

Jamshid Mahdavi maintained this page for many years, both at PSC and later, remotely from Novell. We are greatly indebted to his vision and persistence in establishing this resource.

Thanks Jamshid!.

Many, many people have helped us compile this information. We want to thank everyone who sent us updates, additions and corrections. We have decided to include attributions for all future contributors. (Sorry not to be able to give full credit where credit is due for past contributors.)

This material has been maintained as a sideline of many different projects, nearly all of which have been funded by the National Science Foundation. It was started under NSF-9415552, but also supported under Web100 (NSF-0083285) and is currently supported under the NPAD project (ANI-0334061).

Tutorial

The dominant protocol used on the Internet today is TCP, a "reliable" "window-based" protocol. Under ideal conditions, best possible network performance is achieved when the network pipe between the sender and the receiver is kept full of data.

Bandwidth*Delay Products (BDP)

The amount of data that can be in transit in the network, termed "Bandwidth-Delay-Product," or BDP for short, is simply the product of the bottleneck link bandwidth and the Round Trip Time (RTT). BDP is a simple but important concept in a window based protocol such as TCP. Some of the issues discussed below arise because of the fact that the BDP of today's networks has increased way beyond what it was when the TCP/IP protocols were initially designed. In order to accommodate the large increases in BDP, some high performance extensions have been proposed and implemented in the TCP protocol. But these high performance options are sometimes not enabled by default and will have to be explicitly turned on by the system administrators.

Buffers

In a "reliable" protocol such as TCP, the importance of BDP described above is that this is the amount of buffering that will be required in the end hosts (sender and receiver). The largest buffer the original TCP (without the high performance options) supports is limited to 64K Bytes. If the BDP is small either because the link is slow or because the RTT is small (in a LAN, for example), the default configuration is usually adequate. But for a paths that have a large BDP, and hence require large buffers, it is necessary to have the high performance options discussed in the next section be enabled.

Computing the BDP

To compute the BDP, we need to know the speed of the slowest link in the path and the Round Trip Time (RTT).

The peak bandwidth of a link is typically expressed in Mbit/s (or more recently in Gbit/s). The round-trip delay (RTT) for wide area links is typically between 10 msec and 100 msec, which can be measured with ping or traceroute

As an example, for two hosts with GigE cards, communicating across a coast-to-coast link over an Abilene connection (assuming a 2.4 Gbps OC-48 link), the bottleneck link will be the GigE card itself. The actual round trip time (RTT) can be measured using ping, but we will use 70 msec in this example.

Knowing the bottleneck link speed and the RTT, the BDP can be calculated as follows:

Based on these calculations, it is easy to see why the typical default buffer size of 64 KBytes would be way inadequate for this connection.

The next section presents a brief overview of the high performance options. Specific details on how to enable these options in various operating systems is provided in a later section.

High Performance Networking Options

The options below are presented in the order that they should be checked and adjusted.

1. Maximum TCP Buffer (Memory) space: Nearly all operating systems have some mechanism to limit the amount of system memory that can be used by any one TCP connection. On some systems, each connection is subject to a memory limit that is applied to the total memory used for input data, output data and control structures. On other systems, there are separate limits for input and output buffer space for each connection.

   Today all systems are shipped with Maximum Buffer Space limits that are far too small for nearly all of today's Internet. Furthermore the procedures for adjusting the memory limits are different on every operating system. You must follow the appropriate detailed procedures below, which generally require privileges on multi-user systems.

2. Socket Buffer Sizes: Most operating systems also support separate per connection send and receive buffer limits that can be adjusted by the user, application or other mechanism as long as they stay within the maximum memory limits above. These buffer sizes correspond to the SO_SNDBUF and SO_RCVBUF options of the BSD setsockopt() call. The socket buffers must be large enough to hold a full BDP of TCP data plus some operating system specific overhead. They also determine the Receiver Window (rwnd), used to implement flow control between the two ends of the TCP connection. There are several methods that can be used to adjust socket buffer sizes:

   • TCP Autotuning automatically adjusts socket buffer sizes as needed to optimally balance TCP performance and memory usage. Autotuning is based on an experimental implementation for NetBSD by Jeff Semke, and further developed by Wu Feng's DRS and the Web100 Project. Autotuning is now enabled by default in current Linux releases (after 2.6.6 and 2.4.16). It has also been announced for Windows Vista and Longhorn. In the future, we hope to see all TCP implementations support autotuning, making this website largely obsolete.

   • The default socket buffer sizes can generally be set with global controls. These default sizes are used for all socket buffer sizes that are not set in some other way. For single user systems, manually adjusting the default buffer sizes is the easiest way to tune arbitrary applications. Again, there is not standard method to do this, you must refer to the detailed procedures below.

     Since over buffering can cause some applications to behave poorly (typically causing sluggish interactive response) and risk running the system out of memory, large default socket buffers have to be considered carefully on multi-user systems. (But bear in mind that current shipped defaults are unreasonably small for nearly all environments).

   • For customs applications, the programmer can choose the socket buffer sizes using a setsockopt() system call. A Detailed Users Guide by Von Welch at NCSA describing how to set socket buffer sizes within network applications.

   • Some common applications include switches or commands to permit the user to manually set socket buffer sizes. The most common examples include iperf (a network diagnostic), many ftp variants (including gridftp) and other bulk data copy tools. Check the documentation on your system to see what is available.

     This approach forces the user to manually compute the BDP for the path and supply the proper command or option to the application.

   • There has been some work on autotuning within the applications themselves. This approach is easier to deploy than kernel modifications and frees the user from having to compute the BDP, but the application is hampered by limited access to the resources it needs to monitor and tune.

     • NLANR/DAST has an FTP client which automatically sets the socket buffer size to the measure bandwidth*delay product for the path. This client can be found at http://dast.nlanr.net/Projects/Autobuf/
     • NLANR/NCNE maintains a tool repository which includes application enhancements for several versions of FTP and rsh. Also included on this site is the nettune library for performing such enhancements yourself.

3. TCP Large Window Extensions (RFC1323): These typically enable TCP to negotiate several optional protocol features needed to support large BDP paths.

   The most important RFC1323 feature is the window scale option (WSCALE), which provides a scale factor which is required to permit TCP to support window sizes larger than 64k Bytes. Most systems automatically request WSCALE under some conditions, such as when the socket buffer is larger than 64k Bytes or when the other end of the TCP connection requests it first. If either end fails to request WSCALE at the start of a connection, it is disabled, and cannot be renegotiated later during the same connection. Note that under this logic (which is common to most platforms), a client application wishing to send data at high rates must set its own receive window to something larger than 64k Bytes to ensure that the server properly negotiates WSCALE.

   A few systems require a system administrator to explicitly enable RFC1323 extensions. If system cannot (or does not) negotiate WSCALE, it cannot support TCP window sizes larger than 64k Bytes.

   Another RFC1323 feature is the TCP Timestamp option which provides better measurement of the Round Trip Time and protects TCP from data corruption that might occurred if packets are delivered so late that the sequence numbers wrap before they are delivered. The wrapped sequence numbers do not pose a serious risk below 100 Mb/s, but the risk becomes progressively larger as the data rates get higher.

   However, due to the improved RTT estimation many systems use timestamps even a low rates. Check your system in the TCP feature table.

4. TCP Selective Acknowledgments Option (SACK, RFC2018) allow a TCP receiver inform the sender exactly which data is missing and needs to be retransmitted. Without SACK TCP has to estimate which data is missing, which works just fine if all losses are isolated (only one loss in any given round trip). Without SACK, TCP often takes a very long time to recover following a cluster of losses, which is the normal case for a large BDP path with even minor congestion. SACK is now supported by most operating systems, but it may have to be explicitly turned on by the system administrator.

   If you have a system that does not support SACK you can often raise TCP performance by slightly starving it for socket buffer space, The buffer starvation prevents TCP from being able to drive the path into congestion, and minimize the chances of causing clustered losses.

   Additional information on commercial and experimental implementations of SACK is available at http://www.psc.edu/networking/projects/sack/.

5. Path MTU The host system must use the largest possible MTU for the path. This may require enabling Path MTU Discovery (RFC1191). Since RFC1191 is flawed it is sometimes not enabled by default and may need to be explicitly enabled by the system administrator. Watch for PMTU blackholes. If Path MTU Discovery is unavailable or undesired, it is sometimes possible to force the system to use large packets, but this may have undesirable side effects.

   The Path MTU Discovery server (described in a little more detail in the next section) may be useful in checking out the largest MTU supported by some paths:

Using Web Based Network Diagnostic Servers

There are some diagnostic servers that are useful for troubleshooting some of the networking problems. We point out a couple of them here.

• SYN Test Server: Often users would like a quick way to check if the high performance networking options discussed above (SACK etc.) are turned on. This is a Web100 based server that allows any client on the Internet to do this quickly and easily. It can be accessed either from a text window or from a web browser. In text mode, one can simply type:

  telnet syntest.psc.edu 7960
  

  Alternatively, one can point a web browser on the host to the following URL: http://syntest.psc.edu:7961

• ORNL TCP Bandwidth Tester: Sometimes users would like to run a short network test to a remote machine to see how well connected they are to the outside world. This is a web based java server that does a couple of short network tests and measures bandwidth between the host and the server. Simply visit the following URL: http://www.epm.ornl.gov/~dunigan/java/misc/tcpbw.html

• ANL Network Configuration Tester: This is a web based java server that does a couple of short network tests and provides diagnostic information about the host. This server is particularly helpful in troubleshooting duplex mismatch problems.
  Simply visit the following URL: http://miranda.ctd.anl.gov:7123/
  Detailed explanation about capabilities of tool can be found at the web site.

• Path MTU Discovery Server: This is a web based server you can use to determine the largest MTU supported between the server at Pittsburgh Supercomputing Center and the host you are interested in:

  http://www.ncne.org/jumbogram/mtu_discovery.php

  This tool shows the path and the largest MTU supported on that path from a server at the Pittsburgh Supercomputing Center to the host in question.

TCP Features Supported by Various Operating Systems

Information for other older operating systems has been moved to the historical operating system page.

Detailed procedures for system tuning under various operating systems

See the specific instructions for each system:

• FreeBSD
• on Linux
• on Macintosh OS X
• under NetBSD
• on Microsoft Windows 2000 and Windows XP systems
• or on other historical Operating Systems.

Procedure for raising network limits under FreeBSD 2.1.5

You can't modify the maximum socket buffer size in FreeBSD 2.1.0-RELEASE, but in 2.2-CURRENT you can use

	sysctl -w kern.maxsockbuf=524288

to make it 512kB (for example). You can also set the TCP and UDP default buffer sizes using the variables

	net.inet.tcp.sendspace
	net.inet.tcp.recvspace
	net.inet.udp.recvspace

MTU discovery is on by default in FreeBSD past 2.1.0-RELEASE. If you wish to disable MTU discovery, the only way that we know is to lock an interface's MTU, which disables MTU discovery on that interface.

Tuning TCP for Linux 2.4 and 2.6

The maximum buffer sizes for all sockets can be set with /proc variables:

	/proc/sys/net/core/rmem_max       - maximum receive window
	/proc/sys/net/core/wmem_max       - maximum send window

These determine the maximum acceptable values for SO_SNDBUF and SO_RCVBUF (arguments to setsockopt() system call). The kernel sets the actual memory limit to twice the requested value (effectively doubling rmem_max and wmem_max) to provide for sufficient memory overhead.

The per connections memory space defaults are set with two 3 element arrays:

	/proc/sys/net/ipv4/tcp_rmem       - memory reserved for TCP rcv buffers
	/proc/sys/net/ipv4/tcp_wmem       - memory reserved for TCP snd buffers

These are arrays of three values: minimum, default and maximum that are used to bound autotuning and balance memory usage while under global memory stress.

The following values would be reasonable for path with a 4MB BDP (You must be root):

       	echo 2500000 > /proc/sys/net/core/wmem_max 
       	echo 2500000 > /proc/sys/net/core/rmem_max 
       	echo "4096 5000000 5000000" > /proc/sys/net/ipv4/tcp_rmem
       	echo "4096 65536 5000000" > /proc/sys/net/ipv4/tcp_wmem

All Linux 2.4 and 2.6 versions include sender side autotuning, so the actual sending socket buffer (wmem value) will be dynamically updated for each connection. You can check to see if receiver side autotuning is present an enabled by looking at the file:

	/proc/sys/net/ipv4/tcp_moderate_rcvbuf

If it is present and enabled (value 1) the receiver socket buffer size (rmem value) will be dynamically updated for each connection. If it is not present you may want to get a newer kernel. Generally autotuning should not be disabled unless there is a specific need, e.g. comparison studies of TCP performance. If you do not have autotuning (Linux 2.4 before 2.4.27 or 2.6 before 2.6.7) you may want to set the default tcp_rmem value (the middle value) to a more accurate estimate of the actual path BDP, to minimize possible interactions with other applications.

Do not adjust tcp_mem unless you know exactly what you are doing. This array determines how the system balances the total network memory usage against other memory usage, such as disk buffers. It is initialized at boot time to appropriate fractions of total system memory.

You do not need to adjust rmem_default or wmem_default (at least not for TCP tuning). These are the default buffer sizes for non-TCP sockets (e.g. unix domain sockets, UDP, etc).

All standard advanced TCP features are on by default. You can check them by cat'ing the following /proc files:

	/proc/sys/net/ipv4/tcp_timestamps
	/proc/sys/net/ipv4/tcp_window_scaling
	/proc/sys/net/ipv4/tcp_sack 

Linux supports both /proc and sysctl (using alternate forms of the variable names - net.core.rmem_max) for inspecting and adjusting network tuning parameters. The following is a useful shortcut for inspecting all tcp parameters:

sysctl -a | fgrep tcp

For additional information on kernel variables, look at the documentation included with your kernel source, typically in some location such as /usr/src/linux-/Documentation/networking/ip-sysctl.txt. There is a very good (but slightly out of date) tutorial on network sysctl's at http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html.

If you would like to have these changes to be preserved across reboots, you can add the tuning commands to your /etc/rc.d/rc.local file.

Autotuning was prototyped under the Web100 project. Web100 also provides complete TCP instrumentation and some additional features to improve performance on paths with very large BDP.

Contributors: John Heffner

Tuning TCP for Mac OS X

Mac OS X has a single sysctl parameter, kern.ipc.maxsockbuf, to set the maximum combined buffer size for both sides of a TCP (or other) socket. In general, it can be set to at least twice the BDP. E.g:

sysctl -w kern.ipc.maxsockbuf=8000000

The default send and receive buffer sizes can be set using the following sysctl variables:

sysctl -w net.inet.tcp.sendspace=4000000
sysctl -w net.inet.tcp.recvspace=4000000

If you would like these changes to be preserved across reboots you can edit /etc/sysctl.conf.

RFC1323 features are supported and on by default. SACK is not available at this time

Although we have never tested it, there is a commercial product to tune TCP on Macintoshes. The URL is http://www.sustworks.com/products/prod_ottuner.html. I don't endorse the product they are selling (since I've never tried it). However, it is available for a free trial, and they appear to do an excellent job of describing perf-tune issues for Macs.

Procedure for raising network limits under NetBSD

      sysctl net.inet.tcp.rfc1323

The maximum socket buffer size can be modified by changing SB_MAX in /usr/src/sys/sys/socketvar.h.

The default socket buffer sizes can be modified by changing TCP_SENDSPACE and TCP_RECVSPACE in /usr/src/sys/netinet/tcp_usrreq.c.

It may also be necessary to increase the number of mbufs, NMBCLUSTERS in /usr/src/sys/arch/*/include/param.h.

Update: It is also possible to set these parameters in the kernel configuration file.

options		SB_MAX=1048576		# maximum socket buffer size
options		TCP_SENDSPACE=65536	# default send socket buffer size
options		TCP_RECVSPACE=65536	# default recv socket buffer size
options		NMBCLUSTERS=1024	# maximum number of mbuf clusters

Procedure for raising network limits under Microsoft Windows 2000 and Windows XP

New: The following URL: http://rdweb.cns.vt.edu/public/notes/win2k-tcpip.htm appears to be a pretty good summary of the procedure for TCP tuning under Windows 2000. It also has the URL for the Windows 2000 TCP tuning document from Microsoft.

We are not sure if it still necessary to set DefaultReceiveWindow even after setting the parameters indicated in the URL above.

If your machine does a lot of large outbound transfers, it will be necessary to set DefaultSendWindow in addition to the suggestions mentioned above.

Procedure for raising network limits under Microsoft Windows 98

New: Some folks at NLANR/MOAT in SDSC have written a tool to do guide you through some of this stuff. It can be found at http://moat.nlanr.net/Software/TCPtune/.

Even newer: I've updated some sending window information that was inaccurate. See below.

Several folks have recently helped me to figure out how to accomplish the necessary tuning under Windows98, and the features do appear to exist and work. Thanks to everyone for the assistance! The new description below should be useful to even the complete Windows novice (such as me :-).

Windows98 includes implementation of RFC1323 and RFC2018. Both are on by default. (However, with a default buffer size of only about 8kB, window scaling doesn't do much).

Windows stores the tuning parameters in the Windows Registry. In the registry are settings to toggle on/off Large Windows, Timestamps, and SACK. In addition, default socket buffer sizes can be specified in the registry.

In order to modify registry variables, do the following steps:

1. Click on Start -> Run and then type in "regedit". This will fire up the Registry Editor.
2. In the Registry Editor, double click on the appropriate folders to walk the tree to the parameter you wish to modify. For the parameters below, this means clicking on HKEY_LOCAL_MACHINE -> System -> CurrentControlSet -> Services -> VxD -> MSTCP.
3. Once there, you should see a list of parameters in the right half of your screen, and MSTCP should be highlighted in the left half. The parameters you wish to modify will probably not appear in the right half of your screen; this is OK.
4. In the menu bar, Click on "Edit -> New -> String Value". It is important to create the parameter with the correct type. All of the parameters listed below are strings.
5. A box will appear with "New Value #1"; change the name to the name listed below, exactly as shown. Hit return.
6. On the menu, click on "Edit -> Modify" (your new entry should still be selected). Then type in the value you wish to assign to the parameter.
7. Exit the registry editor, and reboot windows. (The rebooting is important, *sigh*.)
8. When your system comes back up, you should have access to the features you have just turned on. The only real way to verify this is through packet traces (or by noticing a significant performance improvement).

TCP/IP Stack Variables

Support for TCP Large Windows (TCPLW)

Win98 TCP/IP supports TCP large windows as documented in RFC 1323. TCP large windows can be used for networks that have large bandwidth delay products such as high-speed trans-continental connections or satellite links. Large windows support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP

The registry key Tcp1323Opts is a string value type. The values for Tcp1323Opt are

The default value for Tcp1323Opts is 3: Window Scaling and Time stamp options. Large window support is enabled if an application requests a Winsock socket to use buffer sizes greater than 64K. The current default value for TCP receive window size in Memphis TCP is 8196 bytes. In previous implementations the TCP window size was limited to 64K, this limit is raised to 2**30 through the use of TCP large window support.

Support for Selective Acknowledgements (SACK)

Win98 TCP supports Selective Acknowledgements as documented in RFC 2018. Selective acknowledgements allow TCP to recover from IP packet loss without resending packets that were already received by the receiver. Selective Acknowledgements is most useful when employed with TCP large windows. SACK support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP

The registry key SackOpts is a string value type. The values for SackOpts are

Support for Fast Retransmission and Fast Recovery

Win98 TCP/IP supports Fast Retransmission and Fast Recovery of TCP connections that are encountering IP packet loss in the network. These mechanisms allow a TCP sender to quickly infer a single packet loss by reception of duplicate acknowledgements for a previously sent and acknowledged TCP/IP packet. This mechanism is useful when the network is intermittently congested. The reception of 3 (default value) successive duplicate acknowledgements indicates to the TCP sender that it can resend the last unacknowledged TCP/IP packet (fast retransmit) and not go into TCP slow start due to a single packet loss (fast recovery). Fast Retransmission and Recovery support is controlled by a registry key value in:

HKLM\system\currentcontrolset\services\VXD\MSTCP\Parameters

The registry key MaxDupAcks is DWORD taking integer values from 2 to N. If MaxDupAcks is not defined, the default value is 3.

Update: If you wish to set the default receiver window for applications, you should set the following key:

DefaultRcvWindow

HKLM\system\currentcontrolset\services\VXD\MSTCP

DefaultRcvWindow is a string type and the value describes the default receive windowsize for the TCP stack. Otherwise the windowsize has to be programmed in apps with setsockopt.

For a long time, I had the following sentence on this page:

• I presume that there is also a DefaultSndWindow what you would want to use on servers sending data to get higher performance. I have not yet verified this, however.

It turns out that there is not in fact such a variable. My limited experience has shown that, in some cases, it is possible to see very large send windows from Microsoft boxes. However, recent reports on the tcpsat mailing list have also stated that a number of applications under Windows severely limit the sending window. These applications appear to include FTP and possibly also the CIFS protocol which is used for file sharing. With these applications, it appears to be impossible to exceed the performance limit dictated by this sending window.

If anyone has any further information on these specific applications under Windows, I would be happy to include it here.