Why 100Mbps Does Not Mean 100Mbps Transfer Rates
You will not always see 100Mbps upload/download speeds even with a 100Mbps port. Much of the slowdown occurs because as packet travel distance increases, so does latency, which has a large detrimental effect on large file transfers. For smaller files, like those associated with not-too-graphical web pages, this has less impact. Without getting too technical, this is because file transfer protocols that use TCP require that the recipient respond with confirmations of data received, and this is one reason that file transfers over longer distances are slower, in direct proportion with the increase in response times.
See http://www.internetworkexpert.org/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ for a more in-depth discussion on this.
Most download accelerators are able to increase transfer rates by simply employing multiple TCP pipes that dump into the same file. This doesn’t solve the TCP window size problem, but takes advantage of what the uplink is capable of handling. Most modern browsers do this automatically, so download accelerators are really not a necessity any more.
You may wish to optimize your per-TCP connection transfer rates though. To do so, determine your optimal TCP window size based on the expected latency of your most bandwidth intense client-base (see the calculator at the above link). Then, based on that, adjust your TCP/IP stack to adjust below:
To tweak Windows 2008 TCP Window Scaling, please refer to the following:
http://www.minasi.com/newsletters/nws0802.htm
http://www.msfn.org/board/heres-why-tcpwindowsize-does-not-work-vista-t87969.html
Note that Windows 2008 doesn’t allow you to tweak settings like 2003 did. You can make the system adjust it “more aggressively,” but you can’t hard code numbers in.
To tweak Windows 2003 TCP Window Scaling, please refer to the following:
http://articles.techrepublic.com.com/5100-10878_11-5034413.html
You may wish to also try: http://www.speedguide.net/tcpoptimizer.php
To tweak Linux TCP Window Scaling, please refer to the following:
http://www.speedguide.net/read_articles.php?id=121
Note that many other factors come into play for bandwidth calculation. In a hosting environment, your server must compete with other servers in the data center to reach the core routers and from there, must concentrate in various nodes and exchanges to reach a packet’s destination. Along the way, routers must prioritize and queue packets for transmission. We can check the health of this process by performing a traceroute between “slow links.” Network congestion at any one of these nodes can reduce overall transfer rate. On either one of the endpoints, disk I/O, or other system stress may be a bottleneck.
All in all, an 100Mbps, or even an 1000Mbps uplink will not provide transfer rates greater than what the network fabric in between the source and destination can handle, and not greater than what the server / client can negotiate within the TCP pipe.
#18 Feb 2010 – Edited for spelling/grammar.
Slowloris and You
UPDATE: 20090826 – Corrected typo in “Slowloris and You.” It used to say “Slowlaris and You.” I keep getting slowloris confused with my nickname for “Solaris.” =D
Back in July, http://ha.ckers.org/slowloris/ published an exploit against Apache and other web servers (go to the link for further) that takes advantage of multi-threaded applications. It works by tying up web server threads with partial HTTP requests, then sends TCP handshakes to keep the socket open. In general, multi-threaded web servers such as httpd, apache, and apache2 are vulnerable. IIS and most proxies are not vulnerable
CERT suggested using iptables to rate limit incoming port 80 requests. In general, this should be fine for many applications, though CERT has warned that some large clients behind NAT’s may be affected and thus the hitcount/time ratio should be adjusted according to your needs.
http://www.funtoo.org/en/security/slowloris/ offers tips on mitigating this attack by enabling delayed binding on hardware load balancers.
In short, it appears as though the consensus mitigation method involves connection restrictions in the form of iptables or apache modules (most are of limited value, frankly), or shielding the web servers behind load balancers (such as HA-Proxy).
Conficker Update Part 3
According to http://forum.drweb.com/index.php?showtopic=277240 , Win32.HLLW.Shadow.based is a a variant of Conficker/downadup.
Symptom: Every available port from 1024-5000 is used to connect to various servers on destination port 445. Basically, the worm opens these connections to download and wait for malicious binaries.
The removal tools at http://www.bdtools.net/ does not detect this variant, and you have to use Dr.Web’s Cureit to detect and remove it. According to them, the recommended procedure is to install the following hotfixes:
* MS08-067
(http://www.microsoft.com/technet/security/bulletin/ms08-067.mspx);
* MS08-068
(http://www.microsoft.com/technet/security/bulletin/ms08-068.mspx);
* MS09-001
(http://www.microsoft.com/technet/security/bulletin/ms09-001.mspx).
And then run Cureit, a fully functional shareware app.
In case you’re reading this from an infected server, I’ve downloaded and included some of these files here (because if you’re infected, you won’t be able to access certain sites, drweb.com being one).
Dynamic iptables – “Flexible (and fun)”
Have you ever said to yourself that there should be a tool to do x, start building a tool to do it, then about halfway through your little project, somebody glances over your shoulder and says to you “hey, I use a tool like that, it’s called y, you should check it out,” so you do, and that tool is far more comprehensive and well built than the one you were working on?
Well this isn’t one of those times, because this tool hit me from left field while I was researching ways to mitigate a DDoS attack. Though there are many, many ways to do it, if all you have is a Linux box facing the world with nothing to hide its private parts except iptables, then this “flexible (and fun)” toolset is another weapon you can deploy when you get that 2:30AM call saying “our website’s down and I think it’s being DDoS’d.”
The tool is a simple set of scripts that make adding and removing specific IP’s quick and simple. The main site of the author is at http://www.ibm.com/developerworks/library/l-fw/, or is available (hosted locally) here.
Once installed, you can simply ban/unban an IP by typing ipdrop {IP ADDRESS} {on|off}
While perusing this thread at webhostingtalk.com, member dynamicnet mentioned grep-ing for ridiculous levels of SYN_RECV ‘d connections (this is indicative of a TCP SYN Flood attack) and generating ipdrop commands for quick banning of a SYN Flood-ing IP’s. Though you may accidentally drop one or two legitimate IP’s (have a rule already in place so you don’t ban yourself out of a remote box), you’ll likely get the bulk of the attacking IP’s.
Use netstat -n -p|grep SYN_REC | wc -l to count how many SYN_RECV connections you have.
Use netstat -n -p | grep SYN_REC | awk '{print $5}' | sort -u | awk -F: '{print "ipdrop "$1 " on"}' to generate code to ban IP’s in SYN_RECV status.
Use cat /root/.dynfw-ipdrop |awk -F: '{print "ipdrop "$1" off"}' to generates code to “undrop” those IP’s.
Conficker Update Part Deuce
UPDATE 11 Jun 2009: Locally hosted bdtools removal tools (availabe at downadup.org for single computers and network. Cheers =D
Our favorite worm got an update 8 April according to Network World. Read more here…
And of course, at downadup.org.
