dropbox.com
I wanted to bring up Dropbox.com to your attention. I wanted to share what I thought was a fantastic company that’s basically taken the concept of JungleDisk and moved it one step further. How? Instead of really cheap (Jungledisk), dropbox is free for the first 2 gigs. This, combined with the below usability features, results in rapid growth in user adoption. As of January, the company has over 4 million users. That’s over 8 Petabytes of potential disk space usage if every user was a free 2GB plan.
All of the public infrastructure is hosted at AWS, and can thus scale with the userbase. As Amazon charges anywhere from $0.055 to $0.150 per GB (pricing structure), each free user consumes from $0.11 to $0.30 per month. Even with 6.25% utilization of 8 Petabytes, Dropbox.com pays the $0.105 per GB rate, Given their next paid upgrade is for 50 GB and costs $9.99/month, one paid customer can support the storage fees for up to ($9.99 − 50 clients × $0.105) ÷ $0.21 = 22 clients covered in the cost of a single paid user, at full utilization of each user. However, most users won’t be using their full utilization (think Google mail), and their business model becomes more lucrative.
Starting with dropbox is simple. Download the client, register your account, and a folder is created where you can drag and drop your data. All data in this folder is replicated to dropbox servers, and to all dropbox client nodes that you link to your account.
The application ships with an intuitive user interface (no S3 accounts to configure, as this is done for you), and a rich feature set that includes automatic versioning, automatic syncing, cross-platform compatibility, intuitive web publishing, and a simple pricing model.
The one feature that really stood out however, was how they garner feedback for this actively developed product. Users can navigate to https://www.dropbox.com/votebox, where they can submit suggestions that get voted on and commented upon. This popularity contest likely helps the company focus on what projects need developing next.
All in all, this is an exciting company to be a customer of, and I would recommend y’all take a look at it too, as this appears to be what proper execution of a good idea looks like.
Kudos dropbox.com on a product well done.
Why 100Mbps Does Not Mean 100Mbps Transfer Rates
You will not always see 100Mbps upload/download speeds even with a 100Mbps port. Much of the slowdown occurs because as packet travel distance increases, so does latency, which has a large detrimental effect on large file transfers. For smaller files, like those associated with not-too-graphical web pages, this has less impact. Without getting too technical, this is because file transfer protocols that use TCP require that the recipient respond with confirmations of data received, and this is one reason that file transfers over longer distances are slower, in direct proportion with the increase in response times.
See http://www.internetworkexpert.org/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ for a more in-depth discussion on this.
Most download accelerators are able to increase transfer rates by simply employing multiple TCP pipes that dump into the same file. This doesn’t solve the TCP window size problem, but takes advantage of what the uplink is capable of handling. Most modern browsers do this automatically, so download accelerators are really not a necessity any more.
You may wish to optimize your per-TCP connection transfer rates though. To do so, determine your optimal TCP window size based on the expected latency of your most bandwidth intense client-base (see the calculator at the above link). Then, based on that, adjust your TCP/IP stack to adjust below:
To tweak Windows 2008 TCP Window Scaling, please refer to the following:
http://www.minasi.com/newsletters/nws0802.htm
http://www.speedguide.net/read_articles.php?id=2574
Note that Windows 2008 doesn’t allow you to tweak settings like 2003 did. You can make the system adjust it “more aggressively,” but you can’t hard code numbers in.
To tweak Windows 2003 TCP Window Scaling, please refer to the following:
http://articles.techrepublic.com.com/5100-10878_11-5034413.html
You may wish to also try: http://www.speedguide.net/tcpoptimizer.php
To tweak Linux TCP Window Scaling, please refer to the following:
http://www.speedguide.net/read_articles.php?id=121
Note that many other factors come into play for bandwidth calculation. In a hosting environment, your server must compete with other servers in the data center to reach the core routers and from there, must concentrate in various nodes and exchanges to reach a packet’s destination. Along the way, routers must prioritize and queue packets for transmission. We can check the health of this process by performing a traceroute between “slow links.” Network congestion at any one of these nodes can reduce overall transfer rate. On either one of the endpoints, disk I/O, or other system stress may be a bottleneck.
All in all, an 100Mbps, or even an 1000Mbps uplink will not provide transfer rates greater than what the network fabric in between the source and destination can handle, and not greater than what the server / client can negotiate within the TCP pipe.
#18 Feb 2010 – Edited for spelling/grammar.
#24 Mar 2010 – Updated link for 2008 tuning.
Slowloris and You
UPDATE: 20090826 – Corrected typo in “Slowloris and You.” It used to say “Slowlaris and You.” I keep getting slowloris confused with my nickname for “Solaris.” =D
Back in July, http://ha.ckers.org/slowloris/ published an exploit against Apache and other web servers (go to the link for further) that takes advantage of multi-threaded applications. It works by tying up web server threads with partial HTTP requests, then sends TCP handshakes to keep the socket open. In general, multi-threaded web servers such as httpd, apache, and apache2 are vulnerable. IIS and most proxies are not vulnerable
CERT suggested using iptables to rate limit incoming port 80 requests. In general, this should be fine for many applications, though CERT has warned that some large clients behind NAT’s may be affected and thus the hitcount/time ratio should be adjusted according to your needs.
http://www.funtoo.org/en/security/slowloris/ offers tips on mitigating this attack by enabling delayed binding on hardware load balancers.
In short, it appears as though the consensus mitigation method involves connection restrictions in the form of iptables or apache modules (most are of limited value, frankly), or shielding the web servers behind load balancers (such as HA-Proxy).
Conficker Update Part 3
According to http://forum.drweb.com/index.php?showtopic=277240 , Win32.HLLW.Shadow.based is a a variant of Conficker/downadup.
Symptom: Every available port from 1024-5000 is used to connect to various servers on destination port 445. Basically, the worm opens these connections to download and wait for malicious binaries.
The removal tools at http://www.bdtools.net/ does not detect this variant, and you have to use Dr.Web’s Cureit to detect and remove it. According to them, the recommended procedure is to install the following hotfixes:
* MS08-067
(http://www.microsoft.com/technet/security/bulletin/ms08-067.mspx);
* MS08-068
(http://www.microsoft.com/technet/security/bulletin/ms08-068.mspx);
* MS09-001
(http://www.microsoft.com/technet/security/bulletin/ms09-001.mspx).
And then run Cureit, a fully functional shareware app.
In case you’re reading this from an infected server, I’ve downloaded and included some of these files here (because if you’re infected, you won’t be able to access certain sites, drweb.com being one).
Dynamic iptables – “Flexible (and fun)”
Have you ever said to yourself that there should be a tool to do x, start building a tool to do it, then about halfway through your little project, somebody glances over your shoulder and says to you “hey, I use a tool like that, it’s called y, you should check it out,” so you do, and that tool is far more comprehensive and well built than the one you were working on?
Well this isn’t one of those times, because this tool hit me from left field while I was researching ways to mitigate a DDoS attack. Though there are many, many ways to do it, if all you have is a Linux box facing the world with nothing to hide its private parts except iptables, then this “flexible (and fun)” toolset is another weapon you can deploy when you get that 2:30AM call saying “our website’s down and I think it’s being DDoS’d.”
The tool is a simple set of scripts that make adding and removing specific IP’s quick and simple. The main site of the author is at http://www.ibm.com/developerworks/library/l-fw/, or is available (hosted locally) here.
Once installed, you can simply ban/unban an IP by typing ipdrop {IP ADDRESS} {on|off}
While perusing this thread at webhostingtalk.com, member dynamicnet mentioned grep-ing for ridiculous levels of SYN_RECV ‘d connections (this is indicative of a TCP SYN Flood attack) and generating ipdrop commands for quick banning of a SYN Flood-ing IP’s. Though you may accidentally drop one or two legitimate IP’s (have a rule already in place so you don’t ban yourself out of a remote box), you’ll likely get the bulk of the attacking IP’s.
Use netstat -n -p|grep SYN_REC | wc -l to count how many SYN_RECV connections you have.
Use netstat -n -p | grep SYN_REC | awk '{print $5}' | sort -u | awk -F: '{print "ipdrop "$1 " on"}' to generate code to ban IP’s in SYN_RECV status.
Use cat /root/.dynfw-ipdrop |awk -F: '{print "ipdrop "$1" off"}' to generates code to “undrop” those IP’s.
