dropbox.com
I wanted to bring up Dropbox.com to your attention. I wanted to share what I thought was a fantastic company that’s basically taken the concept of JungleDisk and moved it one step further. How? Instead of really cheap (Jungledisk), dropbox is free for the first 2 gigs. This, combined with the below usability features, results in rapid growth in user adoption. As of January, the company has over 4 million users. That’s over 8 Petabytes of potential disk space usage if every user was a free 2GB plan.
All of the public infrastructure is hosted at AWS, and can thus scale with the userbase. As Amazon charges anywhere from $0.055 to $0.150 per GB (pricing structure), each free user consumes from $0.11 to $0.30 per month. Even with 6.25% utilization of 8 Petabytes, Dropbox.com pays the $0.105 per GB rate, Given their next paid upgrade is for 50 GB and costs $9.99/month, one paid customer can support the storage fees for up to ($9.99 − 50 clients × $0.105) ÷ $0.21 = 22 clients covered in the cost of a single paid user, at full utilization of each user. However, most users won’t be using their full utilization (think Google mail), and their business model becomes more lucrative.
Starting with dropbox is simple. Download the client, register your account, and a folder is created where you can drag and drop your data. All data in this folder is replicated to dropbox servers, and to all dropbox client nodes that you link to your account.
The application ships with an intuitive user interface (no S3 accounts to configure, as this is done for you), and a rich feature set that includes automatic versioning, automatic syncing, cross-platform compatibility, intuitive web publishing, and a simple pricing model.
The one feature that really stood out however, was how they garner feedback for this actively developed product. Users can navigate to https://www.dropbox.com/votebox, where they can submit suggestions that get voted on and commented upon. This popularity contest likely helps the company focus on what projects need developing next.
All in all, this is an exciting company to be a customer of, and I would recommend y’all take a look at it too, as this appears to be what proper execution of a good idea looks like.
Kudos dropbox.com on a product well done.
DoS Madness! Part 1: Memory Management / Behind Your Firewall
Alright, so here’s the bottom line. Your site is down. It’s your job not only to fix it this time, but also, to keep this kind of stuff from happening. Well, a DoS Attack in any of its varieties will rely on a single tenet: your service (www.yoursite.com) relies on a successive chain of processes and devices to generate, package, and deliver its content to your clients. A DoS attack is, in practice, any sort of attack that exploits one or more of these subsystem’s vulnerability to being overwhelmed. It then follows, that to reduce the potentiality for a successful DoS attack, one must reduce the ways in which your site’s subsystems will be overwhelmed.
With that thought in mind, let’s take a look at a couple different DoS attacks and the subsystem they target. The next series of articles will break down DoS types by the general subsystem they exploit.
Part 1: Memory Management / Behind Your Firewall
One type of DoS attack that takes advantage of poorly configured web servers is slowloris. This attack opens as many connections as your web server can handle, then keeps the connections open, with occasional
slowloris: hey.
webserver: what?
slowloris: ...
webserver: ...
webserver: are you still there?
slowloris: hey...
webserver: what?
slowloris: ...
statements.
Slowloris can be mitigated by rate limiting incoming connections to Apache and / or using a non-vulnerable front end web server, such as nginx.
Another tool that can be used maliciously is Google’s skipfish. This is a security scanner that can double as a load tester. Basically skipfish will form URLs based off of crawled data and dictionary based guesses. Used maliciously, it can overwhelm a poorly deployed LAMP stack. These servers may run out of memory and crash.
When it does, there may be an OOM_Killer message in your /var/log/messages file as a final, “OMG-help-me dump” right before reboot. Memory is a scarce resource in a system, and to improve performance, the Linux kernel will overcommit RAM to the numerous processes that request an allocation of memory space. Read more about the Linux kernel’s overcommit behavior here. In summary, it’s kind of like simultaneously sitting at five different $5 dollar black jack games with $20 in chips. Most times, a couple games will cash out, releasing chips back into the resource pool, and there is no net loss. The kernel runs faster, and everybody wins. That is, until traffic increases beyond what the system can handle, and everybody loses. When this happens, the kernel has to run from at least one table (and an angry pit boss). OOM_Killer is a program that was designed to handle exceptions to a linux kernel’s overcommit behavior and uses algorithms to determine which process to kill.
A particularly amusing analogy by Andries Brouwer describes OOM_Killer thus:
An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.
So if you went to the win.tue.nl link I showed you, you’ll note that like a scared gambler, the kernel (after 2.5.30) can be instructed to be more reserved when allocating memory. The values in the below files will moderate this behavior.
echo 2 > /proc/sys/vm/overcommit_memory #this will tell the kernel to never commit memory over the swap space and a certain fraction of physical memory.*
echo 0 > /proc/sys/vm/overcommit_memory #(default) the kernel will do what it thinks is best
echo 1 > /proc/sys/vm/overcommit_memory #this will tell the kernel to go hog-wild and never refuse an app memory
*This fraction is defined in “/proc/sys/vm/overcommit_ratio” – the default value is 50 (=50%
But that’s not the cure-all to memory management. That just dictates kernel behavior. App behavior is another story. Here, you have to ensure your app never asks for more memory than your system can afford to give out. Now this is going to differ based on what type of server it is, but the most commonly exploited servers (from my perspective anyways) are LAMP stacks. And of the LAMP stack, the two most common reasons why OOM_Killer is invoked is:
Too many Apache worker processes. Take a look at your MaxClients declaration in your httpd.conf file. Now, looking at the MPM (by default, it’s usually prefork, which means 1 thread per process), multiply the number of processes for your MaxClients by the average size of your Apache processes, and you’ll have the total memory allocation for Apache at full load. Naturally, you do not want this figure to be anywhere close to your physical RAM limit….
For more on LAMP stack tuning, please refer to the following links:
http://httpd.apache.org/docs/2.0/misc/perf-tuning.html
http://dev.mysql.com/doc/refman/5.0/en/server-parameters.html
http://www.interworx.com/forums/showthread.php?p=2346
http://www.ibm.com/developerworks/views/linux/…
Finally, you may wish to consider using a lighter web server, or a reverse proxy cacher to optimize performance. The bottom line is, when faced against an array of incoming attempts to overwhelm and disable your backend applications, you want your system to be robust enough to gracefully handle what it can, and redirect or ignore the rest.
In Part 2: Socket Management / Behind Your NIC, we’ll be looking at SYN Floods and other, non-port saturating attacks. Now as this series progresses, there will be some overlap in mitigation techniques that will impact more than one type of DoS attack. As long as these correlations are mutually beneficial, we’ll be just hunky-dory.
FreeBSD8.0 on Hyper-V
Yes, it works. I’ve recently decided it was about time to get to know a bit more about BSD. I’ve built ZFS boxes on OpenSolaris before, but CARP and PF kind of sealed the deal for me. So I decided to build a test VM on my Hyper-V server. There are some quirks however.
As I’m finding to be a trend for Linux guests, you must replace the default virtual NIC with the “Legacy Network Adapter” (you have to do this for Debian as well). Just follow the installer’s guidelines for the rest of the installation.
After it is installed, rebooting the OS becomes a problem. Hyper-V doesn’t seem to agree very well with the FreeBSD’s ACPI calls so in the interim, you can identify the hung VM by logging into powershell and running the following:
Get-WmiObject -Namespace root\virtualization msvm_computersystem | Format-Table ElementName,ProcessId
Then kill the process id with “kill [pid]“
In task manager,
- Inspect the “Command Line” field from the open vmconnect.exe process. Note the GUID of the VM (after the -G switch).
- Look for that GUID in the vmwp.exe “Command Line” field.
- Right click > End Process
The folks at peach.ne.jp have released a patch to correct this. Fetch it from here or failing that, locally.
On the VM, perform the following:
cd /usr/src
bzcat < /path/to/fbs80-200911-hvpatch.bz2 | patch -p1
make buildkernel
make installkernel
shutdown -r now
More to come as I start wading in.
Originals:
* http://shell.peach.ne.jp/aoyama/archives/577
* http://blogs.technet.com/abeshkov/archive/2008/12/15/3169299.aspx
New Theme
So, blog.jim80.net has a new theme installed. It’s called Open Sourcerer 1.4 by Alan Lord, and it’s got one thing iNove 1.4.5 by mg12 didn’t, flexible width columns. Well, technically, iNove could be made to adjust according to screen resolution, but the resultant view on a widescreen wasn’t very pretty. if anything it just looked silly. Thus, I present Open Sourcerer. It’s easier on the eyes, if it is a bit plain. Frankly, the column width bit is a necessity for a post I’m working on. Hard coded to the previous size, it would have been a nightmare to read. Trust me, I’m doing you a favor.
=P
Sean-One – Log Ya Mods
So, one of my admin colleagues, Sean, decides to write up some nerdy-lyrics to a rap song, and I thought it’d be a colourful addition to this blog.
This is the first verse to KRS-One – Build Ya Skillz. Lyrics possibly NSFW.
Sean-One – Log Ya Mods
2010 erplefoo industries (erplefoo@gmail.com )Check, I control your realm with one password I speak
And spread your auth all over like a raid level 5′s disk seeks
Admins get kicked in they mouth with cleats
Kuz their plans failed to reach servers at the beach
Have a seat quick, I’ll set up your account bits, time to complete shit
Looks like I need to reset shit, I mean set perms properly
I can feel myself becoming an admin monopoly
Others will copy mtree but apply my shit sloppily
Shocking me with static straps not entirely attached to me
Electroshock it seems to be
My true identity is never meant to be
Exposed in the sun that shit’s bright daily(chorus)
But admins mod too much shit and fuckin’ forget to log it
LOG YOUR MODS
Admins talk too much shit
And send quotes up to reddit
LOG YOUR MODS
You can follow Sean on Twitter at @sdfoo
