With that thought in mind, let’s take a look at a couple different DoS attacks and the subsystem they target. The next series of articles will break down DoS types by the general subsystem they exploit.

Part 1: Memory Management / Behind Your Firewall

One type of DoS attack that takes advantage of poorly configured web servers is slowloris. This attack opens as many connections as your web server can handle, then keeps the connections open, with occasional

slowloris: hey.
webserver: what?
slowloris: ...
webserver: ...
webserver: are you still there?
slowloris: hey...
webserver: what?
slowloris: ...

statements.

Slowloris can be mitigated by rate limiting incoming connections to Apache and / or using a non-vulnerable front end web server, such as nginx.

Another tool that can be used maliciously is Google’s skipfish. This is a security scanner that can double as a load tester. Basically skipfish will form URLs based off of crawled data and dictionary based guesses. Used maliciously, it can overwhelm a poorly deployed LAMP stack. These servers may run out of memory and crash.

When it does, there may be an OOM_Killer message in your /var/log/messages file as a final, “OMG-help-me dump” right before reboot. Memory is a scarce resource in a system, and to improve performance, the Linux kernel will overcommit RAM to the numerous processes that request an allocation of memory space. Read more about the Linux kernel’s overcommit behavior here. In summary, it’s kind of like simultaneously sitting at five different $5 dollar black jack games with $20 in chips. Most times, a couple games will cash out, releasing chips back into the resource pool, and there is no net loss. The kernel runs faster, and everybody wins. That is, until traffic increases beyond what the system can handle, and everybody loses. When this happens, the kernel has to run from at least one table (and an angry pit boss). OOM_Killer is a program that was designed to handle exceptions to a linux kernel’s overcommit behavior and uses algorithms to determine which process to kill.

A particularly amusing analogy by Andries Brouwer describes OOM_Killer thus:

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.

So if you went to the win.tue.nl link I showed you, you’ll note that like a scared gambler, the kernel (after 2.5.30) can be instructed to be more reserved when allocating memory. The values in the below files will moderate this behavior.

echo 2 > /proc/sys/vm/overcommit_memory #this will tell the kernel to never commit memory over the swap space and a certain fraction of physical memory.*
echo 0 > /proc/sys/vm/overcommit_memory #(default) the kernel will do what it thinks is best
echo 1 > /proc/sys/vm/overcommit_memory #this will tell the kernel to go hog-wild and never refuse an app memory

*This fraction is defined in “/proc/sys/vm/overcommit_ratio” – the default value is 50 (=50%

But that’s not the cure-all to memory management. That just dictates kernel behavior. App behavior is another story. Here, you have to ensure your app never asks for more memory than your system can afford to give out. Now this is going to differ based on what type of server it is, but the most commonly exploited servers (from my perspective anyways) are LAMP stacks. And of the LAMP stack, the two most common reasons why OOM_Killer is invoked is:

For more on LAMP stack tuning, please refer to the following links:
http://httpd.apache.org/docs/2.0/misc/perf-tuning.html
http://dev.mysql.com/doc/refman/5.0/en/server-parameters.html
http://www.interworx.com/forums/showthread.php?p=2346
http://www.ibm.com/developerworks/views/linux/…

Finally, you may wish to consider using a lighter web server, or a reverse proxy cacher to optimize performance. The bottom line is, when faced against an array of incoming attempts to overwhelm and disable your backend applications, you want your system to be robust enough to gracefully handle what it can, and redirect or ignore the rest.

In Part 2: Socket Management / Behind Your NIC, we’ll be looking at SYN Floods and other, non-port saturating attacks. Now as this series progresses, there will be some overlap in mitigation techniques that will impact more than one type of DoS attack. As long as these correlations are mutually beneficial, we’ll be just hunky-dory.

On October 3, T-Mobile Chief Operations Officer, Jim Alling wrote the following post on the T-Mobile forum site:

Dear valued T-Mobile Sidekick customers:

I realize that for many of you, your T-Mobile Sidekick is how you stay in touch with your friends, family and others.  I sincerely apologize for the impact the current disruption of data services may be having on you.  I assure you that T-Mobile is working very closely with Danger/Microsoft to resolve the issue as quickly as possible.  T-Mobile-supported services, such as voice calls and SMS/MMS, have not been affected and continue to be operational.  Danger/Microsoft has been working, and will continue working through the week, to restore data functionality and other features.

I understand that this data service disruption is very frustrating to our valued Sidekick customers.  For many years, the Sidekick has been, and continues to be, a cornerstone device for T-Mobile.  And we believe Sidekick customers are among the most loyal customers anywhere.  Recognizing that, and to address any inconvenience Sidekick data customers are experiencing, T-Mobile will automatically credit one month of data service to customers who subscribe to T-Mobile Sidekick data plans.  There is nothing you need to do to get this credit – T-Mobile will post the credit to these accounts in the coming days.

We will continue to post the latest information and FAQs to these Forums. I appreciate you being a loyal T-Mobile customer, and appreciate your patience as everyone works hard to resolve the current issues.  Thank you.

Sincerely,

Jim Alling, Chief Operations Officer, T-Mobile USA

Then, after a torrent of discussion on the forum site, the following update was provided earlier today:

Dear valued T-Mobile Sidekick customers:

We are thankful for your continued patience as Microsoft/Danger continues to work on preserving platform stability and restoring all services for our Sidekick customers.  We have made significant progress this past weekend, restoring services to virtually every customer.  Microsoft/Danger has teams of experts in place who are working around-the-clock to ensure this stability is maintained.

Regarding those of you who have lost personal content, T-Mobile and Microsoft/Danger continue to do all we can to recover and return any lost information.  Recent efforts indicate the prospects of recovering some lost content may now be possible.  We will continue to keep you updated on this front; we know how important this is to you.

In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card.  This will be in addition to the free month of data service that already went to Sidekick data customers.  This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill.  For those who fall into this category, details will be sent out in the next 14 days – there is no action needed on the part of these customers.  We however remain hopeful that for the majority of our customers, personal content can be recovered.
===
Dan
Moderator, T-Mobile Forums

At this time, neither Microsoft nor T-Mobile have confirmed conjecture that a SAN update caused the failure:

So yeah..

I would like to know what discounts are T-mobile going to give on a new Phone. I am probably going to move to the Moto Cliq, But I and other sidekick users should get a full phone discount not just a % of it..  (Microsoft should pay for it)

hmm Roz Ho haven’t you her of BACKUP…?

Quoting Hiptop3

“Currently the rumor with the most weight is as follows:

Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved. “

WOW.

Microsoft Do you understand that you are making yourself and T-mobile loose MONEY????

Also with me being a Sidekick owner I feel betrayed by Microsoft not T-mobile.

This outage I was all fine about at first but now it is just to much. We sidekick owners rely on Danger witch is now owned by Micro to keep are data stored on a secure server and that is why us users never backed up are data. I mean the sidekick does not even have a mass contact save Option. The user has to save them one by one. If I do stay with the sidekick I would like to see Options to save all on SD becuase a SIM can only hold around 250..

I have lost business and meetings from this outage and I am not happy.

So to everyone

It is not T-mobiles Fault so do not blame them. There customer service has been AWESOME

Also Danger and Microsoft do not comunicate with T-mobile as much that is why there is not much info.

“I wonder if we call Microsoft and bug them will they give us any info, they will probably say u have to call t-mobile. Well T-mobile is not the one who messed up,.they do not UPDATE THE SAN…..”

After a week of attempting to salvage the data, it would appear as though Microsoft was unsuccessful in doing so. If the SAN speculation is correct, then it was simply a failure of the data’s underlying SAN. The question is, why should a failing SAN bring with it the data of an entire customer base? I severely doubt that this would have occurred had this been a normal hardware breakdown. Well-designed storage solutions are built with the precondition of being able to survive a head failure, network failure, any sort of failure, really, without losing data. One would thus speculate that gross human error was at fault, and frankly, that means that management was not doing their job. Not enough layers of redundancy were built into this system, and not enough protective layers were written into policy to prevent this human error, or whatever it was, from cascading into a data-lost scenario. Data management is a big responsibility, and not enough resources go into its upkeep in many firms. It would thus appear that Microsoft appears to be one of the latter.

Back in July, http://ha.ckers.org/slowloris/ published an exploit against Apache and other web servers (go to the link for further) that takes advantage of multi-threaded applications. It works by tying up web server threads with partial HTTP requests, then sends TCP handshakes to keep the socket open. In general, multi-threaded web servers such as httpd, apache, and apache2 are vulnerable. IIS and most proxies are not vulnerable

CERT suggested using iptables to rate limit incoming port 80 requests. In general, this should be fine for many applications, though CERT has warned that some large clients behind NAT’s may be affected and thus the hitcount/time ratio should be adjusted according to your needs.

http://www.funtoo.org/en/security/slowloris/ offers tips on mitigating this attack by enabling delayed binding on hardware load balancers.

In short, it appears as though the consensus mitigation method involves connection restrictions in the form of iptables or apache modules (most are of limited value, frankly), or shielding the web servers behind load balancers (such as HA-Proxy).

Symptom: Every available port from 1024-5000 is used to connect to various servers on destination port 445. Basically, the worm opens these connections to download and wait for malicious binaries.

The removal tools at http://www.bdtools.net/ does not detect this variant, and you have to use Dr.Web’s Cureit to detect and remove it. According to them, the recommended procedure is to install the following hotfixes:
* MS08-067
(http://www.microsoft.com/technet/security/bulletin/ms08-067.mspx);

* MS08-068
(http://www.microsoft.com/technet/security/bulletin/ms08-068.mspx);

* MS09-001
(http://www.microsoft.com/technet/security/bulletin/ms09-001.mspx).

And then run Cureit, a fully functional shareware app.

In case you’re reading this from an infected server, I’ve downloaded and included some of these files here (because if you’re infected, you won’t be able to access certain sites, drweb.com being one).

Well this isn’t one of those times, because this tool hit me from left field while I was researching ways to mitigate a DDoS attack. Though there are many, many ways to do it, if all you have is a Linux box facing the world with nothing to hide its private parts except iptables, then this “flexible (and fun)” toolset is another weapon you can deploy when you get that 2:30AM call saying “our website’s down and I think it’s being DDoS’d.”

The tool is a simple set of scripts that make adding and removing specific IP’s quick and simple. The main site of the author is at http://www.ibm.com/developerworks/library/l-fw/, or is available (hosted locally) here.

Once installed, you can simply ban/unban an IP by typing ipdrop {IP ADDRESS} {on|off}

While perusing this thread at webhostingtalk.com, member dynamicnet mentioned grep-ing for ridiculous levels of SYN_RECV ‘d connections (this is indicative of a TCP SYN Flood attack) and generating ipdrop commands for quick banning of a SYN Flood-ing IP’s. Though you may accidentally drop one or two legitimate IP’s (have a rule already in place so you don’t ban yourself out of a remote box), you’ll likely get the bulk of the attacking IP’s.

Use netstat -n -p|grep SYN_REC | wc -l to count how many SYN_RECV connections you have.

Use netstat -n -p | grep SYN_REC | awk '{print $5}' | sort -u | awk -F: '{print "ipdrop "$1 " on"}' to generate code to ban IP’s in SYN_RECV status.

Use cat /root/.dynfw-ipdrop |awk -F: '{print "ipdrop "$1" off"}' to generates code to “undrop” those IP’s.

Our favorite worm got an update 8 April according to Network World. Read more here…
And of course, at downadup.org.

Key Genius is a product that leverages the Yubikey to provide a more secure platform for logging into websites. It recently won an award in the Yubiking competition, in partnership with Security Now. The site stores site-specific passwords in an encrypted database, and using a browser extension, replies to valid Yubikey passwords by automatically inserting the correct password for the site. Usernames are not stored by Key Genius at all, so it’s up to the user to supply them to the website in question. This is actually a good thing, as compromise of the Key Genius database cannot in of itself bely a user’s logon credentials. This is a neat product that not only enhances convenience, but does so in a secure manner.

The headline at dailymail.co.uk read “April Fool’s Day computer virus is activated… but fails to cause internet chaos.”

I guess the rumors were unfounded. However, it’s important to note that the virus is still rampant and speculation on the potential uses of such a huge botnet are as well. Some surmise that it might be used to DDOS the crap out of some poor server(s). It might also be used to crack passwords or encryption. Check out http://downadup.org to read more and for removal tools. It’s also a good idea to prepare your network for the potentiality of attack. Don’t be a soft target.

Here’s a couple (read non-comprehensive) ideas on how to not be a soft target:

• Backup, backup, backup
  • Have systems ready to leap into action if necessary, and keep at least one form of backup offline in case of worst-case scenarios.
  • If you don’t already have a backup strategy in place, it’s time to implement one.
• Control access to your critical services
  • Enforce strong passwords – or better yet, employ multi-factor authentication. PPP is a strong candidate for the thrifty.
  • Audit your users – does that guy who quit last year still have an active user account? Do your non-administrative users have access to critical servers?
  • Use fail2ban or iptables to detect and drop password-guessing attacks – even with 10 million + IP’s to choose from, it’s not easy to crack a password/one-time password combination when you only get 3 tries per IP.
• Watch your traffic (not really a botnet vulnerability, but good practice in general):
  • Control your legacy services – seriously, it’s time to retire telnet and other services that transmit passwords in cleartext.
  • https > http – especially when it comes to passwords. Don’t allow users the ability to transmit passwords over http.
• etc…

I’ve hardly compiled a comprehensive list, and I welcome comments for other good practices, but the most important takeaway is to be cognizant of your security stance. Don’t make it easy for the bad guys.

Here’s the first sentence from the source document:

Using equipment costing about $80, researchers from Inverse Path were able to point a laser on the reflective surface of a laptop between 50 feet and 100 feet away and determine what letters were typed.

…read the rest

The gist of the article is that researchers were able to determine key sequences by analyzing the sound waves produced by each key as subjects type. The researchers have also managed to isolate EMI transferred through the power grid to identify keystrokes. Most folks need not worry about directed attacks of this nature, but if you’ve got a secret somebody wants badly, watch your back… and your windows.

Note: Thanks to _dilan for the link!

This isn’t a news blog. I’m not so plugged in that I can write about things as they happen. That’s not my impetus. My impetus is to publish articles about ideas that I believe are sound principles when planning your systems architecture. One such principle is not betting the house on a single technology, and WHT was a prime example of the benefits of such planning. After suffering an attack like that, WHT is online and serving traffic. How? They had DVD backups. Try erasing that. Yes, they’ve only managed to restore up to October 2008, and that was likely a result of corrupted media or backup operator error, but their site isn’t down. It’s an arduous process, writing to permanent media. And it’s an arduous process restoring from it. But in a last-ditch scenario like this, it served WHT well. And it proves my point. Crossing architectural boundaries when backing up data is a necessity, and that means keeping at least one form of backup offline.

Would your business be able to take a hit like WHT’s and keep its doors open?