My Sunday morning was ruined by a notification email from a monitoring service letting me know that my server was down. Eek! I took the normal first steps: try to SSH to the box and remote reboot. Still offline.
I started a chat with my hosting company’s tech support. They said that it looked like a SYN flood was in progress, and that I’d need to handle it on my own. They pointed to a few resources that gave suggestions on what to do, and helped me figure out how to SSH into my box on the hosting company’s private network (because the SYN flood was only affecting public network access). Overall, excellent support from SoftLayer.
Most websites that talked about SYN floods recommended turning on SYN cookies and adding a few iptables rules. With further research (and experience) I discovered that neither of these options really work well. I wasn’t even entirely convinced that I was being flooded. I think all the traffic was legitimate (I run a lot of popular sites).
So what to do? I read a few sites and came up with a solution.
First you need to keep in mind that a lot of default networking settings were developed way back in the day when computers weren’t nearly as powerful as they are now. My server is more than capable of handling the traffic but the default settings were keeping connections from getting through. To fix this, I needed to tell my server that it could handle additional connections. In a shell I issued these commands:
echo 16384 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 16384 > /proc/sys/net/core/somaxconnNext I updated a config file, /etc/sysctl.conf, so these settings will still be there next time I reboot:
net.ipv4.tcp_max_syn_backlog = 16384
net.core.somaxconn = 16384Last, I had to tell Apache to handle more connections as well by editing the httpd.conf file:
ListenBacklog 16384I restarted Apache and BAM! My sites were accessible again! Yay! My load average is still around 0.05 to 0.10 most of the time. Lots of free memory. Fast page loads. Everything works great. All it took was giving my server a little self confidence by letting it know it was capable of handling the traffic it was getting.