Started off 30 minutes into the new year by hitting black ice and completely annihilating my favorite car. Thankfully safe but Eris didn’t make it.
Now for the DDoS fun…
I’ve been experiencing some, for lack of better word, weird issues with my hosted websites. Specifically, logging into the system and showing a “Database connection” error within my CMS install. Typically, a quick restart of mysql would handle this without any issue, but it kept happening every few weeks to few days. A little further investigation initially showed that RAM was running out due to Apache requests and was causing mysql to crash. A quick few changes in the Apache2 configuration solved this issue.
Fast forward a couple weeks, and suddenly the system isn’t taking http requests at all. When testing both remotely and internally, I’m getting either connection reset or nothing at all. Standard troubleshooting: Check system load (.12, perfect); Check logged in users for malicious attempts (negative); Mysql running? (Yup); Then, tcpudmp port 80….well look at that, 15+ requests a second (still ongoing):
11:49:29.755175 IP DEDICATED.SERVER.43485 > aasullivan.com.http: Flags [S], seq 3324596445, win 14600, options [mss 1460,sackOK,TS val 311832455 ecr 0,nop,wscale 7], length 0 11:49:29.774421 IP DEDICATED.SERVER.56992 > aasullivan.com.http: Flags [S], seq 3889640990, win 14600, options [mss 1460,sackOK,TS val 311832482 ecr 0,nop,wscale 7], length 0 11:49:29.821260 IP DEDICATED.SERVER.34260 > aasullivan.com.http: Flags [S], seq 1915664942, win 14600, options [mss 1460,sackOK,TS val 311825002 ecr 0,nop,wscale 7], length 0 11:49:29.837259 IP DEDICATED.SERVER.53969 > aasullivan.com.http: Flags [S], seq 3419901083, win 14600, options [mss 1460,sackOK,TS val 311825016 ecr 0,nop,wscale 7], length 0 11:49:29.860619 IP DEDICATED.SERVER.48308 > aasullivan.com.http: Flags [S], seq 2946818639, win 14600, options [mss 1460,sackOK,TS val 311832564 ecr 0,nop,wscale 7], length 0
Now, there was more last night when I found this and made me initially think DDoS attack since there was a total of three IPs hitting me with these. I blocked these IPs from UFW and regained access to the http server within a minute or so once the cache had cleared and everything stabilized. Further checking into the apache access loss showed everything hitting like below:
POST /xmlrpc.php HTTP/1.0
Looks like there is a way to brute force passwords into this with submitting multiple passwords into the POST of this script at once to try and get in.
So far, everything has since been stable and learned a little more on hardening the server against people with too much free time. The attack is ongoing but thankfully being blocked at the Application level by the OS now so not interfering with normal operations. I’ve added some additional pro-active protection should this happen again so I don’t need to continuously monitor the system as well. Cheers, everyone.