My little VPS went crazy yesterday. Unresponsive, it took ages to log in. I quickly discovered multiple attacks, some attacks on WordPress’s xmlrpc.php and wp-login.php, a sustained attack on imaps from China and several concurrent brute force ssh attacks all at once. The poor little VPS kept running out of memory which caused it to kill off memory hogs – generally Apache and Mysql. At the stage I should add – and probably question my sanity – that it’s good fun thwarting such things and I’ve been doing similar for 10+ years…
Then I discovered that the out of band access to the VPS wasn’t working and I assumed this to be a part of the greater whole. It wasn’t.
My provider, Heart Internet has very good technical support. I’ve used them for ages for my own VPS as well as professionally, as have others partly due to the fact that I recommended them, something I very rarely do. They came up with a very detailed analysis. First off, an Ubuntu upgrade had knobbled /dev/ttyS0 so no out of band access and this was not the fault of the attack, I’d just never needed to use it so was unaware. A quick fiddle in the GRUB config sorted that. Next, the VM framework was itself suffering a high i/o load and that was causing my VPS to pause. These pauses then made matters worse as it seems that mysql inserts were queueing up and then went in with a bang when the VPS got some CPU time again. That, plus the WordPress attack caused the memory killer to terminate Apache and mysql causing further issues when they restarted. This was not eased by my watchdog process that restarts any failed PHP scripts which spend their life pulling in railway data and stuffing it into a mysql db meaning that as soon as mysqld went back in the PHP process would drag in a bag load of data and fire off tons of inserts.
To add insult to injury fail2ban, er, failed to ban. It had upgraded itself at some stage to a non-working state and I really could not spent the time reading the docs to find out what it needed, so it got purged and denyhosts, which I used to use anyway was installed and is working nicely. Not quite the same thing, but ideal for ssh attacks as it simply adds the IPs to /etc/hosts.deny. I still used iptables when, for example I see zillions of spam injections, but those are infrequent and sort themselves out in any case. But it’s still nice to see them suddenly stop when I can be bothered to look.
Having tidied up the little beggars via iptables and protecting the bits of WordPress that were under attack, and with the VM framework finally calmed down by the provider my VPS could once more tick along quite nicely at its usual load of, well, pretty much zero.
Fun nonetheless. Perhaps I’m just weird!