Many of you know I like to peer at our server logs, and today was no exception. A good thing too, as /var/log/secure on one of our newly commissioned servers was full of 100s of attempts from various IPs to login via SSH under all sorts of usernames.
Now port 22 shouldn't be open on any of our systems, so this alarmed me. I use APF for all our systems, so my first port of call was to make sure that 22 had not accidentally been left in the common ingress list (it had been open for a short time during development). Nope. Nothing there.
OK, so the next thing to do is check the firewall is actually running and looks correct, so a swift iptables -v -n- L | less later I'm satisfied that all is well there too.
I have a slight diversionary panic lasting at least an hour when I notice that apf reports the main interface is 127.0.0.1 - but that turns out to be a false alarm and perfectly normal (for our systems).
Next I investigate the local apf allow_host rules file. A typo in there might have left a big hole somewhere. Nope. Empty, in fact, just as it should be.
But what about the global allow_rules? All our systems use a common set. Had I made an error in the master file that got copied accross? Nope. No errors...... all wel....errr..but what's this IP doing here? Hang on. Oh dear......
After wasting literally half a day chaising my tail trying to figure this out, it turned out to be a silly error on my part, and one that I would not have made any difference if I hadn't been too lazy to type less than 10 extra characters.
The other day I had added an additional IP to the server in question. This was the same IP I had previously been using on another server to test plesk migrations - I needed all our servers to let it in via ssh in order for the migration agent to be able to connect. And I had done this by adding it to the global allow rules as a naked IP as opposed to specifying that only connections to port 22 from that Ip were allowed. Why bother with such restrictions when I own the IP and I'm only using it for testing, and might need to open additional ports later on?
Well, I'll tell you why. The result of just entering the IP in the global allow list was that when I added the same IP to a live server, apf had dutifully added it from the global allow file as being allowed for both incoming and outgoing on all ports, effectively rendering the firewall open to all and sundry on that IP.
So there you have it. Don't be lazy when specifying firewall rules! Be specific, even when you are opening something up to an IP you own and will be using yourself!
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.