Apache suddenly stopped working on a fully updated CentOS 5.6 32-bit Virtuozzo VPS with Plesk 8.6.0 and ASL 2.2.11. Trying to start Apache starts one httpd process using nearly 100% CPU. Quering 'service httpd status' afterwards shows 'httpd dead but subsys locked' and Apache isn't serving pages. No amount of killing httpd processes, deleting /var/run/httpd and /var/lock/subsys/httpd and trying to restart Apache will get it back up and /var/log/httpd/error_log isn't showing anything obvious.

The only way I've found to get Apache to serve pages again is to manually kill the /usr/sbin/httpd process and run '/usr/local/psa/admin/bin/websrvmng -a -v'. I have no idea what magic Plesk's websrvmng is applying that I'm not, but after that Apache is serving pages again. The output of 'ps aux | grep apache' now shows Apache processes like '/usr/sbin/httpd -k graceful', while they were plain '/usr/sbin/httpd' before.

I tried downgrading apr (update installed yesterday), but that didn't make a difference.

Does anyone have any idea what is going on here or how I can find out for myself?

I happened to update some Centos 4 and 5 systems yesterday, including APR.

No issues so far, but in my usual pre-upgrade research regarding potential Centos 5 upgrade issues I found a forum where a number of Virtuozzo customers were complaining of very high loads and apache taking ages to serve pages.

Once person suggested it might be the iscsi initiator (which is newly updated along with mkinitrd) and therefore disableing iscsi and iscsid which load pointlessly might be an option.

I also had issues with OOM during the Centos 5 upgrade process, causing yum to get killed initially, then to just unkillably and sort of invisibly hang (there was no appropriate process listed by ps to kill). Rebooting, then doing the yum update again, then yum-complete-transactions (which had nothing to do but cleaned up the warning caused by the hung yum update) resolved all problems.

OK, so none of this sounds like your particular problem, but I wonder if there's a small possibility that it might be an OOM issue that websrvmngr is somehow getting around by restarting apache in a different way?

I'm also crossing my fingers that this APR update, which has a rewritten matching engine, might solve the mod_sec segfault issues we've been having in Centos 4. So far no segfaults, even immediately after the reboot, which was something that used to happen almost repeatably before.

I'm not confident enough to re-enable the big domain blacklists etc yet though. I'll monitor it for a while first. Load is partcularly low at the moment as it is a weekend.

p.s. for some reason the only place the OOM issues were visible were in dmesg - nothing in the Virtuozzo logs. I may have been looking in the wrong places though - it was late at night and I'm coming down with a cold so my brain is not 100%.

We have discovered that after disabling mod_security Apache does stop and start normally. Why Plesk's websrvmng is able to start Apache with mod_security enabled and Apache's own init script isn't is a mystery to us, but we will open a ticket with ASL support.

Interesting! Definitely check for oom conditions IMHO.

dmesg doesn't return anything and there are no mentions of memory problems in /var/log/messages either. We don't have access to the Virtuozzo host in this case.

So the regular init script will start apache, but not the plesk websrvmng? That sounds like a bug in websrvmng, what does Parallels have to say?

No, it's the other way around.

So what is the output of this command:

/etc/init.d/httpd restart

And are there any errors in apaches error log?

Also, graceful means something has and/or is trying to start apache with the graceful option, which means apache will wait until all requests are finished before restarting, and if something is stuck, etc. apache will just wait.

You're asking the same questions ASL support is asking.

It seemingly restarts Apache ok, but the result is the problem I described above: one httpd process eating a lot of CPU and not serving pages.

There was absolutely a bug in APR in Centos 4 that caused segfaults. In C5 there may still be, it was only fixed upstream in 2.2.17 of Apache and may not have been backported in C5. The up to date apache and apr are in the atomic repo that includes the apr fixes.

Can you post a little more detail here? Like the output of the start, and perhaps a trace of the stuck apache process?

And it sounds like if you have a hung apache process that something else is going on. Nothing has changed in modsecurity, and I've never seen this happen so I dont think that modsecurity is the cause. I looked at your case, and you have gettimeday loop which isnt in modsecurity, so if thats where the stuck apache thread or parent is then its definitely not in modsec. Something is waiting on your system and that code isnt in modsec.

So, the big question is, was this system ever working correctly? And if it was, what changed? If it wasnt ever working, do you have a similar system you can examine as well?

This is what httpd is doing in a loop according to strace (of course, the specific numbers vary):





Maybe it's not the root cause, but disabling mod_security does solve the problem and allows us to normally stop and start Apache through its init script.



Yes, it was fine until last Sunday.



The latest updates were installed through yum the day before, which were filesystem, apr and python-numeric. I thought the apr update might be the culprit, but I downgraded that package and that didn't solve the problem.



It was working before, but we don't have access to a similar system.

So that call is not modsecurity, thats apache waiting for something. If you like I can log into the system and see whats going on. Is this a virtual machine like a VPS btw?

Yes, it's a Virtuozzo VPS. See post 1, line 1.