store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Thu Oct 23, 2014 6:59 am

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 34 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: httpd dead but subsys locked - only websrvmng starts Apache
Unread postPosted: Sun May 15, 2011 8:03 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Apache suddenly stopped working on a fully updated CentOS 5.6 32-bit Virtuozzo VPS with Plesk 8.6.0 and ASL 2.2.11. Trying to start Apache starts one httpd process using nearly 100% CPU. Quering 'service httpd status' afterwards shows 'httpd dead but subsys locked' and Apache isn't serving pages. No amount of killing httpd processes, deleting /var/run/httpd and /var/lock/subsys/httpd and trying to restart Apache will get it back up and /var/log/httpd/error_log isn't showing anything obvious.

The only way I've found to get Apache to serve pages again is to manually kill the /usr/sbin/httpd process and run '/usr/local/psa/admin/bin/websrvmng -a -v'. I have no idea what magic Plesk's websrvmng is applying that I'm not, but after that Apache is serving pages again. The output of 'ps aux | grep apache' now shows Apache processes like '/usr/sbin/httpd -k graceful', while they were plain '/usr/sbin/httpd' before.

I tried downgrading apr (update installed yesterday), but that didn't make a difference.

Does anyone have any idea what is going on here or how I can find out for myself?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Sun May 15, 2011 9:55 am 
Online
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2085
I happened to update some Centos 4 and 5 systems yesterday, including APR.

No issues so far, but in my usual pre-upgrade research regarding potential Centos 5 upgrade issues I found a forum where a number of Virtuozzo customers were complaining of very high loads and apache taking ages to serve pages.

Once person suggested it might be the iscsi initiator (which is newly updated along with mkinitrd) and therefore disableing iscsi and iscsid which load pointlessly might be an option.

I also had issues with OOM during the Centos 5 upgrade process, causing yum to get killed initially, then to just unkillably and sort of invisibly hang (there was no appropriate process listed by ps to kill). Rebooting, then doing the yum update again, then yum-complete-transactions (which had nothing to do but cleaned up the warning caused by the hung yum update) resolved all problems.

OK, so none of this sounds like your particular problem, but I wonder if there's a small possibility that it might be an OOM issue that websrvmngr is somehow getting around by restarting apache in a different way?

I'm also crossing my fingers that this APR update, which has a rewritten matching engine, might solve the mod_sec segfault issues we've been having in Centos 4. So far no segfaults, even immediately after the reboot, which was something that used to happen almost repeatably before.

I'm not confident enough to re-enable the big domain blacklists etc yet though. I'll monitor it for a while first. Load is partcularly low at the moment as it is a weekend.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Sun May 15, 2011 9:57 am 
Online
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2085
p.s. for some reason the only place the OOM issues were visible were in dmesg - nothing in the Virtuozzo logs. I may have been looking in the wrong places though - it was late at night and I'm coming down with a cold so my brain is not 100%.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 8:39 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
We have discovered that after disabling mod_security Apache does stop and start normally. Why Plesk's websrvmng is able to start Apache with mod_security enabled and Apache's own init script isn't is a mystery to us, but we will open a ticket with ASL support.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 8:56 am 
Online
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2085
Interesting! Definitely check for oom conditions IMHO.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 8:59 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
dmesg doesn't return anything and there are no mentions of memory problems in /var/log/messages either. We don't have access to the Virtuozzo host in this case.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 11:58 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
So the regular init script will start apache, but not the plesk websrvmng? That sounds like a bug in websrvmng, what does Parallels have to say?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 12:13 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
mikeshinn wrote:
So the regular init script will start apache, but not the plesk websrvmng?


No, it's the other way around.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 1:11 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
So what is the output of this command:

/etc/init.d/httpd restart

And are there any errors in apaches error log?

Also, graceful means something has and/or is trying to start apache with the graceful option, which means apache will wait until all requests are finished before restarting, and if something is stuck, etc. apache will just wait.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 1:13 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
You're asking the same questions ASL support is asking. :)

It seemingly restarts Apache ok, but the result is the problem I described above: one httpd process eating a lot of CPU and not serving pages.

_________________
Lemonbit Internet Dedicated Server Management


Last edited by breun on Mon May 16, 2011 1:46 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 1:15 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Quote:
I'm also crossing my fingers that this APR update, which has a rewritten matching engine, might solve the mod_sec segfault issues we've been having in Centos 4. So far no segfaults, even immediately after the reboot, which was something that used to happen almost repeatably before.


There was absolutely a bug in APR in Centos 4 that caused segfaults. In C5 there may still be, it was only fixed upstream in 2.2.17 of Apache and may not have been backported in C5. The up to date apache and apr are in the atomic repo that includes the apr fixes.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 1:18 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Quote:
It seemingly restarts Apache ok, but the result is the problem I described above: one httpd process eating a lot of CPU and not serving pages.


Can you post a little more detail here? Like the output of the start, and perhaps a trace of the stuck apache process?

And it sounds like if you have a hung apache process that something else is going on. Nothing has changed in modsecurity, and I've never seen this happen so I dont think that modsecurity is the cause. I looked at your case, and you have gettimeday loop which isnt in modsecurity, so if thats where the stuck apache thread or parent is then its definitely not in modsec. Something is waiting on your system and that code isnt in modsec.

So, the big question is, was this system ever working correctly? And if it was, what changed? If it wasnt ever working, do you have a similar system you can examine as well?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 6:39 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
mikeshinn wrote:
Can you post a little more detail here? Like the output of the start, and perhaps a trace of the stuck apache process?


This is what httpd is doing in a loop according to strace (of course, the specific numbers vary):

Quote:
read(103, "", 12828) = 0
gettimeofday({1305548222, 894721}, NULL) = 0
gettimeofday({1305548222, 894783}, NULL) = 0
gettimeofday({1305548222, 894845}, NULL) = 0
gettimeofday({1305548222, 894909}, NULL) = 0
poll([{fd=103, events=POLLIN}], 1, 3000) = 1 ([{fd=103, revents=POLLHUP}])


mikeshinn wrote:
And it sounds like if you have a hung apache process that something else is going on. Nothing has changed in modsecurity, and I've never seen this happen so I dont think that modsecurity is the cause. I looked at your case, and you have gettimeday loop which isnt in modsecurity, so if thats where the stuck apache thread or parent is then its definitely not in modsec. Something is waiting on your system and that code isnt in modsec.


Maybe it's not the root cause, but disabling mod_security does solve the problem and allows us to normally stop and start Apache through its init script.

mikeshinn wrote:
So, the big question is, was this system ever working correctly?


Yes, it was fine until last Sunday.

mikeshinn wrote:
And if it was, what changed?


The latest updates were installed through yum the day before, which were filesystem, apr and python-numeric. I thought the apr update might be the culprit, but I downgraded that package and that didn't solve the problem.

mikeshinn wrote:
If it wasnt ever working, do you have a similar system you can examine as well?


It was working before, but we don't have access to a similar system.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Mon May 16, 2011 8:21 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Quote:
read(103, "", 12828) = 0
gettimeofday({1305548222, 894721}, NULL) = 0
gettimeofday({1305548222, 894783}, NULL) = 0
gettimeofday({1305548222, 894845}, NULL) = 0
gettimeofday({1305548222, 894909}, NULL) = 0
poll([{fd=103, events=POLLIN}], 1, 3000) = 1 ([{fd=103, revents=POLLHUP}])


So that call is not modsecurity, thats apache waiting for something. If you like I can log into the system and see whats going on. Is this a virtual machine like a VPS btw?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Tue May 17, 2011 2:23 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
mikeshinn wrote:
Is this a virtual machine like a VPS btw?


Yes, it's a Virtuozzo VPS. See post 1, line 1. :)

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 34 posts ]  Go to page 1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group