store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Wed Jul 30, 2014 3:11 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 34 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Tue May 17, 2011 3:50 pm 
Offline
Forum User
Forum User

Joined: Tue Apr 20, 2010 2:49 am
Posts: 75
Hi Breun,

Whats the output of uname -a (not that its reliable thanks to ksplice!), sounds like you're a victim of these (probably related) issues :

http://forum.parallels.com/showthread.php?t=110074 (Kernel 85.3 introduced your bug)
http://forum.parallels.com/showthread.php?t=109969

Paul


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Tue May 17, 2011 4:03 pm 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Code:
# uname -vipor
2.6.9-023stab052.4-enterprise #1 SMP Tue May 11 19:40:55 MSD 2010 i686 i386 GNU/Linux


Hm, those issues are about time being off and high load. We actually see neither of those things on this system, but thanks for trying to help.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Tue May 17, 2011 11:55 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
You mentioned you only installed a couple of rpms, did anything chance with virtuzzo? As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Wed May 18, 2011 2:17 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
mikeshinn wrote:
You mentioned you only installed a couple of rpms, did anything chance with virtuzzo?


We have no idea, as we don't have access to the Virtuozzo host node.

mikeshinn wrote:
As nothing change with modsecurity, and you suddenly have a change in apache behavior (and you installed/uninstalled apr), what else changed, if anything?


That's what we've been trying to find out as well, so far we came up with nothing.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Wed May 18, 2011 7:19 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 7876
Location: earth
Keep in mind packages can change from the master side without you being able to detect it on the guest side.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 5:59 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Thu Dec 09, 2004 11:19 am
Posts: 2024
The only changes that are likely to have happened are that the host node will have been upgraded too. Other than the kernel (we are running the latest), I can't see how anything will have changed in the Container.

Our nodes were just upgraded (from 5.5 to 5.6) and we've seen no problems other than the OOM during the yum updates on the Containers themselves that I mentioned before. We had previously updated the kernel via Ksplice and again not problems.

Breun - I've double-checked. There was nothing in the Container's /var/log/messages. Nothing in the container's logs either. But on the hardware node's dmesg (and /var/log/messages) the OOMs can be clearly seen.

Again I know this may be totally unrelated, but it might be worth asking your VPS provider if they would be willing to take a look at /var/log/messages on the host node just in case this is actually the cause of the problem. Having a process unkillably and invisibly hang due to OOM is obviously not what's supposed to happen, so there's a bug somewhere in Virtuozzo.

Faris.

_________________
--------------------------------
<advert>
If you want to rent a UK-based VPS that comes with friendly advice and support from a fellow ART fan, please get in touch.
</advert>


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 9:58 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
I just personally logged into the box, and I can't reproduce this. So either I totally misunderstand the problem, or whatever was wrong with the host node or this VPS has been fixed by someone else. Apache starts/restarts/graceful/etc. with and without modsecurity with no issues, serves websites without issues, no load problems, no delays, etc. I won't post the details (they are in the case), but I can't find anything wrong with this system.

/etc/init.d/httpd restart
/etc/init.d/httpd graceful

Both work fine with modsecurity enabled. So you can rule out modsecurity. Whatever was wrong with this system had/has nothing to do with modsecurity, maybe something was broken in the host node, dunno, but its not modsecurity as apache is working just fine with it now and I didnt change anything (except to change the modsec config from .conf.disabled to .conf).

Or was restarting apache via the init scripts not the issue?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 10:10 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Yes, that was the issue. I just checked and indeed now it just works. I'm stumped...

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 10:25 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
Er, what was the issue? I wish I could say I fixed something, but I didn't change anything. :-)

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 10:35 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu May 19, 2011 6:10 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
Quote:
Stopping and starting Apache with mod_security enabled resulted in a hanging httpd process. That was the issue. But it works fine now. And no one changed anything, so that got me stumped.


Yeah me too. Best guesses right now, something changed in the host node, or maybe this was just apache being slow for some reason and the cause was transient (DNS, i/o load, iscsi issues, etc. etc.)

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu Sep 01, 2011 8:31 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
This problem reappeared, on the same machine. I'll go and re-open the support case.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu Sep 01, 2011 9:21 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
Quote:
This problem reappeared, on the same machine. I'll go and re-open the support case.


As you recall, the last time we looked at this on this system of yours we could not reproduce this issue, it was completely transient, which means its not a problem with modsecurity or even apache, its something else with the system. I personally logged in, and nothing was wrong with the system at that time. So if this is happening again, with the same system, then its not modsecurity thats the cause, something else changed on your system or with the host node, etc. and thats the cause.

As I recall you were using Virtuozzo and Plesk in that case, which are *huge* variables, particularly the VPS which has enormous effects on the entire environment and absolutely can cause bizarre behavior like this. And in the previous case, the problem just went away on its own, we didnt and you didnt change anything. Which means its something else entirely that caused this for you, such as the OS, virtualization technology, control panel, host node, etc. So, have you also opened a case with your control panel, virtualization, host node and OS vendors?

In short, its highly likely the same thing is going to happen again: we wont be able to reproduce this on the same system because its the VPS, OS or control panel software itself thats causing this issue for you. So I highly recommend you open a case with your OS, control panel and virtualization vendors. In all likelihood, as with the previous case you opened, which was also not caused by either modsecurity, or apache, this is likely to the be the same thing here.

In that case it was some artifact of the OS or virtual environment, and its probably the same thing here, which is not something we can fix for you. :-(

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu Sep 01, 2011 9:52 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
We were able to reproduce this issue for hours and hours on end before you went to take a look. And it has been a few hours now again where we've been able to reproduce this issue. I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup. Sadly I'm also pretty sure that everyone else (OS vendor, etc.) will point to mod_security, since everything is working fine when we disable it. Plus we're running ART's httpd because of segfaults we got with the CentOS package, which is also not helping when trying to get support from others. I'm happy to believe that mod_security is not the root cause, but as long as I don't know what is there's not much I can do but disable mod_security. Any ideas on what we can do to find the root cause here?

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: httpd dead but subsys locked - only websrvmng starts Apa
Unread postPosted: Thu Sep 01, 2011 10:32 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3605
Location: Chantilly, VA
Quote:
I agree that's is probably something else locally to this server, since all of our other servers are doing fine with the same mod_security setup.


So lets eliminate variables, whats different between those systems and this one system you have that isnt behaving correctly?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 34 posts ]  Go to page Previous  1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group