store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Tue Oct 21, 2014 9:32 am

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: Strange httpd activity.
Unread postPosted: Wed Aug 18, 2010 5:09 am 
Offline
Forum User
Forum User

Joined: Fri Jul 23, 2010 10:03 am
Posts: 38
Location: UK
A few times now our server load has suddenly started to climb dramatically, as has the use of swap, bringing it to its knees and making it appear to go down to website and webmail users.

We have managed to stop this, each time by doing an lsof -i :80 (after waiting for some time for ssh to respond) and seeing that httpd is creating abnormal amounts of processes to listen in on ports.
We then have to kill all httpd processes and restart httpd.

Unfortunately we have been unable to work out what seems to be triggering this. I have looked at various error and access logs which have shown no unusual activity around the time that the issues started and I am running out of ideas. This problem has only recently emerged and may be something to do with recent updates, as we didn't really experience this until after the ASL 2.2.10 upgrade.

Any ideas you might have would be most welcome.


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 18, 2010 5:33 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
Maybe Apache's MaxClients setting is too much for your machine to handle?

http://httpd.apache.org/docs/2.2/mod/mp ... maxclients

Google for tuning Apache MaxClients and you should find some info.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 18, 2010 6:01 am 
Offline
Forum User
Forum User

Joined: Fri Jul 23, 2010 10:03 am
Posts: 38
Location: UK
hi Breun,
Thank you for the reply.

I have considered this but that does not appear to be the cause.
I have also increased maxclients a few times before and think it may be unwise to increase it further at this point.


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 18, 2010 6:29 am 
Offline
Long Time Forum Regular
Long Time Forum Regular

Joined: Sat Aug 20, 2005 9:30 am
Posts: 2812
Location: The Netherlands
My suggestion actually was that it may have been set too *high* for what your hardware can handle.

_________________
Lemonbit Internet Dedicated Server Management


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 18, 2010 12:41 pm 
Offline
Forum User
Forum User

Joined: Fri Jul 23, 2010 10:03 am
Posts: 38
Location: UK
Oh yes sorry breun I must have misread your post.
Sadly this still doesn't appear to be the cause


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Tue Aug 24, 2010 7:59 am 
Offline
Forum User
Forum User

Joined: Fri Jul 23, 2010 10:03 am
Posts: 38
Location: UK
This still occurs, though less frequently than when I first posted.

It does only happen during working hours so may be less to do with general traffic and more to do with something that either we or one of our clients is doing related to the server.
Still no luck so far with finding anything in common between the times it happened.


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Tue Aug 24, 2010 9:25 am 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 7935
Location: earth
mod_rewrite rules possibly? Look around for .htaccess files


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 25, 2010 9:46 am 
Offline
Forum User
Forum User

Joined: Fri Jul 23, 2010 10:03 am
Posts: 38
Location: UK
I've had a look and can't see anything abnormal with regards to the .htaccess files relating to mod_rewrite.
However, I don't know a huge amount about mod_rewrite and htaccess (what I do know is gleaned from googling it) so I could be missing something.


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Aug 25, 2010 3:25 pm 
Offline
Forum Regular
Forum Regular

Joined: Wed May 12, 2010 5:40 pm
Posts: 137
most of your dynamic sites such as joomla, wordpress, forums, etc will have an .htaccess file in the site's root directory, and in most cases this has directives for mod-rewrite....


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Dec 28, 2011 1:45 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
scott wrote:
mod_rewrite rules possibly? Look around for .htaccess files


I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?

I have quite a few sites that use mod_rewrite for sef - does this conflict with anything in asl?

I turned off mod_security (I actually turned off WAF) and so far the server load has not skyrocketed, though it is a tad high @ 1.45 but thats MUCH better than the 40 i got woke up to this morning

:)


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Dec 28, 2011 4:46 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3656
Location: Chantilly, VA
Quote:
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?


mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: Strange httpd activity.
Unread postPosted: Wed Dec 28, 2011 5:45 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
copied this to viewtopic.php?f=3&t=5638&p=32530#p32530 so it'd be a single thread...

:)




mikeshinn wrote:
Quote:
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?


mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory -
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.



1 - definitely running low on memory

2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...

3 - not virtualized - is a root server @ 1and1 here the specs:

dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)

CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5

5 - dunno - will look at that...

6 - no mod_security before ASL

7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group