store | blogs | forums | twitter | facebook | wiki | downloads | support portal
Atomic Secure Linux
It is currently Sat Aug 30, 2014 12:10 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 12:34 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
installed asl 3 (30 day trial) onto a 1and1 cent os single core w/plesk 10.3.1 yesterday and the load average gets incredibly high making the machine slow to a crawl...

oddly enough I don't see anything using a really high % of the processor in top... restarting apache seems to help for a little while but within a few minutes it goes right back to "way too high"

already added /var/www/vhosts to the ignore list thinking the scan was just taking to long but that didn't do anything to help

What else should I be looking at?

thanks in advance - if I can get this under control I'll definitely become a paying customer!


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 4:40 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3626
Location: Chantilly, VA
Can you tell us a little more about the system, is it dedicated or virtual, CPU/memory/IO, etc.

Also, whats the memory utilization on the syetm, I/O on the system, and whats atop telling you are the threads that are really working your CPU?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 5:35 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)

CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5

io sucks - the partition for /var seems to be be causing a bottleneck (turns red in atop)

|
PAG | scan 3640 | | stall 0 | | | swin 204 | | swout 178 |



LVM | vg00-var | busy 95% | read 976 | write 36 | KiB/w 5 | MBr/s 0.66 | MBw/s 0.02 | avio 9.34 ms |


btw - have never used atop before seeing it on here - way cool

- and the PAG line above is red right now- without modsec enabled but once I turn it back on the server load climbs right away (went to 114 before - I didn't even know it COULD get that high!)

There was a LOT of httpd's spawned (cut the numbers of children it could spawn down to 20, was 40 prior, and also cut the request per child down from 4000 to 2000) neither helped


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 6:28 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
moved this from viewtopic.php?f=1&t=4362&p=32524#p32524


mikeshinn wrote:
Quote:
I am thinking I am running into high loads due to apache spawning way too many children, that being said - what's the deal with rewrite rules in .htaccess?


mod_rewrite rules don't have anything to do with ASL (so there is no conflict). Poorly written rewrite rules can kill your system, they can cause loops, wasted cycles, etc. rewrite rules can be VERY cpu intensive.

If you disabled modsecurity and that helped your load, that sounds like you may:

1) be running low on memory -
2) if load without mod_security is already at 1+, then your CPU is overworked already. Whats the hardware on your system? It sounds like you may already be saturating your system.
3) If the system is virtualized, you actual CPU utilization may be limited (and in such a way that you cant see it) - this can have horrifying effects on performance
4) if you have modsec enabled before you installed ASL, you may have a very inefficient configuration - were you running modsec before you installed ASL?
5) Did you enable any RBL or URI modsec rules? These require a fast local DNS server, if you don't have that your load will suffer accordingly
6) recompiled modsecurity module - some products, like cpanel, may replace the optimized modsec module ASL installs with a non-optimized module. Are you using cpanel or a third parties modsec module?
7) are you using a compiled apache, or an optimized apache provided by your OS vendor? If the former, us your OS vendors apache build. Source builds are hard to get optimized correctly.



1 - definitely running low on memory

2 - runs a bit above 1 w/out mod_sec - you are right about that - system is over loaded...

3 - not virtualized - is a root server @ 1and1 here the specs:

dedicated single core root server @ 1and1 w/CentOS 5 with 1 gig ram, Parallels Plesk Panel 10.3.1 (64-bit)

CPU AuthenticAMD, AMD Athlon(tm) 64 Processor 3500+
Version Parallels Plesk Panel v10.3.1_build1013110726.09 os_CentOS 5

5 - dunno - will look at that...

6 - no mod_security before ASL

7 - it's using the apache that came with plesk 10.3.1: 2.2.3-53.el5.centos


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 6:36 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3626
Location: Chantilly, VA
1 GB of RAM may be too low for your system, and a steady load of 1 on a box without mod_security running generally means the box is doing way too much work already. It shouldnt be that high, that basically means there is always at least one process just waiting for some resources to do some work (not a good thing with a modern fast machine, that means a LOT of work must be going on). I bet whats happening is apache is swapping, can you see how your system is doing with paging in and out?

The WAF will use memory, its how it does it job quickly. With a system with that limited amount of RAM you may need to disable clamd, and disable the spam and malware rules. The caching rules (spam, malware, etc.) use a parallel search method for speed and trade RAM for speed, they may be beyond your systems current usage if its already got a load of 1. So try disabling them and see if that helps. But if you are already hurting on RAM and your load is always at or above 1 without a WAF, you may need to get a faster system. It sounds like it may be just at its limit now.

Now the /var i/o issue is interesting. If its high with modsec disabled, do you have both your logs and websites on the /var partition? You could be running into a perfect storm of i/o (file is accessed, apaches writes a log entry when that is accessed, atime is updated). Is anything else going on in /var, like some app that writes a lot to that directory?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 6:47 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
turned off most of the ASL bits and pieces using some of the directions @ https://www.atomicorp.com/wiki/index.ph ... ble_ASL.3F and the server load is back to normal .25, .19, etc

no red LVM entries in atop either...


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 6:55 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
there's tons of activity on /var - mysql db's, domains, logging, and more is on that partition under plesk

right now it's running fine (still under .50 for the past few hours) but everything ASL related is turned off


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 6:58 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3626
Location: Chantilly, VA
Could you be more specific about what you turned off?

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 7:18 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
yes I can - turned off clamd and all the "stuff" inside ASL Configuration (via web interface)

ALERTS_USE_DB no
CLAMAV_ENABLED off
PSMON_ENABLED no
OSSEC_ENABLED no
MODSEC_ENABLED no
PHP_CHECKS no
SSH_STRICTMODE no
RKHUNTER_ENABLED no
MODEV_ENABLED no


load average was a little high still but not stupid high, stayed around 2 ish

- then I killed ossec-analysisd and everything went back to normal

tomorrow I'll bring mod_security back up and start with that...

I think it's really a matter of it being a low end server


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 7:41 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3626
Location: Chantilly, VA
Quote:
- then I killed ossec-analysisd and everything went back to normal


So that may explain things a bit, analysisd is the engine that reads your logs to figure out if attacks are occurring or if suspicious things are going on. If thats causing load, then its means a few things may be occurring:

1) You might just have that many logs (especially with the high I/O), unlikely for a normal system but I've seen some cases with huge logging going on from third party apps (splunk for example)
2) You might have a really insanely large log file that its reading (it doesnt care, it will read it, but a huge log file is a lot of work). Check to make sure you have log rotation setup for all your logs, you might have something big that should have rotated that you really don't need.
3) I/O bottleneck on /var. ASL stores its log in /var, the system does too, generally mysql does this as well and some control panels also put their websites in /var - so in general an awful lot of works goes on in /var. You likely have some serious contention going on.

I'm not sure why OS and control panel vendors do this, on our system where performance is key we like to put mysql on its own partition (and on its own server if we can), the web sites on another partition and the logs on their own as well.

Another trick is to disable atime on /var. That will speed up throughput and descrease latency on the parition as the system doesn have to record the access time everytime a change occurs (like with logs):

https://www.redhat.com/support/wpapers/ ... uning.html

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Wed Dec 28, 2011 8:29 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
thanks for all your excellent info - will delve into this again some time tomorrow and see what's what on /var

afaik - I don't have anything "special" on that system that would add additional logs and there's only about 40 active domains on it right now and of those maybe 4 are reasonably busy...


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Thu Dec 29, 2011 6:34 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
haven't had time to revisit any of this yet - one of my other 1&1 servers was compromised last night and it took a little while to find out what happened...

(there was a file uploaded into a wordpress theme image cache that was in fact not an image)

turns out ASL 3 works great on that machine! (it's a dual core amd w/2 gigs)

ASL Active
Anti-Virus Active
IPS/IDS Active
Kernel Protection Active
WAF Active
DoS Protection Active

I let asl do an initial scan after I had cleaned it out and it didn't find anything as far as I know, /root/asl-malware-scan.log is empty

is that the right file to be looking at?

:)


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Thu Dec 29, 2011 7:29 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin

Joined: Wed Dec 31, 1969 8:00 pm
Posts: 7890
Location: earth
Yup, thats the one it will run right after an install.


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Thu Dec 29, 2011 8:37 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
excellent - it's nice to know the machine is clean AND currently protected by ASL's full arsenal!

decided to upgrade to something better so I know it can handle ASL instead of futzing around with getting it to work on the single core...

btw - the coupon you all had posted on facebook for the 35% off - is that going to be valid when the 30 day trial ends?

it was that facebook post that pushed me into trying asl - your social marketing efforts will at least gain you one new customer

:)


Top
 Profile  
 
 Post subject: Re: REALLY high server load average (like 30 - 40!)
Unread postPosted: Thu Jan 05, 2012 2:15 pm 
Offline
Forum User
Forum User

Joined: Tue Dec 27, 2011 12:27 pm
Posts: 29
Location: Golden, CO
So I installed ASL onto a new quad core w/4 gigs 'o ram and aside from 2 random issues I am digging it!

In the ASL GUI - when you click on an event in the event details window - the window it opens has an old message listed AND it's the same one for every message...

And I have also had 2 seemingly random crashes - where is the best place to look for what made the server crash?

Thanks in advance!

-Bill


Attachments:
File comment: this an example of what I see when I click on an event... Any ideas? :)
capture146.jpg
capture146.jpg [ 133.44 KiB | Viewed 1796 times ]
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group