store | blogs | forums | twitter | facebook | wiki | mailing lists | downloads | support portal
Atomic Secure Linux
It is currently Thu Apr 24, 2014 6:50 pm

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: malware-blacklist.txt - a memory hog?
Unread postPosted: Sun Nov 28, 2010 10:42 am 
Offline
Forum User
Forum User

Joined: Sun May 29, 2005 7:27 am
Posts: 15
I recently found out that my httpd processes use a lot of physical memory. Each process use 141 MB (!) at startup, which is way above normal. After some investigation it was clear that mod_security / malware-blacklist.txt was the source of the problem (malware-blacklist.txt is apparently used in 10_asl_antimalware.conf). Emptying this file saves me 90+ MB for each httpd process, which is a lot.

So, is this an expected behavior or a bug? A memory leak? Maybe some kind of misconfiguration?
malware-blacklist.txt looks so innocent (only 164 kB in size), so something must be wrong?
A server with 20 httpd processes will eat nearly 2 GB of memory just because of this.


Top
 Profile  
 
 Post subject: Re: malware-blacklist.txt - a memory hog?
Unread postPosted: Sun Nov 28, 2010 12:02 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
Its not a bug, its a byproduct of how parallel searches are done. The malware domain list is around 10K entries long, if we did a normal search of that list the process would slow down the webserver (thats how it used to be done back in the modsec 1.x days) but wouldn't use much memory. Now we do parallel searches which are lightning fast, and require more memory. So its a trade off, speed for memory, or memory for speed. So, if you don't have enough memory then you probably shouldnt use the antimalware rules.

Given the night and day performance enhancement parallel searches gives us, its the only way to do lookups in milliseconds on large lists. Any other method would kill the server. Most people don't have any issues with this ruleset memory being pretty cheap these days. So if memory is an issue for you, then you won't be able to use the antimalware blocklist.

A box with 1.5 GB of memory will do just fine.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
 Post subject: Re: malware-blacklist.txt - a memory hog?
Unread postPosted: Mon Nov 29, 2010 5:42 pm 
Offline
Forum User
Forum User

Joined: Sun May 29, 2005 7:27 am
Posts: 15
Thank you for the explanation, very clear. I generally prefer speed over memory (and absolutely love the word parallelism :) ), so it sounds like a good decision.

Memory is not a problem under normal load. I'm more concerned about what will happen under heavy load, when apache goes wild and forks a lot of processes.


Top
 Profile  
 
 Post subject: Re: malware-blacklist.txt - a memory hog?
Unread postPosted: Mon Nov 29, 2010 6:54 pm 
Offline
Atomicorp Staff - Site Admin
Atomicorp Staff - Site Admin
User avatar

Joined: Thu Feb 07, 2008 7:49 pm
Posts: 3548
Location: Chantilly, VA
I havent seen anything untoward happen under very very heavy load, and havent heard any reports either - so you should be good. We're working on some enhancements to the parallel matching that might be able to cut the footprint by 33-50%. It might be possible to cut it another third, but I need to do some testing of a complex branching method first.

_________________
Michael Shinn
Atomicorp - Security For Everyone

Co-Author of Troubleshooting Linux Firewalls.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

» Feed - Atomicorp

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group