Hi,
I am wondering if there is a maximum of lines that a file invoked in a rule can have, for example, the file malware-blacklist.txt, How many items can hold?

As many as you like, but there's a panalty in system load and memory usage.

In our case (and many others too), too many line/rules = apache segfaulting.

Faris.

Thank you Faris,
but it seems there is a limit of some sort (at least in my servers), I have tried with a list of 4,000 items and there was no problem, then I increased the list to 7,500 and then the rule didn't catch the items in the list.

Sergio.

And no segfaults in /var/log/httpd/error_log or rules processing failure in any domain's log?

Try splitting the file into two, and load them separately.

If that works, try loading the two files in the reverse order. If that doesn't work then there may be a some strange character within the file you are using that causes the problem or something like that)

The full malware-blacklist and domains-blacklist files used to be huge.
The malware-blacklist by itself was nearly 90K in size, though obviously that's roughly (or exactly?) equal to the number of characters not the number of lines.

I don't recall how big the domains-blacklist file was, but it was also huge.

Faris.

You should be able to do over 40,000 on a regular centos 5 32-bit box w/ 1 gig of ram no problem. We do that and far more on our systems. Cpanel boxes probably wont be able to do that much given the way they build things.

Yes Scott, that is what I tought, that maybe Cpanel has some limits.

Scott, how RBL works with modsec rules? Does the RBL rule uploads everything from the RBL site to the server memory? or the rule searchs the RBL list every time someone connects to the server?

I ask this because if I can't use @pmFromFile with so many IPs in it, maybe creating my own RBL list could do the trick and the best is that I could have just one centralized file for use with different servers.

Sergio.

The RBL is going to do a lookup on each new IP that connects to the system. In terms of the total number of entries, its orders of magnitude higher. The limitation there is the number of entries storable in the named cache. The cost here is that you add latency for the time required to do that lookup, so you want a good fast nameserver.

So, @RBL will not work as intended, thanks for your input.

I will try to check on cpanel if they can check on this.

Sergio.

Well it works as intended Its just going to add some latency to the system, depending on the speed of the nameserver.

Finally I found the error on my file, thanks to Brian Rectanus.

There is a bug that will be fixed on the future modsec 2.5.12, the error is that the file that I uploaded to the server had MSDOS carriage returns instead of UNIX, so, using the command DOS2UNIX from shell it removed and replaced the CRs and that did the trick.

Sergio.

Thats pretty much a universal UNIX vs DOS issue. Make sure you always run dos2unix when you copy over a file you edit on a Windows box, only Microsoft adds control Ms to the end of every line and it breaks on every platform except Windows. That might not sound like a big deal, but given that ever OS on earth works the same way except Windows its pretty annoying that they do this with TXT files.

"Live to learn" as my grandmom saids, lol, and it always has been true.

Sergio.