Today was all about nailing down the 2.6.32.19 kernel update. Upstream** made quite a few changes that believe it or not were effecting ioncube loader from the kernel side. I know I mentioned this before, but this is exactly why building community packages pays off for our security products. Its like the ultimate QA process smoke test.

Anyway, that was cleared up as well as some of the internal changes that need to be made to support dracut in RHEL/CentOS 6, and Fedora 11+. Leaving only one of the sillier problems to deal with, which is how to version it. I'm sure you've probably noticed other packages using "<package name>-<version>-<release>.art.el5.<arch>.rpm" by now, with the "el5" part indicating the distribution it was built under. Until now we've only ever had to build one kernel for all platforms, and we could get away with simple basic naming conventions. Furthermore kernels are more or less agnostic to whatever you have installed in userland, it doesn't matter if you build the kernel on fedora 13 and run it on centos 5. Until now that is...  still pondering the most elegant way to do it. Presumably someone will ask either internally here, or externally so I'll list why we won't be doing it per distro:

• Per distro kernel builds don't work.
• If they did work, they would take 15 times longer than they do now.
• That would equal about 2000 minutes total, or about 33 hours (assuming nothing goes wrong).
• Per distro kernel builds don't work.

Which leads me to other updates, since kernel builds do take so long even with the current design I took a diversion into implementing FPM in the PHP 5.3.3 packages. I looked around fedora to see if they had tried to do this yet without any success. Perhaps its even too new for them. At any rate, the good news is that it was possible to implement as a module for CentOS/RHEL 5 called php-fpm. The bad news is it will not be possible to support this with CentOS/RHEL 4, there are simply too many changes that would need to be made in other packages. There were no issues getting this implemented in Fedora 11, 12, and 13.  We still have no method of evaluating the implementation of this package, so input on what works with this is just as important as knowing what doesn't.

If you'd like to give FPM a shot, you can install it with:

yum --enablerepo=atomic-testing install php-fpm

** Upstream is packager-speak for whoever owns the project code you're working with. Linus Torvalds, Mysql.com, php.net, etc.

In an ongoing effort to reduce overall ASL complexity, we've been working to expand the capabilities of one part or another where they overlap. This summer I gave our intern the project (among others!) to see if he could merge the rkhunter rootkit signatures into the ossec rootkit database.  I'm happy to report that not only did that project succeed, but that we also have a great utility to handle synchronizing rules between the two projects going forward.

So, on to why this is important. For starters  rkhunter is a great project by Michael Boelen, written entirely in shell. As a long time unix guru I thought I had seen everything that could possibly be done in shell before I saw rkhunter first hand (and check out his other project lynis). Its a poem. If you need a shell example cheat sheet, rkhunter is it.

Daniel Cid the author of OSSEC mentioned once that rkhunter was part of his inspiration for his project, but that he needed something more client/server centric to handle a large number of machines.  OSSEC has too many traits to appreciate in this narrative, so I will digress to the bullet form:

• Client/Server system
• Trigger events based on alerts, ie: malware detected in /path/to/file
• Alert suppression, probably not a good idea when we detect the system has been owned. That being said, rkhunter will harass you daily. OSSEC will not :P
• Whitelisting, if for example there is a false positive. You can keep yourself from being locked out.
• The analysis also allows us to look at running processes. Perhaps we can combine file checks with some more advanced process based meta rules. We're still looking into this.
• Ossec supports inotify**

** This is the real reason. Inotify is a kernel subsystem that allows you to trigger an event based on a change to the file system. It moves malware detection into the realm of real-time analysis. Whereas rkhunter is a tool that scans based on a direct userland event (cron, running it manually, etc), inotify events are immediate. This means that we only look at the specific file when it changes, resulting in both a vast reduction in overhead  and an immediate response.

So what does this mean for rkhunter in ASL? For now business as usual, but if we can duplicate the capabilities completely in OSSEC then it would eliminate that in our design.

I'm also happy to report that the OSSEC project has accepted our rootkit updates, and it will be showing up in future releases.

So this saturday I found myself ultra-motivated to get this project back into gear. As a security company, supporting PHP and Mysql for enterprise environments isn't exactly our core business. Which begs the question why we get involved in non-core projects like this (some might call this a "distraction"). I've found over the years that the raw internal experience with these kinds of applications puts us right in the path of the same kinds of problems anyone using ASL every day. After all you're running a business that needs to be secure... not a security business. And who better to secure your environment than an outfit that uses the same thing you do.

Anywho... its been a challenge to get all the moving parts lined up across various operating systems. Some notable issues so far:

1) ncurses, mhash, and dbase modules are gone in 5.3.x. Mostly this means a smooth upgrade using "Obsoletes" tags, but I'm still unclear on how mhash is supported.

2) I found a bug in mysql 5.1.50. They left out an include file, hooray! Its called my_compiler.h, and if you try to link against it, it will break.  This is one of those situations where linking against a native mysql (5.0 or 4.1) saved the day on the enterprise distros. Just to give you an idea how long this takes, I didnt have a problem until 2 days into the project when I hit the Fedora 11 and up builds.

3) Did I mention its taken 2 days to get this far? This is a pretty major change.

4) Zend Optimizer is *not* available for PHP 5.3. After 5.3 has been out for over a year, so either the demand isn't there for it or there is some major barrier to doing it.

5) Ioncube does support 5.3.3. You're in good shape there.

6) So does eaccelerator.

7) Firebird / Interbase database supported. It took over a day to compile this on our new build server (i7-930, overclocked to 3ghz, triple channel memory etc.). In comparison this system will compile the linux kernel used in ASL in 14 minutes. Watch it never get used.

8) enchant and some other internationalization modules have been added.

9) readline has been replaced libedit. There are apparently license issues with readline that I did not investigate further.

10) People have asked about FPM. I did nothing on this, since I have nothing to test it with. Its going to need further research.

11) suhosin, havent started on this.

12) sqlite needed to be updated to 3.7. Im unsure as to what effects this may have on other applications.

13) This is linked against the native mysql libraries. Theres a good reason (see item #2)

PHP 5.3.3 is available from the atomic-testing repo, you can upgrade with:

yum --enablerepo=atomic-testing upgrade php