Our move to EC2 progresses but I'm now faced with a realization that ASL may not be able to follow us. I'm still fiercely loyal to ASL but I'm facing some serious challenges with it and I wanted to voice them so that maybe they can be fixed (if they are indeed fixable).

The first problem is that using an Elastic Load Balancer pretty much destroys the usefulness of OSSEC. EC2 isn't without its quirks and one of those quirks is that your EC2 instances don't have a routing IP. Your box is assigned an internal EC2 IP so you can talk to other AWS instances for free (sort of a VPN). This makes some things that are simple (like running multiple IPs on a box) a real challenge. The problem with the ELB is that it talks to you through the AWS network. External IPs are fed to you in the header. OSSEC, of course, doesn't know what to do with this and winds up banning the ELB. Since it would take some work to make OSSEC work with the headers (which can be easily spoofed, sadly) we had to drop it.
http://osdir.com/ml/ossec-list/2011-05/msg00171.html

Second, it doesn't appear the kernel can be used by EC2 in its current format. I think this could be done with some changes, though.
http://aws.typepad.com/aws/2010/07/use- ... n-ec2.html

Third, for some reason modsec rules are choking Apache. I've opened a ticket on it but I've never had any issues loading modsec (had it running on a C4 server with the same resources roughly).

Has anyone else used ASL with EC2? Am I doing something wrong?

as luck might have it, I've been working on another project using EC2, and Scalr.

So here are my highlights:

1) If there is any way you an NOT use their garbage kernel, do it. Its broken it all sorts of happy performance annihilating ways. evenfd() is non-existant for example
2) They have their own pretend version of centos called "AWS" or something. Avoid at all costs. I expect them to abandon it in a year, leaving you without a maintainable OS.
3) Is the ELB nginx? It is in scalr, in the example we're working with you can clean up the IP issues pretty easily with mod_rpaf (in atomic now!)
4) You already ran into the issue with IP addresses that we did. You can only have 1 IP on the system which makes multiple SSL certs difficult... but their is a fix! Dont use their load balancer. Go get a normal box somewhere else (or two,or three), run nginx on that, and put all your IP's & Certs there. You can redirect from the LB back to your cloudy boxes.
5) scalr is open source, and we're setting up our own private infrastructure for that now so definitely check that out if you haven't already. They have their own site (scalr.net) where you can manage your ec2 systems from, pricing just changed which is why our client asked us to set up a private one. Its considerably cleaner than the amazon interface.
6) Amazon is not magical unicorn performance dust. Its nothing more than their flavour of xen. My thread on performance benchmarks here viewtopic.php?f=1&t=5576 was inspired by how poorly it performed in general, so I've been looking into how to get more out of it. For the record the BEST Ive been able to squeeze out of a medium (2G) amazon system running Magneto was 4 requests per second, thats after tweaking... and it started at .27/second. Compare that against the forum server you're reading now, which does over 200.

I found pretty quickly, using the supplied AWS images wasn't going to be a good/manageable deal long-term. We went with a pre-built i386 CentOS base install someone else had made and I customized it (installed all the Ruby stuff Chef needed along with a full upgraded to CentOS 5.7 -- it's only a base install, so there isn't anything more than this) and then I made my own AMI out of it that increased the EBS space from 2GB to 16GB.

As far as kernels go... I don't know a lot about them so I used the one that originally came with the CentOS AMI I used. For a base install, it seems to be doing ok for the traffic I'm sending to it. Albeit, I'm not expecting the same performance out of it as I have on our current dedicated box. However, there is a speed increase due to our database using RDS -- our primary reason for going with EC2.

I can give you access to the private AMI... there shouldn't be passwords or keys hanging around, but just in case, this is why I haven't made it public. If you're interested, PM me your AWS Account Number.

Finally, on to my main issue...

The issue I'm having is that it's impossible for me to run mod_security on the instances without it running a high load. I ran with all the base rules installed and then set "SecRuleEngine" to "Off". Even with it off it ran very high CPU, 1 min averages was at 2.5-3.75 whereas when I completely disabled mod_security they run anywhere from .4-1.5 (this is with medium load, probably about 5-25 hits per second). When it was completely enabled I was running anywhere from 7.5-10+. I did nothing different, and Joel (highland) even attempted a fresh install on another instance and the results were identical.

The configurations were the same as they were with our dedicated server (with less rules though), except we're using i386 on EC2. I was thinking that could be an issue (x64 works, but i386 has issues), but all our packages for Apache2 and mod_security are coming from the atomic repo, and I'm not sure that could be an issue.

Like I said earlier, it appears that theres a much deeper problem with the amazon system. Not even running mod_security on the system and the performance is worse than a comparable KVM instance by orders of magnitude. That shouldn't be happening with just using xen as a virtualization layer. My guess here is that its the kernel (what are you using?) but thats just a guess at this point.

I am using kernel ID aki-f5c1219c (Linux ip-10-144-65-60 2.6.18-xenU-ec2-v1.2 #2 SMP Wed Aug 19 09:04:38 EDT 2009 i686 i686 i386 GNU/Linux).

Source should be here:
http://ec2-downloads.s3.amazonaws.com/x ... 2-v1.2.tgz

thats just the xen source, not the kernel. If they're really only using 3.1 then a lot of the more important features (eventfd like I mentioned earlier, this was added to 4.0) arent in there. I suspect that has a lot to do with the performance problems.

Heh, I was asking magneto about this too. They responded with:

"We noticed that you are using AWS, Please explain why you chose to do so."

Clearly they've had the same performance issues with it that we have.

If I understand your issue, OSSEC isnt the issue - its just being told "this is the source". Thats going to happen with any proxy in front of anything, the logs will always report the "wrong" IP. The solution is to install a module like RPAF for applications like apache, so that apache will correctly report the REAL source IP by using the X-Forwarded-For header and not the source IP from the load balancer:

http://stderr.net/apache/rpaf/

[rpaf] changes the remote address of the client visible to other Apache modules when two conditions are satisfied. First condition is that the remote client is actually a proxy that is defined in httpd.conf. Secondly if there is an incoming X-Forwarded-For header and the proxy is in it's list of known proxies it takes the last IP from the incoming X-Forwarded-For header and changes the remote address of the client in the request structure. It also takes the incoming X-Host header and updates the virtualhost settings accordingly. For Apache2 mod_proxy it takes the X-Forwared-Host header and updates the virtualhosts



It should work. We're looking into this for a customer already.



I don't think its modsec. As Scott mentioned we are working on a project with the same platform and ASL (including modsec) isnt even installed and in our case Magneto cant even scale in this virtual environment, it just grinds to a halt. Even a simple "hello world" php app doesn't want to scale, we get orders of magnitude faster performance on non-AWS systems with the same RAM, OS, etc. So we can reproduce this without ASL or modsecurity. What we havent been able to do is figure out why their environment seems to perform so badly.

Something is very wrong in the AWS universe, and its not modsec (in our case its not even being used on those systems). Something else is throwing off performance in that environment massively.

Every install of Magento I have ever done has always been painfully slow -- one dedicated, one in a virtualized Hyper-V machine. IMO, you have to have a fantastic server just to run Magento... unless something's changed in the last four months...



While performance isn't going to be nearly as good as a dedicated server, I've not experienced any major issues on my small instance. Albeit, I didn't do any major benchmarks, but we're running WordPress and custom software and I haven't had any performance issues that I didn't expect (expect the modsec one).

Best case, we could get the AWS after upgrading to an 8G instance with all the optimization I could muster to do no more than 4/reqs a second. So in other words changing from a 2G/32-bit instance to an 8G/64-bit made no impact. Basically the result is that when they have to handle real world traffic they end up scaling to 30-40 servers, and that gets expensive fast.

Ive got great general metrics otherwise. For static content you'll get about 30% of a similarly sized virtual system. 2200/reqs was the KVM 1G instance baseline, AWS 2G and 8G performed at 600/reqs. Again ram made no difference in that test. Also we excluded the load balancer (nginx), when testing that directly that would get up to 1600/reqs on static content, but in any kind of interactive code it didnt make an impact and the results would drop down to 4/reqs.

That alone is a pretty interesting result, the load balancer (nginx) is considerably faster at handling static content than apache is. Is it that nginx is just better in general at static content, or is there something going on with the apache environment that is causing a bottleneck?

Hmm...

We've used BrowserMob in the past for some load testing and I just did a basic test on our EC2 instances which resulted in 189,182 hits over the course of an hour to a page that has about four queries to our primary database (an RDS instance) along with session storing done on a separate EC2 instance (both instances & RDS are small in us-east-1b; uses memcached & mysql) and got fair throughput, though, obviously, the server was heavily loaded. Currently, we don't get even close to this amount of hits over an hour period across our the entire network.

Our highest avg load time was 1.16 secs, but that was with a chef-client run (checks in every 30 mins or so)... our average was about 700-800ms. I believe this is with about 50 hit/sec. Server load at that time was very high though (22.82, 25.25, 21.96). There wasn't any major noticeable speed issues though on any of the sites we host. Of course, this is without ASL. BrowserMob uses EC2 instances for its tests, so these were done in the same general area of our EC2 instances.

I am running the exact same test on our live server to compare load and speed. Speed is going to be a bit slower just due to our database being at RDS while the web server is in Houston. This server has 8GB of RAM and a hyper-threaded quad core though, so it's not at all a "fair" comparison. Also, in this scenario the session database is on the same server. Edit: the load test is over and the average time was 900ms-1sec load. That's not bad at all for MySQL lag with RDS. The server load was pretty high (16.10, 13.95, 11.34). This was running ASL though, so there's a fair amount of difference there. It did have 70K less hits though with an ending hit count of 117,156. This is somewhat to be expected since throughput is a good bit less.

I can share any of these results if anyone wants to look at them along with CloudWatch data.

As a side note, since Magento uses it, while our network uses parts of Zend Framework, we only use and load bits and pieces. We've found it to be very slow, partially (maybe largely?) due to autoloading. Out of curiosity, how is your disk usage in your tests as reported by AWS CloudWatch?

Lets try to use the same standards here to consolidate the results, this is what I'm using to measure the system:

ab -n 50000 -c 100 http://site/somephp.php

A good simple test script would be just to have it invoke <? phpinfo(); ?>. If you've got something more complicated try that in addition to the above. Also take note of the load on the instance while this is running, and if you get any dropped sessions.

When I run that against the AWS I get around 400/reqs (that is against a c5-64 8G instance). The load will hit around 40-50.

That's roughly what we were seeing. BrowserMob actually stresses our platform application by making 50 concurrent users that access it like a client would. We were seeing loads pushing upwards of 30 but, oddly enough, no real slowdown in the EC2 Apache response. That was using a db.m1.xlarge RDS instance for MySQL and a m1.small EC2 instance running Apache (without modsec).

See what you get with ab

Hmm, have you tried the c1.medium instances? My AMI only works with small and medium, so I decided to try medium (not realizing it wasn't part of the standard EC2 offering) and it performed well, at least compared to the small instance. It is only slightly more expensive, too.

I got a five min load average of 12.40. This is using just a basic <?php phpinfo() ?> script.



I ran these tests from a small instance so they were done semi-locally in the same us-east-1b zone.