Virtual patching is an invaluable tool for immediate remediation to fix vulnerabilities in web applications. Atomic Secured Linux and the Atomicorp.com/Gotroot.com modsecurity rules contain thousands of Virtual Patches which we update everyday.

Sometimes you may need to patch a vulnerability in an application that we are not aware of, such as with a custom application.  This paper outlines exactly where and when Virtual Patching is appropriate, how it can be integrated into the Incident Response process, and how it can be integrated into the incident response process, and the proper steps for creating and testing real-world examples.

This article outlines exactly where and when Virtual Patching is appropriate, how it can be integrated into the Incident Response process, and how it can be integrated into the incident response process, and the proper steps for creating and testing real-world examples.

1. Commonsense tip one: Speed!

Don't get bogged down, a virtual patch is about time. You need to get that patch in place ASAP, otherwise, what’s the point?

2. If you can get the exploit, that’s a great way to test the patch for effectiveness - your application might still work, but did you fix the vulnerability?

If you can't get the exploit, you will need a lot of detail about how the attack works. If you discover a new vulnerability yourself, make sure you can test it against your new patch. And if you have nothing, then you will need to know how your app works. Get as much information as you can.

3. Exploit! = Vulnerability.

Just because the exploit didn't work doesn't mean your application is now safe. Try to understand what’s going on with your app. If two variables are vulnerable to a SQL injection attack, you may have a pattern with this in other parts of the application. It might be time to start writing some new patches for other problems. Where there is one hole, there are usually more. So be paranoid. Also, some products share the same code base. Take a look at your apps to see if the share similar variables, directory structures, libraries, etc. I've seen some shared PHP code, for example, that lead to the same remote PHP code inclusion vulnerability in well over 100 applications. Stay alert to the fact that a vulnerability in another application might be the advanced warning you need to patch yours. And, that brings me to the next tip, number 4.

4. Write multiple patches for different things

Don't try to cram it all into one regexp unless it’s easy for you and others to maintain. Yes, if you run thousands and thousands of regexp's you might experience performance issues but modsecurity 2.5.x and higher is very fast so don't get too hung up on performance.  Get the rules in place, and don't feel like you have to jam all your patches into one big obfuscated patch. Unless you are a regexp god, or you have serious performance problems, you really should stick with simple regular expressions and multiple rules and patches. There is always a need to evaluate your own organization’s priorities for performance and security. Sometimes performance is more important, other times it is security.

5. If you don't know how to write regular expressions, use a tool or find an expert

If you don't know how to write regular expressions, use tools like RegEx Coach, Expresso and others to help you debug and understand what’s going on with your regexps.  Your best bet is to seek the help of someone that does understand regular expressions to make sure your rules are doing what you want.

6. To that end, ‘Keep it Simple!’

Keep your patch simple and work your way up to complex. It’s much harder debugging a 5 line, chained, back referenced, nested regexp than a simple set of patterns. Start simple, and work your way up. Worry about speed, efficiency and perfecting your rule after you get your patch working. So, if it works for you, if your performance is acceptable, and you like it - then it’s a good patch. Don't get caught trying to make your patch "perfect". Make it good enough.

7. Make your patches easy for other people to understand and maintain.

Comment them so someone can know what the patch fixes (or does not fix), give it a Unique ID, version number and revision, and make sure you log when it fires so you can debug if it breaks something. Particularly when trying to go fast, there is a risk of overwriting changes without backing those changes up. Quick and dirty change tracking can be very useful in the process of rapidly developing a defense.

8. When writing a patch, decide what your goal is.

Do you want to define the correct behavior of the application, the behavior of the vulnerability, or both? Remember, only bite off what you can chew. If you don't have the data to define the normal behavior of your app, just stick with the vulnerability. If you don't have any information about the vulnerability, except some vague report of a SQL injection vulnerability in your products search function - it may be time to define the allowed behavior of that function. And sometimes, you need both. 

9. The easiest type of patch to write is to define the behavior of the vulnerability.

 It’s also a lot less likely to create a false positive and upset your users. The most secure type of patch defines the correct behavior of your application, and in the wild it’s best to do both. So, do both. Always remember, these two types of rules are not mutually exclusive, defense in depth is your friend - write more rules and write them for both cases (positive and negative rules,if you will).

10. Here’s the good news.

The vast majority of unstructured attacks stick with the script, literally. We see a lot of automated attacks that follow the published exploits to the letter. You can get a lot of mileage out of patches that just cover the known exploits for that threat.

(see tip 12 for how you can use this)

11. Perfect is the opposite of good enough.

Don't try to make your patch anything other than one that works for you. If it works for you, it’s good enough. If you want to publish it, please do - but don't let that stop you from writing an "ugly" patch. Any patch that works is worth something, anything else is useless. Also, patches do not have to be one size fits all. If you have to tweak the patch for a box that’s otherwise supposed to be identical, tweak it. Worry about why the boxes are not actually identical later.

12. Sometimes, the best you can do is to write a rule that’s just a tripwire.

For example, maybe you can't write a patch that works - it won't stop the attacks, or it breaks your app. What you can do is write some rules that detect the behavior or actions of the attacker before they exploit your app, such as rules that detect a recon probe for your vulnerable application (i.e., multiple attempts to find myphpadmin in multiple directories) or that detect a general type of attack (PHP remote code inclusion).

You can use tripwire rules to trigger other events, like a firewall rule to block your attacker before they can attack your vulnerable application. You can also write tripwires to fire on other vulnerabilities and use that  information to block your attacker. For example, the attacker tries to find a phpbb vulnerability, but you aren't running phpbb. That’s fine; just write a quick phpbb rule:

It won't hurt your box to detect that; anyone that tries to access anything associated with phpbb gets blocked by your firewall, and now all their other attacks fail (see tip 14). Is this method perfect? No, nothing ever is, but you would be surprised how many new attacks you end up blocking by simply setting some tripwires out there for older ones or general attack patterns, like PHP code inclusion attacks. (See the gotroot.com rules and the  modsecurity core rules for examples.)

13. Test your patch for both cases.

That means you have to test for both the vulnerability and whether your application still works. If you can't fix the hole, then the patch is just wasting cycles.

14. Evolve your signatures and rules.

Don't try to make them perfect, if you think a sig, rule or patch is weak, add a better version of it after your old tried and true patch – and run them both until you can prove you don’t need the older rule. Sometimes you  need to take a few stabs at a vulnerability to get it right. Don't be afraid to have overlapping and duplicate or similar patches/rules/sigs. Remember defense in depth, use it to your advantage.  And remember, humans make mistakes, especially us defenders, so be careful that you don't remove rules that worked before and may appear to not be working again.  Its much wiser to add a newer better rule and to leave the older rule still in place.  If you see the older rule still tripping (make sure you put your newer rules first), you'll know your newer better rule maybe isn't as good as you thought. 

15. Check your tripwires for new unknown vulnerabilities.

You may find some new thing that your attacker has discovered which you are temporarily blocking because of their carelessness in hitting your tripwires. Take that information and craft new patches.

Additional thoughts for MSSPs:

MSSPs often have little to no influence on the customer environment other than the security equipment we manage for them. Most of the time this is network oriented equipment such as IPS and firewall. Of course, MSSPs can (and do) advise customers to use layered defense. Unfortunately, security and server management departments within companies are usually several organizational layers apart.

What MSSPs can do is block things on a network level via the IPSs we manage for these customers. However, as most will know this is indeed limited (hence the effort with mod security). The following guidance is to help MSSPs setup a reverse proxy method to provide an additional layer of security.

• If you run a reverse proxy, start small and work your way up to complicated rules. You don't want to scare off your system ownersbecause you came on too strong. So, don't try to do more than just virtual patches for known vulnerabilities. Work your reverse proxy patches/sigs in slowly, and show value over time if your organization is reluctant.
• If you can narrow your rules to specific customers or domains, be prepared to do that. One size definitely does not fit all. To show value, generate reports on attacks. The ability to show not only if the proxy is working, but that if you remove it things will get worse, is only going to help you. Charts are a nice touch that helps to explain things quickly.
• If your customers are afraid of False Positives, don't try to deploy any rules, tripwires, or patches until you have adequate log data to analyze. False positives seem to be more important to some users than false negatives. Keep in mind, a false positive is an availability attack on their apps, so a false positive sometimes is worse. This is by no means to diminish the importance of stopping attacks. Recovery is usually more expensive than prevention - but this is not always true in real time. Understand your customer and save yourself some grief.
• To that end, start with just logging rules. It is a great way to test rules without breaking anything. (This should have been in my original list, this is by far the best way to test any new patch or rule.)
• Make sure your reverse proxy is not breaking your character encodings. This one can really bite you if you don't configure things correctly and test adequately. Keep in mind that character encoding problems are subtle, so ask your users to check their data carefully when testing. No one likes it when all the documents they just published had their punctuation marks changed to question marks because of a mis-configuration.
• Communicate with your system/application owners in advance about the risks and benefits associated with virtual patching and proxy based IPS’. The upside for them is that they get better protection, protected logs and, in some cases, more time to test the real patch (or even the opportunity to not have to install a patch, always a plus for complex systems.) The downside is that you will probably make some mistakes along the way and temporarily break their application - be honest and set their expectations in advance. And make sure you can commit to supporting their apps based on their support schedule. Make sure you know what applications your system/application owners are using. If they deploy a new app, make sure they tell you. It also helps to know what version the application is, and if it’s a customer application, what programming language(s) it is written in.
• Some languages have special needs, and vulnerabilities.
• Make sure you are subscribed to all the security lists for those applications. You want to know about the problem before your customer. After all, if you are or will be the one writing the virtual patch, shouldn't you be the one leading the charge?
• Set-up a rapid response process for false positives (see tip 7).

  If you have a ticketing system, make it easy and obvious for users to file false positives reports, and give them multiple ways to contact you. Your ticketing system might be getting blocked too. Your contact information, such as E-mail, phone, etc., is a good thing to give a user when you block them. Feedback, feedback, feedback. With any IPS it’s vital that, when you block something, you redirect the user to a simple and easy to understand page that explains what happened. A 403 error is not helpful to a non-technical user. A webpage that explains what happened, why it happened, and gives them contact information for the proxy/security team to "fix it," is helpful. If you can give them an alternative means to finish their task, that’s even better. For example, if you have an HR app that takes resumes and you block that process, provide an e-mail address for them to submit their resume, and if you are running moduniqueid, include that ID information on the error page so you can track down the specific event. Modsecurity logs this unique ID with each event, making it much easier to track down the event in your audit logs. 

• Last tip: Redirects are good for EVERYONE. If you can redirect a user to a webpage, as opposed to just firewalling them off, redirect them every time. Nothing is more frustrating to a user than a network blackhole. Blackholes are a great way to stop the bad guys, but it’s hard on your help desk and worse on your users. It leaves them both confused at best. A redirect is a great place for contact information, such as a link to a case management system and other self help links. Help your users to help themselves, and, in turn, make it easier on yourself.

And finally, purely technical notes:

• Make sure you know what phase you should be running your patch at. Looking in the wrong phase will cause you to completely miss the attack.
• Make sure you apply the right transform functions to the data you are working with to normalize it and to prevent the bad guys from evading your signature. For instance, some attackers will encode their payloads to avoid some signatures and rules. So, for example, if you work with hex data you need to make sure you are transforming that data into the format of your rules (usually ASCII). However, if you are not using hex (or some other encoding format) you may want to just write a rule to detect it and block all those cases, as it is most likely an attack.
• Anchor your regular expressions, and use advanced functions to improve the performance and accuracy of your rules. Try not to bite off too much at once though; if regular expressions are not your cup of tea, work things in slowly so you can teach yourself what the effects are and test your changes methodically.  Be paranoid with your rules. Bad guys will try all sorts of things to get around them. For instance, if you have a rule to look for "<iframe>" and the bad guys changes his attack to "< iframe>" (note the space), and your rule(s) don't allow for that space you can still be successfully attacked.  So think like an attacker, and be paranoid about your rules.  What would the bad guys do to get around your rules?  Then ask someone to test them!
• Which brings us back to testing.  When testing, it is always best to ask someone else to double check your work.  It's much harder to catch your own mistakes.
• And last, don't forget to define the actions you want your rules to take.  If you want to deny an attack, you need to configure your rules to do that.  If you have them set up to log only, don't forget that they won't stop anything.