My biggest question: How do this affect the performance of both the web server and email server?
Mod_security is an Apache module, so it won't affect your e-mail server.
I have heard having too many mod_sec rules will mean a performance hit. I get HUGE traffic so want to be very careful.
As long as a server is not short on RAM we haven't seen any obvious performance problems, but I don't know what you call HUGE traffic, so I guess you just need to go and do some load testing. Or maybe start with just a small number of rule sets and enable more from time to time to and see how it goes.