Do you need swap? Yes. Without swap you could have bad things happen. How bad? Well, consider the comic on
this page, except that malloc might not be so nice. Swap exists so you can pretend you have more RAM than you have. The cost is performance. Your hard drive is a snail compared to RAM's souped-up motorcycle, so storing RAM to disk is not the best idea but trying to allocate RAM and having your memory manager go "Sorry, all out" could cause mayhem (I don't know exactly what would happen under Linux but why taunt happy fun ball when you don't need to?).
In a normal Linux system, you run pretty close to full usage for performance reasons (being that RAM is so fast and usually abundant). Your virtual memory manager will offload less used blocks of memory to make room for other things that need it short term. This keeps your RAM fresh without needing to destroy anything. A small amount of swap usage is to be expected on any machine.
I would say 512MB - 1GB of swap should be more than sufficient for most systems. A full swap means you've got problems.