Upgrade your RAM first, but it's only really going to help if you're on a 64-bit system running a 64-bit OS (and 64-bit apps?). RAM will help ZBrush a lot, it's basically all it really seems to care about, and it will eat it up as fast as you can plug it in.
It may be some issue with Vista 64, i know its been used on XP 32/64, and vista 32, but not sure about vista 64. That video card should be great for it, so really unless its a video driver issue i cant tell you why it wouldn't be working.
Just a note for anyone running a 64 bit version of Windows (particularly Win7), go to the Java web site with the 64 bit version of IE and grab the 64 bit version of Java. I don't know the technical aspects of it, but doing that has made the game run better through multiple revisions.