The resolution should be a no-brainer if you have an LCD screen as they only run at one native resolution — any others will look terrible in comparison. If it's a CRT monitor (fat back), it's more a case of going as low as you can before it starts to look bad. The higher the resolution, the more strain on your graphics card.
Be sure to also have downloaded the latest drivers for your card from
NVIDIA's site. Drivers can make a
huge difference, and those supplied with the card are typically the first and slowest drivers available.
While you're at it, you'll want the
latest Psychonauts patch too.
Your hardware looks like it should be OK for running Psychonauts, although you may have to lower some settings. Stuff like anti-aliasing is particularly efficient at slowing you down, so make sure it's turned off. The only potentially weak point I can see is the graphics card — I ran the game pretty smoothly with comparable CPU and RAM, but that was with a GeForce 6600 GT.
If you're getting really bad stuttering and freezes with that hardware, there must be an external factor at play. It could be drivers, or it could be your machine overheating like you say.
Overheating
will cause problems, although hardware should turn itself off in that eventuality. Pretty much any CPU and RAM available won't allow itself to overheat, but I think only later graphics cards have shut-down safety measures. As such, older graphics cards like yours could theoretically melt themselves into an early grave if the fan stops working or something.