On my Mac os X 10.6 machine at work I’ve suffered from slow scroll speed using vim with syntax highlighting in Terminal.app. The screen refreshes very slowly when using colored syntax highlighting in vim and only in Terminal.app (xterm is fine, but I don’t like to have to start x11 and all just for a terminal, and anyway Terminal.app is fine otherwise).
Apparently something got broken during Terminal.app version 2.0.2 and Terminal.app version 2.1, the latter even being 64-bit. Who knows what it was but I can find any reason to use the new Terminal so I will use the old one on the new machine.
I’m backing up my old copy of terminal, but you should probably find your own to be safe. The scroll speed works fine with the old Terminal.app. Just drag it over from an intel machine and it should work find. All my bash profiles and vimrc loaded just fine, so I didn’t need to set anything else up.