I have a sneaky suspicion that I'm out of luck for VESA compatibility in DOS with a Voodoo5, but I'm just wondering if anyone knows for sure.

For example, TES Legend : Battlespire  uses high colour 640x480 VESA mode, and will not work on V5 but works just fine on an old ATI Rage (with Mach64 vga bios). 

Did 3dfx ever release any type of VESA patch or driver for its 2D/3D chips? If not , then I'll be forced to finish the cooler for my Obsidian2 X-24 (and yes, I do know what the Quantum cooler looked like and how it was installed) and pair it up with a true VESA vga.

Find Voodoo Heaven here ...
http://www.thedodgegarage.com/3dfx/index.htm

The actual error that I get is "Cannot set 15 bit 640 x 480 mode"

Sounds like the 3.0 VBE bios in a voodoo 5 is not  fully compliant.  The game itself doesn't offer any other modes that I know of yet, but there might be a command line switch or something.

Otherwise, do you know of any other external VESA 3.0  emulators similar to UNIVBE that can function with newer cards?

The Voodoo5's VESA definitely doesn't have support for the very old HiColor (32k) 15bit mode, so I'm afraid you're out of luck (unless you can choose 8 or 16bit Color.

Yep, that could actually be if the Game relies on old DOS INT# I/O requests.

Old DOS installers also often fail when they check the remaining Hard Disk space and see values >2GB free, often leads to internal overruns and weird errors, for example.