We had an Early 2008 Mac Pro and decided to finally upgrade the video card. We found an ATI Radeon HD 5770 (Mac version) for a reasonable price. We needed to drive 3 screens from the card but hit some issues.
The Radeon HD 5770 has 3 ports – one DVI port and 2 mini display ports. I was able to drive 2 of the monitors without issue using the DVI port and a Mini DisplayPort to DVI cable. Attempting to use the third port resulted in no video on the third screen.
Googling found many articles about using 3 screens for the 2009, 2010 and mid 2012 Mac Pro models, but nothing official mentioned the early 2008 Mac Pro model. Even worse there were mixed opinions on whether it was possible to drive 3 screens at all.
Luckily the solution was pretty simple in this case.
You need to ensure the type of DVI to Mini DisplayPort adapters you are using are active adapters as passive adapters (which I had) will not work for 3 screens.
You will note that in most cases the active adapters cost quite a bit more then the passive ones. In this case I can confirm I bought some cheapies off eBay and these still worked without issue, allowing the Mac Pro to drive all 3 screens at native resolutions.