Damon,
As far as bandwidth.. it's really not a good comparison.. SP/DIF is a stereo data stream so its always the same. Firewire is a much more versatile protocol which can handle many more streams at once.
I understand about wanting to use the superior convertoers of the Apogee but just using the SP/DIF on the G5 will be very limiting I think.... And thats without checking the Latency.
Doesn't the Rosetta 800 have an ADAT I/O? Or an option for one at least surely???
I personally would use the ADAT I/O on the MOTU to interface to the Rosetta, plus you also gain by having 16 Analogue I/O's this way
Al.