I don't really get it...are you telling me that the rosetta is what's causing the latency since you say that the symphony card has "virtually no latency"?
I'm not so sure...
Here's the apogee page:
Apogee Electronics > Products > The Symphony System where it says that the symphony system has a latency of 1.6 ms at 96 khz.
Here's the page of the ssl madi extreme -alpha link that has 0.9 ms round trip latency at 96:
Solid State Logic | Music
To calculate what the symphony latency at 44.1 would be you'd have to add a little more than 2x1.6 resulting in what, 3.5 or so....if then the rosetta converts a bit slower than the Alpha link then I guess ....but at the same time I don't quite understand where the 5.7 ms comes from that my core audio driver states when I select symphony in the menu....