It's well known outside Digi that their CoreAudio drivers are limited, slow, and impact system performance negatively. As a result you won't get optimum performance in Nuendo compared to other interfaces.
I think you have four choices, not sure whether the second and fourth one will work -
1 - If you're not needing the HD card then remove it and interface the Apogees with a card using better CoreAudio drivers. Best option if you can.
2 - Use the AD/DA16x as now through the HD card. If you have a firewire card in the Rosetta800 then plug it into the Mac and set up an Aggregate Device in AudioMidiSetup, combining the Digi8x8 and the Rosetta8x8 into a single 16x16. Then launch Nuendo and choose this interface.
3 - Get an inexpensive adat firewire interface like the M-Audio 32channel adat>firewire box (forget what it's called). Disconnect the AD/DA16's from the HD card and use lightpipe from them into the firewire box and select this in Nuendo. Ignore the HD card in Nuendo entirely.
4 - Connect the AD/DA16 units to the firewire bus, but you should check with Apogee to see if they cooperate when daisy-chained that way.
Caveats on the second and fourth solutions -
Will the HD card and your Apogee's aggregate? Do they play nicely? I don't know that. Also Apogee's older firewire solutions are peculiar. They work but, as an example, their docs all state only one firewire interface under OSX. Since 10.4.x you can have multiple firewire devices of the same brand under OSX just fine - but note Apogee's firewire docs have 2005 dates on them so they aren't updated. The positive indication is that they are natively supported under OSX and so should perform as expected. Bottom line is give it a try and/or call Apogee. Aggregate Device increases I/O latency.
I recommend solution 1, 3, or 4, these eliminate the HD card from the mix but you can decide what works for you.