Wow, that is terrific!
As for the delay fading on the left, did you try evening out the output allpass sizes? Looking at your tap layout you might want to make the left output allpass sizes a bit larger than the right.
Also, could it be from output taps that are within a loop allpass? (does it happen when the loop allpass gains are at 0?)
I'll bet the choppy sound is something not being sized correctly (taps outside of loops, that sort of thing)
I can't wait to hear it.
-Casey