is there anything inherently shaky in clocking through a SPDIF connection?
Yes. S/PDIF is a horrendously poorly designed interface. This is because it combines the clock and audio coding onto the same signal. The receiver is supposed to recover the clock from this signal as well as extract the audio data. This turns out to be a non-trivial task, and one that almost always leaves the recovered clock contaminated with signal correlated jitter artefacts. So any clock you transmit over an S/PDIF connection that is also carrying audio signals will probably be worse than any internal clock.
You can reasonably (but not ideally) send a clock over an S/PDIF interface if the interface does not carry any audio signal, but even then the poor bandwidth of the interface does not lend itself to particularly accurate clock reticulation.
Clocks are hard. Really hard. Even proper word clock distribution systems are far from ideal, but S/PDIF is particularly flawed, and should have no role in any transmission that involves clock recovery.