Commercial 5.1 DVDs? Try the VLC player.
VLC media player for Mac OS X
Use the Apple Audio MIDI setup app (in Utilities folder) to set up your connections. If you already setup the mac to play Quicktime 5.1 files you might already be good to go.
I wish the Apple DVD player could do this naturally, but it can't decode Dolby 5.1 due to licensing restrictions.