I've built a few dozen traps (6") and panels (3") with Roxul Safe and Sound. When cutting I use a mask and when handling directly I use gloves. I have 16 panels and traps in my project studio.
I have never had the slightest irritation with it, either while building or while in the studio. I wrap my panels and traps in fabric, but I don't use any other sealant.
Rockwool is a different animal from fiberglass, and I don't think it should cause the same respiratory irritation. It's certainly not my experience. Assuming you don't beat your panels with a bat, it shouldn't be producing any irritating dust.
Obviously it is important to clean up after construction.
Anyway, with simple precautions you should be fine. I'll stake your life on it.
