Check the drivers? What OS version are you using? I'm a Mac user so probably not much help, but those that can help need a bit more info.
Windows is famous for confusing drivers on USB devices? Also you may want a powered USB hub, the kind that plugs in to A/C, so your devices all get enough buss power.
Hot plugging USB should not kill your gear, but a driver would get confused if it suddenly lost the device it was communicating with, it gets stuck in a circular state, forgetting what it's doing, so to speak. Sometimes blocking out the port. I use a lot of strange USB convertors in my non audio stuff (all PC), happens a lot, mostly with serial type connections.