It's a great question, and the short answer is that using a switch box won't hurt anything except possibly the sound.  Most speaker level switch boxes have impedance protection so that solid state amps won't explode when you engage more than two sets of speakers.

Some things to consider...

The headphones play about 4 times louder than the speakers in many instances. That makes them low level background monitors during headphone use.

To turn the speakers off, we use speaker cables that are terminated with banana jacks so that we can simply lift one of the jacks out of the binding post hole on each channel.  It takes no longer than flipping two switches, but has none of the fidelity robbing consequences.

-Steve