Many people actually do just that.  The results are generally positive, and would be on par with our dedicated headphone amps with one exception - there wouldn't be an attenuation resistor in the path making it possible for the amp to get too loud and or hum with some of the more efficient and lower impedance phones.  Of course a series resistor isn't terribly hard to install in your headphone cable adaptor so even that's not a big risk.

Steve

Nope, no issues!