You might need to hack up the speaker-mic to get it to work. I don't know what the 817 will think of the HT-type PTT which is hooked up to the mic element. The 817 has direct connections to the PTT switch and mic element. I can't remember the exact hookup of the HT speaker-mics, but I think it puts the PTT switch inline with the mic and circuitry, so if you seperate them and plug it into the RJ jack, it should work fine.
I built an adapter to use an aircraft headset with my 817, just needed to put in a PTT switch and a resistor and capacitor, the most basic mic driver circuit I could find. Might not be perfect, but it works. The headphones go directly to the speaker jack and I used half of an ethernet cable for the PTT and mic. I have pictures at my web site: http://home.teleport.com/~kd7bcy/adapter.html
As for audio quality, I would assume that hooking up a speaker-mic is more for convenience than quality, but I don't think there would be any problems once everything is setup and the mic gain set(a setting I haven't tested with my headset yet beyond a few on-the-air checks, I need to dig out my test equipment and see what the output looks like, but people haven't seen much difference between the settings so I usually leave it on the default 50)