The D-104 has a high (very high) impedance crystal element originally designed to work with tube sets. It was also designed to give emphasis to the higher frequencies between 2k-3khz in order to give it 'punch'. It was the the speech processor equivalent.
The problem is that modern solid state rigs by the 1960's used low Z mics. Astatic developed a two transistor amplifier for mounting in the mic base. It takes a 9v battery and has a volume/modulation adjustment. It can still sound a bit pinched or tinny.
If you really want the best audio, you can build a very simple FET amplifier which provide a +1Mohm impedance to the crystal, but also amplifies to a level which makes the radio happy. http://kd5rug.org/anarchy/PDF/d104amateurradio.pdfhttp://www.qsl.net/hcara/Improving_the_D-104_N9WB_rev2.pdf
There are at least half a dozen variants on the FET amp, two or three have been published in QST, as early as 1969 possibly by Lew McCoy. I recommend a QST search and also on the web to find one you like. Parts can be had at Radio Shack and it shouldn't take an hour to rig up.
So, don't rip up that D104 needlessly. You can get great audio from it and help rehabilitate it's reputation. Oh, it IS possible for the crystal to get damaged by dropping it, or absorbing moisture. But obviously yours is working.