If I have a narrow filter in the receiver, the audio is poor. If I widen the bandpass, the selectivity suffers. I see a similar problem on the transmitter. I am using a Flex Radio 1500 to view the transmitter output, and when the transmit audio is satisfactory, I see part of the lower sideband being transmitted. If I try to tighten up the passband, the audio suffers.
There is no solution to this, because there are two conflicting requirements.
If you want "full" wide bandwidth audio, you can't pass it through a narrow
filter. If you have good low frequency response then there isn't enough
filtering to remove the carrier and part of the opposite sideband. That's due
to the limitations of crystal filters built with real world parts. You can't get
a lot of audio through at 400Hz while suppressing the carrier (only 400Hz
away) and the opposite sideband (800Hz away) with a typical filter roll-off.
(Asymmetric filters may be better in this regard than symmetric ones.)
For good audio "punch" you want to emphasize the speech frequency range
that contributes the most to communications effectiveness - that usually
means filtering out the low frequency components. The point is to be
effective, not to sound "natural". That allows you to put more power into
those portions of the audio spectrum that contribute to intelligibility at
the far end. The typical 2.1kHz bandwidth for SSB intentionally sacrifices
some audio range for better selectivity.
For SSB to sound good the BFO frequency has to be carefully positioned relative
to the filter passband (in both the transmitter and the receiver), or you clip too
much on the high or low end of the audio. The narrower the bandwidth, the more
important this is.
Meanwhile there is a lot of information on homebrew crystal filters available from
folks such as G3UUR, W7ZOI, and others. Here is a useful summary page that
might point you in some useful directions:http://www.qrp.pops.net/crystal-tester-2010.asp
With some fairly simple measurements of crystal parameters and some math you
can build pretty good filters, especially making use of some of the software that
is also available.