Yes, filters are great. But I have never had very many. I have a 400 Hz filter on my SB-101, but I seldom use it any more. The rest of my radios only have the standard SSB filters. It would be nice to have better filters for working CW, but I have learned how to effectively use audio filters. I have an old MFJ-752C, a Autek QF-1A, and some home brew stuff that works pretty good.
I have a Heathkit HR-1680/HX-1681 pair that I built while I lived in Germany (DA2EU). The setup is CW only and only has the standard SSB filter and an audio filter. I worked 100s of QSOs from all over Europe, Asia, Africa, etc.., without any issues, even though the bands were pretty crowded. With only an audio filter, you still get lots of QRM, but you learn to listen through it.