so this mfj antenna tx rx switch is junk then? its supposed to short the SDR ports center pin to ground when the radio keys up, and I made for sure have the SDR on the correct port of the switch.
Generally, yes. *(MFJ)=junk / Mighty_Fine_Junk
But, you can be seeing the phenomenon you are (with so much RF on the shield of the coax) due to common mode current.
Chokes can be your friend here. Judicious use of type 31 and / or type 43. As well as a good ground, bonded to the service panel of the house (or wherever else your service drop bonds the ground to neutral).
I used to get a nasty AC hum on my RX recovered audio. Found out common mode current was causing it.
You can always try an external antenna. Short antenna to ground right before you go to TX. Do not have the SDR antenna connected to any coax or anything else. If this fixes your problem, you know your coax ground isn't 'ground'.
--Shane
WP2ASS / ex KD6VXI