I had a similar problem with the SO-239 connector in an MFJ-249 antenna meter. Sometimes the VSWR would just whack the single meter on the device between a low reading and "infinity to one". In that device, I traced the "wonkiness" to cracked solder between the pc board's solder tabs and the center pin of the RF connector. Sometimes it would touch and complete the circuit connection, and sometimes the cracked solder was enough to do a total disconnect.
I don't remember if the center pin of the rear antenna connector in the FT-817/818 radio family solders directly to the main board or not.
One other thing you could try is to change the menu setting for which bands use the front BNC or rear UHF style connector. If front connections fix the "wonkiness", then most of your radio is exonerated and the problem is solely at the RF connector on the rear.
Good luck on your quest to exorcise this gremlin.
Tony (W9MT)