Simple is good. It makes for a neat but functional station. But certain things are nice to have, IMHO.
1) I like to monitor RF Power Output and SWR as I operate. This way you can tell if things are operating OK.
2) I like to also monitor the DC Current and DC Voltage going into the rig. I use an Astron RS-35M (linear power supply, with meters).
3) The antenna must be disconnected from the rig when not in use. The ability to select one of several antennas is nice, too. Some people also disconnect the power and any interface cables.
4) Like you said, you need a good-sounding speaker. It does not need to be hi-fi, but you don't want thin sound with rattles or distortion.
5) Keep the cable from the power supply to the rig as short as possible to reduce voltage drop during transmit.
6) Make sure that there is no RF interference to the rig from the antenna. Pay attention to grounding and ferrites if there is. See K9YC information.
7) Always turn on the power supply first, then the rig (to avoid transients). Likewise, turn the rig off first, then the power supply.

Make sure that nothing in the shack or house is generating noise. I have a printer that raises the noise floor by 2 S units when it is on. I keep it unplugged when not in use. Use an AM radio and check each room in the house. I was surprised at the number of things that were noisy in my house. Some light fixtures here were real hash generators. Keep all those off when you're on the air. If you interface a computer to the radio, check that for noise also. A battery charger here for a Bose portable speaker is also very noisy and is off when not in use.
Enjoy your simple but effective set up. 73, John W2XS