I would be more than glad to help you out with whatever info I can. I am not a ham (yet) but I hope to be soon. I also have a lot of experience using the rtlsdrs with various software and also am a Linux user. I have four SDR receivers running on my computer, GNUradio was the first one I used, then I spent a lot of time using Linrad, and recently I found online that you can run SDR# under Mono without stuttering, by simply upping the audio buffer delay. It works for me.
To use your rtlsdr as a panadapter, of course it has to receive your IF frequency, whatever it is. If your IF is 10.7 MHz,you will either need an upconverter, or you'll need to tear into it so you can bypass the tuner chip and use the using either the I and Q inputs on the e4000 or just the I or Q on the R820T.
That will almost certainly break its ability to receive VHF/UHF. (You'll need to do it using direct conversion.)
SDR# is really quite slick but mono is slow, so if you use Linux, running SDR# under mono might not work so well for you unless you have a multicore machine.
I am not sure if it uses 64 bit under mono, it may not, but it definitely does use multiple cores. But its not much effort. If your machine is fairly recent, in my experience its really quite easy to get it working. Once its running, SDR# can do a lot.
I also like GQRX more for some things. For FM stereo reception its quite good. Linrad is probably the most stable one under Linux and it has the most configurability. It also has the ability to adjust the crystal error to parts per billion, not parts per million, which is desirable for higher frequencies, especially.
One thing about Linrad which isn't at all obvious in the Linrad install is that you should use a much lower setting in Linrad for amplitude of the SDR, because its an 8 bit SDR. I usually use 30, instead of the default which is 2000! They should make that clearer.
It can eke slightly more gain and lower noise figure out of the e4000 tuner (which is getting hard to find)
Of all of them it seems to be the one most geared for use as a amateur radio communications receiver.
Also, there is another program too, SDR-J that looks quite interesting, that I have not used.
If you are into digital modes I would definitely look at it - it has native support for quite a few digital modes, it seems. I have not used it yet.
Linrad has some very slick features like a notch filter and a very good noise blanker. They each have their strong points and weak points. I use my RTLSDR to listen to HF a fair amount. With an upconverter, they make a surprisingly decent receiver if you are willing to try to work around their limitations. For example, a preselector helps a lot - I use a magnetic loop as a preselector.
It depends a lot on your hardware's raw CPU power and what kind of things you want to do, which one is the best choice.
The main site for driver development is here: https://sdr.osmocom.org/trac/wiki/rtl-sdr
Debian recently added rtl-sdr, gr_osmosdr and the most recent Gnuradio (3.7.2) debs to their package management system for their stable distribution, (in backports, I think) which must mean it is already in their newer ones too.
their names are rtl-sdr, librtlsdr0, librtlsdr-dev, and for Gnuradio (make sure to install the 3.7 branch), Also, you should install gr_osmosdr. Its not immediately obvious to me which package supplies gr_osmosdr (I have a source install so I am not using the debs) but its probably libosmosdr
The easiest non-package manager way to install everything is using the great "build-gnuradio" script which will install and build everything from source off the net.
You could also use pybombs, which builds everything automatically and makes updating easy when you need to.
That is where I think everyone is going. Pybombs looks quite slick.