If your using a soundcard with a max 20kHz B/W, how do you propose transfering data beyond that bandwidth? That will surely be your limiting factor.
You could go to greater than 64QAM, but you will start to need expensive profesional modem chips.
In broadcasting, we increase the data rate by not only increasing the modulation vector points (ie BPSK>QPSK>16QAM>32QAM>64QAM) but also by increasing the number of carriers transmitted. In DVB-T (the system I work on) we use 64QAM over 8k carriers (8,000 carriers!) within an 8MHz RF bandwidth
To the best of my knowledge, to reach 100kbps, you will need >100kHz bandwidth, so if intending to stay with soundcards will need a very expensive one.
By moving away from soundcards to USB data transfer, you can get some 3Mbps to play with. But then your problem will be where
to use this? The only ham bands where you could run that kind of necessary bandwidth will be in the microwave regions.
The big question though is why?