Combining horizontal and vertical separation allows you to practically add the dB's of each together.
Unfortunately it doesn't work that way.
Vertical separation works because of the nulls off the ends of the antennas. If the
antennas are also separated horizontally they are no longer in the nulls, and only
the horizontal separation provides isolation.
My recommendations would be:
1) use the HT for transmitting and the mobile rig for receive. That reduces your
isolation requirements due both to the lower transmit power and the fact that the
mobile rig probably is less susceptible to desense. If you do use the mobile rig to
transmit, use the lowest practical power setting. If you are using an HT to talk
to the repeater, there isn't a lot of advantage of having it run 50 watts output.
2) isolation is improved if the ENTIRE transmitter and receiver are separated as
much as possible, because neither is well shielded as is, and even putting them in
fully shielded boxes you still have leakages from the cables, power leads, etc.
So separate the rigs by 100 feet or more and bury the power and audio leads
between them to reduce the RF via that path. The ideal situation might be two
trees about 300' apart along the crest of a ridge, using one for the transmit
antenna and one for receive. With such spacing there will be directions where
the repeater transmits better than it receives or
vice versa, but it makes
a more practical system when you have limited capabilities for isolation by
other means.