AFAIK, as long as you keep the radials straight, a bit of a slope will not increase total loss, but will change the impediance as you say. The radials' job is to help the field lines back to the feed point, so any vertical angle should work as long as it's pointing straight back to the feed. Since your antenna will be sort of a half-reversed half-regular ground plane, I imagine it would affect the radiation pattern compared to the same vertical on a perfectly flat radial field.
Buried radials mean more loss, the deeper the lossier, so if you bury your radials so deep that none of the energy gets through to them (useless radials) you'll have an almost perfectly "matched" antenna in terms of VSWR.
I'll now wait to hear from somebody who knows their antenna theory better than me and has the EZNEC plots and practical experience to show for it. I'll just say that I've run verticals on ground sloping just as much as you have, without any particular issues matching the impedance.