Bob,
I think you are misunderstanding the duplexer terminology. There is no "variable loss" feature per se in a duplexer. There are basically two adjustments on each duplexer can. One sets the pass frequency - that is the frequency that will pass through the can with minimum loss. This is adjusted with the large tuning rods on the cans (not rotating the cans as you suggest). The second adjustment is the reject adjustment - this is the frequency that the can will attenuate. This frequency is adjusted with the capacitor adjustment on each can.
There are therefore two losses that be can measured in the can. One is the loss that the pass frequency experiences. This should be less than about 0.5 dB per can. The other is the loss of the reject frequency. This should be very high in each can - in the order of 25 dB. The adjustment procedure described in the Sinclair document on repeater builder accurately describes how to adjust the duplexer to these specifications. If you have 6-7 dB path loss, then either you have a defective duplexer, the duplexer is not made for your frequency range, your test bench is not set-up correctly, or you are not following the proper procedures.
Your comment that you are at -125 dBm with the duplexer but -118 dBm without it is very confusing. The duplexer will lower the sensitivity of the receiver so if you start at -118 dBm bare receiver sensitivity, the receiver can detect a signal of 0.282 uV. The duplexer, when properly tuned, will cause the receive frequency to be attenuated by roughly 1.5 dB. This means your receiver system sensitivity will drop to about .335 uV or -116.5 dBm. When sensitivity goes down, the dBm number becomes greater (less negative).
What is also important is that the duplexer should reduce your transmitter power that gets into your receiver input by 85 dB or more. This is essential to minimize the desense from the transmitter and of course, in an extreme case, avoid blowing out the front end of the receiver. This is the closest you can come to the notion of desired attenuation from a duplexer - but this is not called "variable attenuation" in the trade. The correct term would be "reject loss" or "reject attenuation".
Your transmitter output power will suffer the same 1.5 dB loss as your receiver sensitivity does. So if your bare transmitter in your MSF-5000 puts out 75 watts, you will only get 53 watts out from your duplexer.
Also be aware that termination impedance has a profound effect on the performance of a duplexer. If you bench test a duplexer with laboratory grade loads and then connect it to an antenna system that not close to a 1:1 SWR, you may experience very different results in the field from what you measured on the bench. If you cannot bring down the SWR of the antenna, consider inserting a matching circuit between the duplexer and the antenna feedline to provide a better match.
Also don't overlook the need to use high quality, double shielded coax from your repeater connections to the duplexer and from the duplexer to the antenna. This applies on the test bench and in the field. Leakage from low quality coax can negate a substantial part of the isolation provided by a high quality duplexer and can throw off test procedure results.
I hope that helps your understanding.
- Glenn W9IQ