When most techs think of using a "VNA," they are often thinking of one-port return loss. When using a two port analyzer, most are thinking tracking generator.
You can use a one-port VNA under return loss operations to tune a pass filter with a correct 50 ohm load attached to the other end. You can tune the filter for best return loss and pretty much bet that the filter is optimized for best pass-through characteristics at the frequency you are tuning. This is helpful at times when tuning a pass cavity between a known 50 ohm device and a slightly mismatched antenna. You can look into the input of the cavity and tune the cavity for best return loss while looking into the slightly mismatched antenna and compensate slightly for the mismatch.
I have found that the best way to tune a pass/reject, or a reject only duplexer is with a two-port VNA and run the plot in "insertion loss" mode. The top of the graph should be set to zero dB and the bottom at -100 for instance. You need to use phase-stable cables between the analyzer and the duplexer. Be sure to terminate the other side of the duplexer with a 50 ohm load while you're tuning the one side. You will be able to see the notches and pass curves in a visual manner and get them right on target with your marker table, depending on your analyzer.
If you are asking can you tune a duplexer with a one-port VNA using the return-loss method, I would say no. You can expect the return loss to be low at frequency X when the cavity is properly tuned to frequency X, but it would be near impossible to tune the reject notches with a one-port VNA. Although in theory you would think that really bad return loss at a certain frequency would mean a reject notch, it isn't always dead-on. You really need a two-port analyzer, calibrated, and using phase-stable cables to do your very best work.
Best regards and best of luck,
N4DBM.