It is a very interesting test, it was first described in amateur circles in QEX magazine years ago. The notched noise test could also be used for testing the ultimate IMD performance of a transmitter. A notched noise test is used for testing DRM shortwave transmitters and can show many other problems besides IMD performance.
The problem with many of these tests is that it suffers from the NIH(not invented here) so these perfectly good tests are not carried through into the ARRL or RSGB reviews.
Its only Adam VA7OJ and SM5BSZ who are investigating the potential of these advanced tests to ultimately tests RX and TX performance. Its an overwhelming consumer of resources and time that few individuals can spare. We very fortunate having the likes of Adam, Leif and Rob Sherwood doing these full range of tests. We certainly cant rely on the ARRL and RSGB exclusive to give a clear and accurate picture of ultimate performance. Although to the credit of the RSGB they do more often mention the warts of various radios and brands.
Another important test which is should also be included is receiver RX Inband IMD this would also show some of the fatiguing receiver reports that we get with new radios.
More thorough testing of AGC performance like that which SM5BSZ has done would also be useful. AGC performance can make a huge difference in the real world. Things like mushing up a CW pileup and not making CW signals pop out of the noise are related to RX IMD, IMD within filters and other passive parts.
K4ERO also had some interesting test methods for transmitters in the June 2014 QST magazine. He discussed in his article the use of triangular waves to test transmitters. His test would make it much easier for the average ham to spot transmitter problems. Dynamic testing for IMD performance and reporting what the real IMD levels are in 5khz steps would be far more useful than 2 tone numbers. A transmitter that IMD suppressed 80 db at 20khz is better than one that is 40db at 20khz. It would also be easier for the average ham to understand.
Ultimately the amateur service should agree on a set of standards which no transceiver manufacture should go below. Then at least we can say thats the standard and you fail or pass. Thats how most international testing standard work, simply pass or fail. At the moment with so many points of views and questions to what test methodologies should be used for receiver and transmitter testing, it can look like you picking on one brand or another. A set of defined standard would take out the "brand bias" and produce test results that focuses on performance and meeting standards rather than glorifying excessive pursuit of high numbers that have no relevance in the real world or for no good reasons. The Icom IC7850 is a good example, excessive irrelevant performance metrics that wont amount to much for most hams, while the TX performance of such an expensive radio will be marginal at best.
The last few issues of QST has had a focus on product testing and the ARRL's latest test equipment acquisitions. It will be interested to see if they use this test equipment to report the failings of many manufacturers in a honest technical manner. If they could produce data like SM5BSZ we could have faith in their equipment and test methods. I do however think the ARRL would do us all a favor if their performance charts would have a upper limit marker indicating things like excessive dynamic range and sensitivity would serve no useful purpose beyond stroking your ego. This certainly would stop the undeniable obsessive focus on receiver performance number chasing while ignoring things like TX IMD, RX IMD distortion, IMD induced in the RX chain(NPR) and instead focus on DSP noise reduction and noise blanking performance which is far more important to the average ham.
It would be a great day in ham radio when you can actually a buy a perfect radio, they all seem to be broken in one area or another regardless of price!
I have been following the threads about Sherwood Engineering's testing and have two thoughts on the subject.
1. Noise Power Ratio (NPR) testing was mentioned in was our best tool when adjusting an analog microwave system to handle 600 channels of LMX-600 systems. The LMX-600 was 600 SSB channels spaced at 4 kHz. intervals starting in the 100 kHz. reason and extending upward from their. The system would accommodate 600, 900 or 1,200 SSB channels and handled almost all medium to long haul telephone conversations in the USA in the 60s through early 80s. NPR testing was a wonderful tool.
2. None of the current testing addresses delay. While most of the current test results are meaningful to CW contesting, some addresses SSB contesting concerns none of the test really address to rag-chew groups in non controlled net environments. Most often there will be 3 to 12 stations operating in a very interactive VOX environment where increased "doubling" has been noticed when SDR transceivers are used by some of the participants. Digital delay in both the transmitter and the receiver path seem to be the main contributor to this. The delays seem to be ignored by current test methods.
73 - Mike - K9JRI