1) Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?
I have a copy of SM7CPY's thesis in which he mostly describes research about the human ear's/brain's ability to copy Morse code, but also with some CW decoders: PL232. TONO 7000E, Telereader CD 670, CWR 900 and PROCO CD 630. These are from the early 1990s or even earlier and therefore don't use very sophisticated technology; just a bandpass filter plus level detector.
For perfect copy at 16wpm (he didn't test 20wpm) and with wideband masking noise (100 .. 2500 Hz, tone frequency 800 Hz) the decoders require something between -4 and +3 dB of SNR.
In his studies, a human decoder required -7 dB of SNR under the same conditions.
The difference between his and your noise bandwidth (2.3 kHz vs 3 kHz) would account for roughly 1 dB of difference in SNR; I am not exactly sure how big the influence of the speed difference is.
Regarding further tests with decoders I would be interested in any news, but my own research is going into the direction of the capability of humans to copy CW under low SNRs.
On my Morse Code training website (http://lcwo.net/
) I will soon roll out a test everyone can take, in which the test subject has to copy a number of random groups with varying SNR; in the end I hope to be able to generate a SNR/CER plot for each individual person, but also get some insight into differences between individuals.
Thanks for this information - great news on the upcoming lcwo.net SNR/CER feature.
I stumbled on this report http://www.dtic.mil/dtic/tr/fulltext/u2/a019493.pdf
from Sept 1975. It has also some data on human operators - see Table II on page 19.
The report shows also near perfect human copy at -7 dB SNR at 2kHz - speed was 35 or 25 WPM in these tests. Error rate grows to 5-6% at 25 WPM and to 12-15% at 35 WPM when SNR drops to -13 dB (@2Khz).
It would be really nice to produce some updated charts on human vs. machine performance SNR/CER limits and to understand better how human brain really works when decoding CW.
Did the SM7CPY's thesis provide details on how long the test was? How many characters were decoded at 16 WPM?
Based on the above report noise adds significant mental load as operators complained about fatigue under noisy conditions.
You may want to add a simple questionnaire to your website to collect also some data on the fatigue experienced after the test.
I would be very interested in the data that you will collect through this upcoming CER/SNR test feature in lcwo.net.