FLDIGI Morse decoder test results and metrics question

[AG1LE]

I spent some time today running some tests to find out the limits of FLDIGI v3.21.64  CW decoder in terms of CER (character error rate)  vs.  SNR  (signal-to-noise ratio).

Test results are available here: http://ag1le.blogspot.com/2013/01/morse-decoder-snr-vs-cer-testing.html

I am looking for advice:

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?

2)  What other tests or metrics would you recommend to drive further improvements in the FLDIGI CW decoder? 


Happy New Year and 73
Mauri  AG1LE

[DJ1YFK]

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?

I have a copy of SM7CPY's thesis in which he mostly describes research about the human ear's/brain's ability to copy Morse code, but also with some CW decoders: PL232. TONO 7000E, Telereader CD 670, CWR 900 and PROCO CD 630. These are from the early 1990s or even earlier and therefore don't use very sophisticated technology; just a bandpass filter plus level detector.

For perfect copy at 16wpm (he didn't test 20wpm) and with wideband masking noise (100 .. 2500 Hz, tone frequency 800 Hz) the decoders require something between -4 and +3 dB of SNR.

In his studies, a human decoder required -7 dB of SNR under the same conditions.

The difference between his and your noise bandwidth  (2.3 kHz vs 3 kHz) would account for roughly 1 dB of difference in SNR; I am not exactly sure how big the influence of the speed difference is.

Regarding further tests with decoders I would be interested in any news, but my own research is going into the direction of the capability of humans to copy CW under low SNRs.

On my Morse Code training website (http://lcwo.net/) I will soon roll out a test everyone can take, in which the test subject has to copy a number of random groups with varying SNR; in the end I hope to be able to generate a SNR/CER plot for each individual person, but also get some insight into differences between individuals.

73
Fabian

[AG1LE]

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?

I have a copy of SM7CPY's thesis in which he mostly describes research about the human ear's/brain's ability to copy Morse code, but also with some CW decoders: PL232. TONO 7000E, Telereader CD 670, CWR 900 and PROCO CD 630. These are from the early 1990s or even earlier and therefore don't use very sophisticated technology; just a bandpass filter plus level detector.

For perfect copy at 16wpm (he didn't test 20wpm) and with wideband masking noise (100 .. 2500 Hz, tone frequency 800 Hz) the decoders require something between -4 and +3 dB of SNR.
In his studies, a human decoder required -7 dB of SNR under the same conditions.

The difference between his and your noise bandwidth  (2.3 kHz vs 3 kHz) would account for roughly 1 dB of difference in SNR; I am not exactly sure how big the influence of the speed difference is.
Regarding further tests with decoders I would be interested in any news, but my own research is going into the direction of the capability of humans to copy CW under low SNRs.

On my Morse Code training website (http://lcwo.net/) I will soon roll out a test everyone can take, in which the test subject has to copy a number of random groups with varying SNR; in the end I hope to be able to generate a SNR/CER plot for each individual person, but also get some insight into differences between individuals.

73
Fabian

Hi Fabian

Thanks for this information - great news on the upcoming lcwo.net  SNR/CER feature.

I stumbled on this report http://www.dtic.mil/dtic/tr/fulltext/u2/a019493.pdf from Sept 1975.  It has also some data on human operators - see Table II on page 19. 
The report shows also near perfect human copy at -7 dB SNR at 2kHz  - speed was 35 or 25 WPM in these tests.  Error rate grows to 5-6% at 25 WPM and  to 12-15% at 35 WPM when SNR drops to -13 dB  (@2Khz).

It would be really nice to produce some updated charts on  human  vs. machine performance SNR/CER limits  and  to understand better how human brain really works when decoding CW.

Did the  SM7CPY's thesis provide details on how long the test was?  How many characters were decoded at 16 WPM?   
Based on the above report noise adds significant mental load as operators complained about fatigue under noisy conditions. 
You may want to add a simple questionnaire to your website to collect also some data on the fatigue experienced after the test.


I would be very interested in the data that you will collect through this upcoming CER/SNR test feature in lcwo.net. 

73
Mauri AG1LE 

[PA0BLAH]

My comment is that it is best to use psophometric weighting of the noise,

or do not add noise over the whole audioband, and use that to test people and to calculate the signal to noise ratio, but use noise over the same spectrum as a regular 500 Hz wide CW filter passes through.

[N6YWU]

Wouldn't testing with audio band-pass filtered 1/f or "pink" noise be more representative of actual over-the-air radio noise?  Assuming such, I've been adding the output of a Voss-McCartney pink noise generator to my iOS Morse Code decoder unit test code. 

I'm also working on a signal fading test, but I'm not sure what my fading modulation signal should be.  Something like a dB signal strength change per second rate with random changes of slope direction at some time constant?