Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: [1]   Go Down
  Print  
Author Topic: FLDIGI Morse decoder test results and metrics question  (Read 1943 times)
AG1LE
Member

Posts: 114


WWW

Ignore
« on: January 01, 2013, 10:05:03 PM »

I spent some time today running some tests to find out the limits of FLDIGI v3.21.64  CW decoder in terms of CER (character error rate)  vs.  SNR  (signal-to-noise ratio).

Test results are available here: http://ag1le.blogspot.com/2013/01/morse-decoder-snr-vs-cer-testing.html

I am looking for advice:

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?

2)  What other tests or metrics would you recommend to drive further improvements in the FLDIGI CW decoder? 


Happy New Year and 73
Mauri  AG1LE
Logged
DJ1YFK
Member

Posts: 182


WWW

Ignore
« Reply #1 on: January 02, 2013, 08:59:02 AM »

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?
I have a copy of SM7CPY's thesis in which he mostly describes research about the human ear's/brain's ability to copy Morse code, but also with some CW decoders: PL232. TONO 7000E, Telereader CD 670, CWR 900 and PROCO CD 630. These are from the early 1990s or even earlier and therefore don't use very sophisticated technology; just a bandpass filter plus level detector.

For perfect copy at 16wpm (he didn't test 20wpm) and with wideband masking noise (100 .. 2500 Hz, tone frequency 800 Hz) the decoders require something between -4 and +3 dB of SNR.

In his studies, a human decoder required -7 dB of SNR under the same conditions.

The difference between his and your noise bandwidth  (2.3 kHz vs 3 kHz) would account for roughly 1 dB of difference in SNR; I am not exactly sure how big the influence of the speed difference is.

Regarding further tests with decoders I would be interested in any news, but my own research is going into the direction of the capability of humans to copy CW under low SNRs.

On my Morse Code training website (http://lcwo.net/) I will soon roll out a test everyone can take, in which the test subject has to copy a number of random groups with varying SNR; in the end I hope to be able to generate a SNR/CER plot for each individual person, but also get some insight into differences between individuals.

73
Fabian
Logged

AG1LE
Member

Posts: 114


WWW

Ignore
« Reply #2 on: January 02, 2013, 12:44:06 PM »

1)  Has anybody done similar tests comparing multiple CW decoder software packages with noisy signals?
I have a copy of SM7CPY's thesis in which he mostly describes research about the human ear's/brain's ability to copy Morse code, but also with some CW decoders: PL232. TONO 7000E, Telereader CD 670, CWR 900 and PROCO CD 630. These are from the early 1990s or even earlier and therefore don't use very sophisticated technology; just a bandpass filter plus level detector.

For perfect copy at 16wpm (he didn't test 20wpm) and with wideband masking noise (100 .. 2500 Hz, tone frequency 800 Hz) the decoders require something between -4 and +3 dB of SNR.
In his studies, a human decoder required -7 dB of SNR under the same conditions.

The difference between his and your noise bandwidth  (2.3 kHz vs 3 kHz) would account for roughly 1 dB of difference in SNR; I am not exactly sure how big the influence of the speed difference is.
Regarding further tests with decoders I would be interested in any news, but my own research is going into the direction of the capability of humans to copy CW under low SNRs.

On my Morse Code training website (http://lcwo.net/) I will soon roll out a test everyone can take, in which the test subject has to copy a number of random groups with varying SNR; in the end I hope to be able to generate a SNR/CER plot for each individual person, but also get some insight into differences between individuals.

73
Fabian

Hi Fabian

Thanks for this information - great news on the upcoming lcwo.net  SNR/CER feature.

I stumbled on this report http://www.dtic.mil/dtic/tr/fulltext/u2/a019493.pdf from Sept 1975.  It has also some data on human operators - see Table II on page 19. 
The report shows also near perfect human copy at -7 dB SNR at 2kHz  - speed was 35 or 25 WPM in these tests.  Error rate grows to 5-6% at 25 WPM and  to 12-15% at 35 WPM when SNR drops to -13 dB  (@2Khz).

It would be really nice to produce some updated charts on  human  vs. machine performance SNR/CER limits  and  to understand better how human brain really works when decoding CW.

Did the  SM7CPY's thesis provide details on how long the test was?  How many characters were decoded at 16 WPM?   
Based on the above report noise adds significant mental load as operators complained about fatigue under noisy conditions. 
You may want to add a simple questionnaire to your website to collect also some data on the fatigue experienced after the test.


I would be very interested in the data that you will collect through this upcoming CER/SNR test feature in lcwo.net. 

73
Mauri AG1LE 







Logged
PA0BLAH
Member

Posts: 0




Ignore
« Reply #3 on: January 03, 2013, 02:33:51 PM »

My comment is that it is best to use psophometric weighting of the noise,

or do not add noise over the whole audioband, and use that to test people and to calculate the signal to noise ratio, but use noise over the same spectrum as a regular 500 Hz wide CW filter passes through.
Logged
N6YWU
Member

Posts: 16




Ignore
« Reply #4 on: February 10, 2013, 05:02:06 PM »

Wouldn't testing with audio band-pass filtered 1/f or "pink" noise be more representative of actual over-the-air radio noise?  Assuming such, I've been adding the output of a Voss-McCartney pink noise generator to my iOS Morse Code decoder unit test code. 

I'm also working on a signal fading test, but I'm not sure what my fading modulation signal should be.  Something like a dB signal strength change per second rate with random changes of slope direction at some time constant?
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!