Towards Bayesian CW Decoder

[AG1LE]

I spent few sleepless nights to implement my first version of Bayesian CW decoder. It is still in alpha level quality but it does decode Morse code to -10dB SNR (3Khz) using 32Hz Matched filter almost perfectly.

I used FLDIGI v.3.21.64 as the basis and I documented my learnings in here:

http://ag1le.blogspot.com/2013/01/towards-bayesian-morse-decoder.html


It is not ready for prime time yet but if anybody is interested developing this further please let me know.



73
Mauri AG1LE

[PA0BLAH]

minus 10 dB over 3 kHz is plus 10 dB over 30 Hz . Will not be difficult to copy in your head when listening after a filter that small.
I suppose you have to limit the speed to maximal 18 wpm

Bob

[AG1LE]

minus 10 dB over 3 kHz is plus 10 dB over 30 Hz . Will not be difficult to copy in your head when listening after a filter that small.
I suppose you have to limit the speed to maximal 18 wpm

Bob

Hi Bob
I used  20 WPM for the test audio files.  At 20 WPM the FLDIGI optimized Matched filter uses abt 32-35 Hz bandwidth.
If you try to listen noisy morse with such a narrow filter it sounds a bit awkward.

Do you have some formula to calculate SNR at different bandwidths?  

I was trying to find a simple SNR / BW equation for my BER/SNR study
but defining  bit rate for Morse is not obvious as the symbol size varies so much.
I scrapped the idea of measuring BER and used CER vs. SNR instead ]http://ag1le.blogspot.de/2013/01/morse-decoder-snr-vs-cer-testing.html]


73
Mauri AG1LE

PS. I generated the test audio files using Octave - see the code here http://ag1le.blogspot.de/2012/06/morse-code-detection-using-modified.html
The SNR definition is also there.

[PA0BLAH]

Well,

A matched filter has an impulse response that is the mirror of the expected signal. with the result that the output, which is the convolution of the 2 peaks, which make the signal detectable when buried in noise. The power distributed over time in the signal is collected to one moment at the output, so that makes the signal to noise ratio optimal.

In Morse code you have different speeds and different characters, so matched filtering will require a set of at least 40 filters, for each expected speed.

That is what I thought but I probably will be wrong.

A local ham PA0LQ made a combination of a noise blanker, synchronised with the powerline, and a an analog filter optimised for impulse response, kind of Gaussian filter I suppose. That has about the bandwidth you mention and lots of hams over the world duplicated it and were very satisfied with the results.

http://www.wireless.org.uk/blanker.htm

I suppose you made your noise by a random number generator added to a sine wave Morse signal in your wav files.

It looks to me better to limit the bandwidth of the noise to 500 Hz in order to get realistic comparison.

White noise has a power which is proportional to the bandwidth, so 3 kHz noisepower is  20 dB more then 30 Hz, but because it is interesting to compare with human copy by head, you have to think about the fact that your ear is less sensitive for high frequency noise. In telecommunication the work for that reason with psophometric weighting.

All that is unnecessary when you limit your bandwidth of the noise and his power to a few hundred Hz.

[AG1LE]

Well,

A matched filter has an impulse response that is the mirror of the expected signal. with the result that the output, which is the convolution of the 2 peaks, which make the signal detectable when buried in noise. The power distributed over time in the signal is collected to one moment at the output, so that makes the signal to noise ratio optimal.

The original version of matched filter  I created was using convolution in time domain: http://ag1le.blogspot.com/2012/05/fldigi-adding-matched-filter-feature-to.html. The problem with that approach was slight delay in decoding - for best results  this created 2 -3 seconds of time delay in order to buffer and calculate the result. Also, I had a problem stitching the audio segments together despite some overlap in buffering.  But you are right - it was pretty amazing to see stuff deeply buried in noise coming visible! 

Also, due to the fact that the envelope of the filtered signals didn't look like nice square wave of dits and dahs, more like noisy triangles I started working on another project. I implemented algorithm from "self organizing maps" to calculate best match against known codebook  patterns.  This code is now used in FLDIGI ("SOM" decoding feature).

In Morse code you have different speeds and different characters, so matched filtering will require a set of at least 40 filters, for each expected speed.
That is what I thought but I probably will be wrong.

Dave W1HKJ  took my matched filter code and re-implemented it in frequency domain using FFT/FIR  based methods. Current version in FLDIGI  is doing real time filtering and instead of using 40 different filters it automatically calculates the filter parameters when you change the station you listen. The software also tracks the Morse speed automatically and adjusts the filter bandwidth to the speed.  If you have a chance to look at the source code it is actually pretty slick implementation. 

I suppose you made your noise by a random number generator added to a sine wave Morse signal in your wav files.
It looks to me better to limit the bandwidth of the noise to 500 Hz in order to get realistic comparison.

Thanks  for this tip.  I will look into how to add a lowpass filter in the noise generated by Octave code. 
I have also used the PathSim  program that has capability to adjust the noise bandwidth but also add different
delay effects simulating ionospheric propagation effects. 

White noise has a power which is proportional to the bandwidth, so 3 kHz noisepower is  20 dB more then 30 Hz, but because it is interesting to compare with human copy by head, you have to think about the fact that your ear is less sensitive for high frequency noise. In telecommunication the work for that reason with psophometric weighting. All that is unnecessary when you limit your bandwidth of the noise and his power to a few hundred Hz.

Thanks - the answer was so obvious that I am almost embarrassed. I am using computer generated white noise and I was aware that human auditory system has different sensitivity to noise at different frequencies.  Using "psophometric weighting" search in Google provided so many new sources that I will study this subject bit more. 

Bob,
as you seem to have very good insights in these topics I am asking the following:

Are there any other good metrics when comparing Morse decoder performance with human copy by head?   

So far I have   the following:

• CER - Character Error Rate
   SNR  - Signal to Noise Ratio
   WPM - speed in words per minute
   Psophometric Noise distribution - to adjust to human auditory system
  

I am interested in how brain detects and decodes Morse code as well as the learning process itself. 
Having a set of consistent quantifiable metrics would help  - it looks like Fabian DJ1YFK has also similar interests in this field.

73
Mauri AG1LE   

[PA0BLAH]

Yes CER and so on are OK, I think so.

Bayesian is another approach than matched filtering. Concerning detection by distinguishing signal power from Gaussian noise, the last mentioned  is theoretical optimal.

When your floor is slightly dusty you don't notice, but you detect the dust by wiping the floor with a broom to a centerspot. When you collect that way all the dust on a small spot you can decide was there dust or was there no dust(signal)  on the noise floor.

The method is used to find out wether or not there is a signal of known shape in noise. Military radar tx/rx use it, and only reception is used in anti submarine warfare, each submarine type has a certain sound signature, that is implemented in a matched filter to detect them at large distance by mere listening, in order not to get localised by them.

You have maximum shift register sequences, so named because with a shift register of n stages long you can generate a binary signal of 2^n-1 before it repeats. Also called pseudo random sequence, because it looks like uncorrelated random bitstream, but it is not random, because when you know n bits you can tell all the upcoming bits and the exact bitstream of the past.

When you make a filter with the impulse response the mirror sequence, and you transmit with a radar antenna the sequence, and listen with the matched filter it peaks a pseudo Dirac pulse out, when the signal is detected. All the power (dust) deep in the noise is wiped to one point in time of one bit wide.

When you want to distinguish 2 signals, you need 2 orthogonal signals (cross correlation  zero) otherwise the matched filter detection will respond (less) on the undesired bit sequence.

So when you make a FIR filter with impulse response 111010101 it will peak with one DOT 1 as output on a V signal  as input.
Delay is 2 characters.Wiping the noisefloor takes time.

However when you make in your software 40 matched filters working in parallel for 40 Morse characters, those 40 bitsequences are not orthogonal with each other, so it will not be so that one responds and the others completely not. but the response of the filters can be forcasted and the strongest one is the best fit.

Different CW speed is not an objection in the digital world because you have your signal as a set of samples and by defining the time axis between the samples you change the speed.

Actually, this kind of research is interesting and rewarding, but I hate decoders and do not promote polluting the CW crowd, with guys just buying a radio and an antenna, a Morse decoder and mixing up.

First of all they have to quit a rag chew when there is some qrm, long before a human cw decoder have to quit. Second: Big Brother is watching you, a lot more then you probably think. I don't want that, and the last ressort that Google and the like can't yet spy are the point to point radio connections we make with CW. And the third reason is that CW is not optimal for machine decoding as compared with PSK31, due to AM and suboptimal coding scheme, but it obviously is for human decoding. So those guys will be happier with PSK31 and so on.

So look at rtty or TOR the last with FEC and you have a fixed signal length, that will make it more rewarding and less questionable to decode by machine.

It may be interesting for you to study the way GPS satellites are detected, and the way WSPR detects signals in noise.

G3PLX warned that in noise, adding redundancy is not very helpfull when you keep the bandwidth the same. According Nyquist you have then to add signal levels, which are harder to distinguish of each other in the noise. So he designed on request QPSK, which has four phases instead of 2, and the redundancy is used with a Viterbi algorithm to make an optimal decision.
That creates a win-loss situation. Your redundancy is used to correct errors but the number of errors increases due to the doubling of the phase levels to distinguish.

Also widening your bandwidth in order to increase redundancy with FEC and keeping the 2 levels instead of making 4 levels, is punished by the fact that the noise power increases when using more bandwidth. You can use time.that helps but then you get the delay you said you don't want.