Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: [1]   Go Down
  Print  
Author Topic: Sparse coding to reduce noise in Morse signals  (Read 7529 times)
AG1LE
Member

Posts: 139


WWW

Ignore
« on: May 11, 2014, 09:40:23 AM »

I did some experiments with sparse coding methods in order to remove noise from CW audio signals.

There is a signal processing algorithm called Matching Pursuit that decomposes any signal into linear expansion of waveforms that are selected from a redundant dictionary of functions. A matching pursuit isolates the signal structures that are coherent with respect to given dictionary. I spent some time trying to find a good set of dictionary "atoms" aka  wavelets that would work well with noisy CW signals.

The details of this experiment is available here:
http://ag1le.blogspot.com/2014/05/sparse-representations-of-noisy-morse.html

This is a spectrum display showing a CW pileup before processing:


Here is the same pileup after MPTK processing:


Has anybody else been experimenting with MPTK  toolkit?  Any recommendations for wavelets?

73
Mauri AG1LE
Logged
PA0WV
Member

Posts: 141




Ignore
« Reply #1 on: May 11, 2014, 02:32:42 PM »

What is a "redundant dictionary of functions" ?

As far as I know you can expand a function in a series of orthagonal functions usually sine waves, but not necessarily, Some times a set of Bessel functions , or Walsh functions suits better.

Especially Walsh functions may be  interesting in CW decoding

« Last Edit: May 11, 2014, 02:36:06 PM by PA0WV » Logged

Using an appliance without CW is just CB
KI6LZ
Member

Posts: 602




Ignore
« Reply #2 on: May 11, 2014, 02:40:56 PM »

Wavelets are a whole different animal. Think they are also based on orthogonality but do not involve true sinusoidal functions.
Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #3 on: May 11, 2014, 03:37:12 PM »

What is a "redundant dictionary of functions" ?

As far as I know you can expand a function in a series of orthagonal functions usually sine waves, but not necessarily, Some times a set of Bessel functions , or Walsh functions suits better.

Especially Walsh functions may be  interesting in CW decoding
The mathematics behind the Matching Pursuit algorithm is explained in Mallat's paper:
blanche.polytechnique.fr/~mallat/papiers/MallatPursuit93.pdf.
As far as I understand the algorithm works also with a smaller dictionary of time-frequency atoms (like Gabor functions) but getting small enough error term becomes increasingly difficult. One of the parameters you can assign for MPTK toolkit matching pursuit decomposition software is the target signal-to-noise ratio. It may take either much longer time or you don't reach your target SNR without having some redundant functions in the dictionary of time-frequency atoms.

Walsh functions might be interesting to try out and see if they could improve the results. Perhaps some sort of combination of Walsh functions and Morlet wavelets might also work.  I did some experiments earlier with modified Morlet wavelets http://ag1le.blogspot.com/2012/06/morse-code-detection-using-modified.html and for very low SNR cases  they seemed to improve ability to extract CW signals. Matching Pursuit algorithm provides a great framework to find all CW signals in the baseband.

The nice thing about this MPTK toolkit is that it is written in C++ and is very fast. Right now it works with WAV files but it may be possible to tweak it so that it could be integrated with  FLDIGI  or something similar.  Being able to automatically extract all (even low SNR) CW signals from baseband would be a nice feature.

73
Mauri AG1LE



Logged
W0WCA
Member

Posts: 40




Ignore
« Reply #4 on: May 11, 2014, 04:39:19 PM »

You're kidding, right!
Logged
KI6LZ
Member

Posts: 602




Ignore
« Reply #5 on: May 11, 2014, 04:45:42 PM »

If they can use wavelets to identify fingerprints there should be a way to dig out signals in the noise.
Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #6 on: May 11, 2014, 06:02:46 PM »

You're kidding, right!

Unfortunately I missed April 1st deadline when preparing this Wink

If you want to confirm the results yourself I have posted links where you can download and compile the software, provided example WAV file and even what dictionary I was using to produce the results above.  I think this might be a useful tool for folks who are interested in latest noise reduction techniques and posted this to find out if anybody else in this ham radio community has been playing with MPTK toolkit. The dictionary "atoms" I used were clearly not optimized very well for CW. 

With ever increasing computing power and more capable software defined radios the good old CW  mode might also benefit of these fairly new signal processing methods, right? 

73
Mauri AG1LE

Logged
ZENKI
Member

Posts: 997




Ignore
« Reply #7 on: May 12, 2014, 02:20:37 AM »

Looks like a new mode in the making (DCW) digital CW. Maybe Joe Taylor is working on it.
Logged
N3QE
Member

Posts: 2426




Ignore
« Reply #8 on: May 12, 2014, 03:29:33 AM »

I think you have two obvious choices for sparse codes:

1: Sparse code consisting of three symbols: "Dit (including empty following dit-space)", "Dash (including empty following dit-space)", and "intercharacter space" (actually one dit short because taken up by previous dit or dah). This is how, for example, some old reperferators stored morse code onto paper tape. It may have been called "Wheatstone's Method" used for dense efficient coding onto transatlantic cables.

2: Sparse code consisting of 26 letters, 10 numbers, and a couple punctuation and common prosigns.
« Last Edit: May 12, 2014, 03:35:52 AM by N3QE » Logged
N3QE
Member

Posts: 2426




Ignore
« Reply #9 on: May 12, 2014, 06:39:05 AM »

I have done some of my own testing where I take wide-bandwidth audio recordings of 160M contest CW, and convolute them with "well known callsigns". In this case the "alphabet" of well-known callsigns are the most well-known 160M regulars. This is able to pull callsigns well out of the noise better than the human ear, but the computational power to do this (especially at varying WPM's and varying center frequencies) is nowhere near real-time.

I have also done the above with contest RTTY (and a different "alphabet" of well-known RTTY callsigns).

I think CW skimmer does something similar, but with a "broader" and "better" choice of callsigns and an algorithm that deals with varying center frequencies and varying WPM's. CW Skimmer obviously has very good callsign knowledge, and is working way above the "dit" and "dah" and "character" level, probably working much more like a human decodes iffy CW via probabalistic decision tree and pattern recognition rather than my brute force attempts. I observe it can also back up the decision tree and correct a previous decoding when it is obvious there's a better choice given by later data. Very impressive!
« Last Edit: May 12, 2014, 07:04:15 AM by N3QE » Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #10 on: May 12, 2014, 12:51:22 PM »

I have done some of my own testing where I take wide-bandwidth audio recordings of 160M contest CW, and convolute them with "well known callsigns". In this case the "alphabet" of well-known callsigns are the most well-known 160M regulars. This is able to pull callsigns well out of the noise better than the human ear, but the computational power to do this (especially at varying WPM's and varying center frequencies) is nowhere near real-time.

I have also done the above with contest RTTY (and a different "alphabet" of well-known RTTY callsigns).

Very interesting...do you have this software available somewhere? 

 
Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #11 on: May 18, 2014, 03:33:14 PM »

I think you have two obvious choices for sparse codes:

1: Sparse code consisting of three symbols: "Dit (including empty following dit-space)", "Dash (including empty following dit-space)", and "intercharacter space" (actually one dit short because taken up by previous dit or dah).

2: Sparse code consisting of 26 letters, 10 numbers, and a couple punctuation and common prosigns.

Thanks for sharing this idea. To express this "Dit" function in a formula, something like this:   

x =  step (Td - t)*sin ( Ts - Fs*t)

where
   step(x)  returns 1 if x is positive, zero otherwise (Heaviside step function)
   Td   is the duration of "dit" ( 30 ms in example below)   
   Ts   is start time value  (to ensure sin() function starts at 0 at t=0)
   Fs   is sampling frequency
   t     is  time  ( 0...90 ms in example below)

To illustrate the function,  see http://www.wolframalpha.com/share/clip?f=d41d8cd98f00b204e9800998ecf8427edb07r97on7

I will check if I can add this kind of "atom" to MPTK toolkit and re-run the tests.

73
Mauri  AG1LE
Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #12 on: May 19, 2014, 07:45:03 PM »

I did some more experiments  to demonstrate how this new Matching Pursuit method can reduce noise from CW signals.
New test results are here http://ag1le.blogspot.com/2014/05/noise-reduction-method-for-morse-signals.html

Original signal (4 second sample)


Noise reduced signal (4 second sample)


Based on the suggestion above I used new atoms in the dictionary and the results sound much better than in the original experiment.
I also tested the run time of this new algorithm - you can process WAV file in almost real time with a ThinkPad X301 laptop.

Anybody else willing to give this a test spin?  The software is open source and available for Windows, Linux and Mac platforms.

73
Mauri AG1LE
Logged
KI6LZ
Member

Posts: 602




Ignore
« Reply #13 on: May 19, 2014, 07:55:03 PM »

Looks like significant noise reduction but it also took out some weak signals.
Logged
AG1LE
Member

Posts: 139


WWW

Ignore
« Reply #14 on: May 19, 2014, 08:06:25 PM »

Looks like significant noise reduction but it also took out some weak signals.

Try to download the audio files and listen the difference.  I think it is possible to improve this method even further. 
I spent some 3 hours experimenting with different atoms and parameters and I am pretty sure that we can do much better.
Compared to traditional band pass filtering this method finds and extracts (almost) all CW signals in the base band while reducing noise.
Pretty cool, eh? 

73
Mauri AG1LE 
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!