eHam

eHam Forums => CW => Topic started by: AG1LE on September 22, 2013, 10:46:29 PM



Title: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 22, 2013, 10:46:29 PM
Back in June I presented some of my work related to Morse Code decoders at an event organized by New England Artificial Intelligence group.
Some 70+ people participated including couple hams.  We had some good conversations after the presentation. More details and presentation slides are in here:  http://www.meetup.com/intelligence/events/119478492/ (http://www.meetup.com/intelligence/events/119478492/). There is also a youtube video recorded from the event:  http://www.youtube.com/watch?v=ckjo-xw-s0w (http://www.youtube.com/watch?v=ckjo-xw-s0w)

I tried to highlight the huge gap there is between best human CW capabilities vs. computers running AI / machine learning algorithms.
This is probably nothing new to the active readers of this forum but the audience in above event seemed genuinely interested in solving the Morse decoding problem(s) and had some great suggestions.

I have been working on a new CW decoder that combines Bayesian approach with probabilistic models of channel fading, speed transition, timing variation, Morse and letter transitions. This software builds a trellis of alternative decoding paths sorted by probability, taking into account  calculated conditional probabilities. While there is a lot of computation done for each 5 ms sample, this approach seems quite promising in terms of decoding accuracy. Once I get some more test cases done I can publish the first results. The software is currently a standalone C program but could be easily integrated to other software like FLDIGI. Let me know if you are interested in this.       

73
Mauri AG1LE


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: N0IU on September 23, 2013, 04:19:54 AM
In the time it would take to write a program that could come anywhere close to human capabilities, you could learn Morse code yourself!  :D


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: W1JKA on September 23, 2013, 06:38:10 AM
Re: N0IU

    What you said is true, but consider that once the program IS written how much easier it would be for the non handicapped instant no code multiple guess amateur extra licensed hams with their out of the box appliance shacks to use their push button fingers in the CW mode and help keep the this ancient mode alive. ;)


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 23, 2013, 06:58:11 AM
In the time it would take to write a program that could come anywhere close to human capabilities, you could learn Morse code yourself!  :D
Scott

Listening ( & replaying) captured CW audio clips where the software has particular problems adapting to speed changes or some other issues has already improved my rusty CW copying skills.
I agree that spending more time on band could be faster way to learn but this project has helped me also to gain better understanding on machine learning / AI algorithms. There has been tremendous progress made in machine learning systems in the last 5 - 10 years - just think about "self driving cars"  or  "autonomous robots" like Mars rover. Some of these algorithms are available as open source for anybody to apply to another problem, such as Morse decoding. 

Part of attractiveness of ham radio is experimentation with new technologies, right?  This problem is not too difficult for a single individual to make some progress and learn by doing.
The nice side benefit is improving my own personal CW copying skills. And perhaps some other hams could find some use for the software as well?

73
Mauri     


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: WX2S on September 23, 2013, 07:53:22 AM
Quote from: Siri
I have no doubt that with sufficient effort and AI Morse code decoder will be in human alive. It's already happened with Chesson Jeffrey. By the way, I'm voicing this in with Siri, and you can see some of the aftereffects.

Translation:

I have no doubt that with sufficient effort an AI Morse code decoder will beat any human alive. It's already happened with Chess and Jeopardy. By the way, I'm voicing this in with Siri, and you can see some of the aftereffects.  ;D

73,
Wx2s.


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: N0IU on September 23, 2013, 09:26:25 AM
Mauri,

As someone who came up through the ranks "back in the day", I was really just giving you a hard time about developing a machine-based system that you could CW as well as an experienced operator. (That's why I put the smiley face on there!) I can certainly appreciate the fact that this is not only an amateur radio pursuit as much as it is an academic pursuit to try and see just how far AI can take us.


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: K9MRD on September 23, 2013, 10:14:22 AM
Mauri,

I have been a ham for 55 years and spend about 90% of my time on CW.   I would appreciate reading updates about this project.

Wayne
K9mrd


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: N3QE on September 23, 2013, 10:36:37 AM
The standard of comparison here, would be CW Skimmer.

In general I don't think the tools have to be as good as a trained human at single channel decoding. But at full-spectrum decoding... wow CW Skimmer is a powerful tool.

Very nice video of CW Skimmer from a cold start in a 160 test: http://www.youtube.com/watch?v=ZarmnghuQK8


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: KC8Y on September 23, 2013, 03:16:51 PM
I would like to see "CW Skimmer."

What is the website (URL) for the download ???


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: KC8Y on September 23, 2013, 04:23:51 PM
Found it :)


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AK4YH on September 23, 2013, 08:13:14 PM
If you can make it fit in an Arduino board...

Gil.


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: NI0C on September 23, 2013, 08:57:04 PM
Mauri,
In another thread, you mentioned you were interested in some comparisons between your software and human decoding.  When you develop some relevant sound files, please share them with us; I'll see what I can do with them (in the spirit of John Henry, Johnny cash's "steel drivin' man").
73,
Chuck  NI0C


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: NI8R on September 24, 2013, 03:03:01 AM
Everyone knows this stuff works in all emergency situations. No worries. Trust the software.



Greg kc8iir


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 27, 2013, 05:13:43 PM
Mauri,
In another thread, you mentioned you were interested in some comparisons between your software and human decoding.  When you develop some relevant sound files, please share them with us; I'll see what I can do with them (in the spirit of John Henry, Johnny cash's "steel drivin' man").
73,
Chuck  NI0C

Hi Chuck

I posted some sound files with -10 dB to +20 dB SNR @ 2kHz BW.  See my latest blog post on CER vs. SNR testing http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html (http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html).  The sound files are here: https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW (https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW).

73
Mauri AG1LE


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 27, 2013, 05:27:15 PM
If you can make it fit in an Arduino board...

Gil.
Gil
I don't have an Arduino board but the software is posted in Github: https://github.com/ag1le/morse-wip (https://github.com/ag1le/morse-wip).
It has currently about 3335 lines of code and has some F2C library dependencies that I am planning to remove.
It also requires a lot of cleanup work but I got it working the other night. First test results are here: http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html (http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html)

If you have time and energy perhaps this algorithm can be packaged to fit in Arduino footprint? 

73
Mauri AG1LE


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: NI0C on September 28, 2013, 09:15:51 AM
Mauri,
In another thread, you mentioned you were interested in some comparisons between your software and human decoding.  When you develop some relevant sound files, please share them with us; I'll see what I can do with them (in the spirit of John Henry, Johnny cash's "steel drivin' man").
73,
Chuck  NI0C

Hi Chuck

I posted some sound files with -10 dB to +20 dB SNR @ 2kHz BW.  See my latest blog post on CER vs. SNR testing http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html (http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html).  The sound files are here: https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW (https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW).

73
Mauri AG1LE

Mauri,

Many thanks for these sound files.  I just found your posting last night, so have not had much time to listen to them, but my first impression is that there's a huge difference in going from -10dB to -8dB, and to a lesser extent to -6dB.  I'm not good at copying a continuous stream of 50 wpm code groups , but I am able to "grab" some of them and get them down (similar to copying single code signs on Rufz or CWFreak).   I'll try to give you something more quantitative on your blog.

One suggestion might be to record some 20 or 25 wpm code groups, to get more human participants in your experiment.  This might also help you separate out the error rate due to SNR from error rate due to reaching the upper limits of a human subject's code proficiency.

73,
Chuck  NI0C



Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 28, 2013, 06:27:51 PM

One suggestion might be to record some 20 or 25 wpm code groups, to get more human participants in your experiment.  This might also help you separate out the error rate due to SNR from error rate due to reaching the upper limits of a human subject's code proficiency.

73,
Chuck  NI0C

Chuck
I posted  a set of files  -10db ...20dB SNR @2kHz at 20 WPM speed.  Also, I checked again - the original files had 40 WPM speed. 
The file names are  rand<X>db<Y>wpm.wav where X is  SNR and  Y is WPM, files are here https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW (https://www.dropbox.com/sh/2j9chxsizxjeqgp/Gj97asYzpW)

Do these work for you?

73
Mauri AG1LE


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: NI0C on September 29, 2013, 10:19:37 AM
Mauri,

Thanks for the new set of files.  I spent some time this morning with the 20 wpm set (20 wpm is about my typing speed!) and was able to do solid copy (essentially zero CER) on the -6 dB and stronger signals. My CER was around 5% at -8 dB, but I fell apart at -10 dB, with only perhaps 50% copy.  This is a very preliminary "quick look," based on copying only a minute or so from each file, (I didn't bother with any of the files stronger than + 6 dB). 

I'm sure there are others with better "ears" than mine who could do better, and perhaps at lower signal to noise ratios.  Your -10 dB files at 20 and 40 wpm are good practice files for me, and might help me train my ears. 

73,
Chuck  NI0C   


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AG1LE on September 29, 2013, 11:17:17 AM
I spent some time this morning with the 20 wpm set (20 wpm is about my typing speed!) and was able to do solid copy (essentially zero CER) on the -6 dB and stronger signals. My CER was around 5% at -8 dB, but I fell apart at -10 dB, with only perhaps 50% copy.  This is a very preliminary "quick look," based on copying only a minute or so from each file, (I didn't bother with any of the files stronger than + 6 dB). 
Thanks Chuck!
 
I will use these data points as reference. Human auditory system seems to have a pretty good filtering capability what comes to Morse code type signals. To achieve similar results with this software I would need to get roughly 16 dB improvement by filtering the original signal that has 2KHz bandwidth. Since the noise in these files is white noise 16 dB translates to roughly 50 Hz bandwidth [16 dB = 10*log10(2000/50)].

I did some work back in January trying to compare the impact of matched filtering vs. FFT filtering in FLDIGI. The plot below shows the big improvement that filtering brings on CER figures:
(http://2.bp.blogspot.com/-yoavzbN1H6Q/UOOwwIzugOI/AAAAAAAAAYA/vWP8hFjtrFM/s640/Morse+BER+vs+SNR+tests.png)

Once I get this new algorithm integrated better with FLDIGI I can re-run the above CER vs. SNR tests to see how much improvement the Bayesian probabilistic correlator-estimator provides. According to the doctoral thesis I used as the source for this algorithm there should be ~ 4 ... 6 dB improvement.

Thanks again for taking the time for this experiment!

73
Mauri AG1LE


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: NI0C on September 30, 2013, 06:05:32 AM
Mauri,
I wish you continued success with your project.  Although I've taken a couple of graduate courses in digital signal processing, that was over 30 years ago, and I'm pretty rusty.  Nevertheless, it was fun helping you out in a small way using manual CW copying skills that I have tried to maintain.
73,
Chuck  NI0C


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: W5LZ on September 30, 2013, 04:01:52 PM
The human brain/ear can make an enormous amount of 'adaptions' that no 'AI' can do (at present).  That 'wet' computer can associate what it hears with what should have been heard where a set of algorithms can't unless it's a humongus number of algorithms.  I wish you good luck with your project.
 - Paul


Title: RE: Wetware vs. AI Software - Pursuit of Ultimate Morse Decoder
Post by: AK4YH on September 30, 2013, 10:59:41 PM
Quote
I don't have an Arduino board but the software is posted in Github: https://github.com/ag1le/morse-wip.

Thanks Mauri, I will keep an eye on it. Very interesting in any case. I use a Bayesian filter for spam deletion and it works great, so... Your program being in C it should be adaptable to many platforms.. An arduino has 32K or program memory.. I'me just starting messing with it..

Gil.