Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

donate to eham
   Home   Help Search  
Pages: [1] 2 Next   Go Down
  Print  
Author Topic: Morse Learning Machine Challenge  (Read 24358 times)
AG1LE
Member

Posts: 149


WWW

Ignore
« on: September 03, 2014, 06:07:10 PM »

My Morse Learning Machine Challenge was approved by Kaggle today.
http://ag1le.blogspot.com/2014/09/morse-learning-machine-challenge.html

The goal of this competition is to build a machine that learns how to decode audio files containing Morse code. I hope to attract people from the Kaggle community http://en.wikipedia.org/wiki/Kaggle who are interested in solving new, difficult challenges using their predictive data modeling, computer science and machine learning expertise.

During the competition, the participants build a learning system capable of decoding Morse code. To that end, they get development data consisting of 200 .WAV audio files containing short sequences of randomized Morse code. The data labels are provided for a training set so the participants can self-evaluate their systems. To evaluate their progress and compare themselves with others, they can submit their prediction results on-line to get immediate feedback. A real-time Kaggle leaderboard shows participants their current standing based on their validation set predictions.

I have also provided  sample Python Morse decoder  to make it easier to get started. While this software is purely experimental version it has some features of the FLDIGI Morse decoder  but implemented using Python instead of C++.

Please help me to spread this message to attract participants for the Morse Learning Machine challenge!
https://inclass.kaggle.com/c/morse-challenge

73
Mauri AG1LE
Logged
PA0WV
Member

Posts: 344




Ignore
« Reply #1 on: September 04, 2014, 02:09:27 AM »

What you want is a machine that make statistics of signal mark and space length.
When done and there is an obvious separated probably density function for each "sort" marks and spaces (will be 5 'sorts')
You can make an alphabet of all used combinations separated on the middle and the longest space sorts.

Finally you end up with a Morse alphabet, (all used separate Morse characters in the transmission) which is possibly upset when speed is changed, BUT not having   knowledge what Latin or Japanese character is coupled to what Morse code.

That is a prototype of a standard speed independent Morse decoder, with a missing translation table, isn't it?

The complete one programmed for international Morse code you can find on my website
http://pa0wv.home.xs4all.nl/pdfbestanden/Veron.pdf
« Last Edit: September 04, 2014, 02:13:19 AM by PA0WV » Logged

AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #2 on: September 04, 2014, 04:04:57 AM »

What you want is a machine that make statistics of signal mark and space length.
When done and there is an obvious separated probably density function for each "sort" marks and spaces (will be 5 'sorts')
You can make an alphabet of all used combinations separated on the middle and the longest space sorts.

Finally you end up with a Morse alphabet, (all used separate Morse characters in the transmission) which is possibly upset when speed is changed, BUT not having   knowledge what Latin or Japanese character is coupled to what Morse code.

That is a prototype of a standard speed independent Morse decoder, with a missing translation table, isn't it?

The complete one programmed for international Morse code you can find on my website
http://pa0wv.home.xs4all.nl/pdfbestanden/Veron.pdf
Hi Wim
Thanks for the link - very informative article (after I used Google translator on it).
Building statistics on  mark/space pairs is certainly one element of the decoder.
I tried similar approach in here http://ag1le.blogspot.com/2013/02/probabilistic-neural-network-classifier.html

If you plot mark/space pairs from real world noisy CW you get a chart  like this

Having a classifier that accurately figures out each and every mark/space pair is bit tricky.
I have used Probabilistic Neural Network (PNN)  in this experiment and also in the Kaggle challenge example Python decoder.

I am also happy to report that KR1LLR has entered the Morse Learning Machine Challenge after this little Reddit comment chain
http://www.reddit.com/r/amateurradio/comments/2fexx6/morse_learning_machine_challenge/

I am looking for European challengers,  any takers?

73
Mauri AG1LE
Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #3 on: September 04, 2014, 02:52:58 PM »

Within the first 19 hours the Morse Learning Machine Challenge https://inclass.kaggle.com/c/morse-challenge has created quite unexpected amount of interest! 

My ham radio blog http://ag1le.blogspot.com/2014/09/morse-learning-machine-challenge.html has had all time record of hits - 718 unique views today. 
Also,  I see many new people signed up to Kaggle and downloaded the sample materials, including the Python Morse decoder I created for this competition.

I can also see cross-posts around the world, such as this https://plus.google.com/+HajoDezelski/posts/FUGgqWFGn1y and
this http://www.qrpforum.de/index.php?page=Thread&postID=74755#post74755.

Thanks so much eHam community to spread the message!

73
Mauri AG1LE


 
Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #4 on: September 04, 2014, 09:35:22 PM »

I am very impressed on the progress in this competition.   Order of magnitude improvement in the leaderboard https://inclass.kaggle.com/c/morse-challenge/leaderboard/public score in less than 24 hours since the competition was launched!

73
Mauri AG1LE
Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #5 on: September 05, 2014, 07:00:48 PM »

On the 3rd day of the Morse Learning Machine Challenge https://inclass.kaggle.com/c/morse-challenge
we have a new record score 0.35 on the public leaderboard: https://inclass.kaggle.com/c/morse-challenge/leaderboard

Tremendous progress in very short time!

73
Mauri  AG1LE
Logged
N4OI
Member

Posts: 333




Ignore
« Reply #6 on: September 06, 2014, 06:46:32 AM »

While this is fascinating from a computer science viewpoint, wouldn't it make more sense to apply all this effort to invent or improve some true, error detecting/correcting digital mode instead of CW?

Just sayin'. 73
Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #7 on: September 06, 2014, 11:14:52 AM »

While this is fascinating from a computer science viewpoint, wouldn't it make more sense to apply all this effort to invent or improve some true, error detecting/correcting digital mode instead of CW?
The whole point of this exercise is that CW is a human language.  

While PSK31, JT65A etc. are much better digital modes from error detecting/correcting perspective,
(skilled) humans can copy CW  in their head, much like we copy speech, almost unconsciously.

Current generation of CW decoder software doesn't match with human's ability to adapt, fill in gaps,
deal with timing variances, ignore sending errors, copy despite propagation variations etc.  
Skilled humans have some cortical 'algorithms' learned by thousands of repetitions that
puts humans lightyears ahead of current capabilities of computer software.

The point of this exercise is to figure out these 'cortical algorithms', generalize them  
and create learning machines that you can teach other skills. Having a general purpose
learning machine opens up  many new application areas that we probably cannot even imagine yet.

CW has many features that makes it an ideal testing ground for this kind of development, such as
  • * Morse code has very simple rules, so getting started is easy - just few lines of code
  • * You can start from first principles and iteratively add more complexity as you learn more
  • * You have free, fresh training material available from many humans  by just opening your radio
  • * CW is used in so many different QSO contexts (contest, pile-up, ragchew, etc)
        that makes it really hard to apply rule based systems, so learning based approach is more plausible
  • * With CW you have many benchmarks, for example verified contest logs and audio recordings


I think we are quickly approaching an era where these 'general purpose learning machines' start to appear,
a bit like in 1970's when the first PCs arrived. General population didn't really understand back then why anybody
would need a computer at home.  Roll forward  3-4 decades and now everybody has not just one,
but several with different form factors (desktop, laptop, tablet, smartphone, ...)

Does this make any more sense now?

73
Mauri AG1LE

« Last Edit: September 06, 2014, 12:15:50 PM by AG1LE » Logged
AC2EU
Member

Posts: 1144


WWW

Ignore
« Reply #8 on: September 06, 2014, 11:40:51 AM »

I went to the kaggle site and I still don't understand what it is.
So in the case of the morse decoder there is no real contest or anything to win?
Even if there was, from I what I could gather, you get "bragging rights" as prize?

Could you tell me what Kaggle is/does?
It looks like a good way to get your your ideas and hard work ripped off.
What I am missing?



Logged

AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #9 on: September 06, 2014, 12:00:59 PM »

I went to the kaggle site and I still don't understand what it is.
So in the case of the morse decoder there is no real contest or anything to win?
Even if there was, from I what I could gather, you get "bragging rights" as prize?

Could you tell me what Kaggle is/does?
It looks like a good way to get your your ideas and hard work ripped off.
What I am missing?

Why do people spend thousands of $$$ on radios and antennas, practice CW hundreds of hours and participate CQ WW and other ham contests?   Is it because they expect to get big $$$ rewards or  celebratory bragging rights?

Kaggle is promoting data science as a sport.  Just like in ham radio individuals and teams around the world are interested in participating data science contests and challenges.  There are some competitions (mostly corporate sponsored) where you can win big $$$ and negotiate a IPR licensing deal with the sponsor.   There are also academic competitions where the winner can travel to a conference to present his methods in front of his peers. There are competitions that are more tutorials, for learning new knowledge.

Since this is first ever MLM challenge I positioned this between academic and tutorial type contest.  I gave all my IPR away for free with GPLv3 license so I requested participants to do the same.  Perhaps in the future if this gathers more momentum we can get some corporate sponsors  (ham gear makers, ARRL, etc.)  or academic sponsors (Universities with data science/machine learning/signal processing curriculum)  and launch MLM v2 contest with some real $$$ prizes.

In this MLM v1  you definitely will get worldwide bragging rights if you manage to solve the challenge.
It is doable, but I tried to make it difficult enough...  BTW,  you need to hurry if you want to win,  Aaron KR1LLR has a big head start - his leaderboard http://inclass.kaggle.com/c/morse-challenge/leaderboard/public  score is already at 0.35.



73
Mauri AG1LE
Logged
STAYVERTICAL
Member

Posts: 875




Ignore
« Reply #10 on: September 06, 2014, 03:12:16 PM »

A generalised machine learning algorithm is a myth.

Nature is resplendent with examples of organisms - each of which employs novel ways of adapting to its unique environment.
Even humans are in this category, put a native New Yorker in the backwoods and see how long they last.
The field of AI is like the event horizon of a black hole - time gets progressively slower, until it stops altogether in the singularity.
Its mantra " real AI in another 10 years!" has been ringing in my ears for 40 years.
There is no magic algorithm in the vein of Wolfram et al, which describes the world, the universe and everything.

As a longtime CW user I can give you insights into how CW is decoded in the human brain.
It is not a simple signal processing problem of dits and dahs, but - as in human speech recognition, refers to the knowledge base
accumulated over the users life.

For example, in a typical CW QSO with a friend, I will use my knowledge of that persons interests, life, style of sending,
to correct for errors in sending, interpolate meaning from missing characters, and essentially "fill in the gaps".

When doing contest or "touch and go" DX qso's, I will switch to automated protocol based CW operation where I interpolate,
throw out redundant information, and generally follow a recipe book .
This is not to indicate dis-interest - but rather to gather essential information, as speed of communication is the desired outcome.
I may also revert to using abbreviations e.g. using OP to indicate my name instead of NAME.

The point I am making in sharing this information, is that CW is a LANGUAGE and is treated as such by experienced CW ops.
Most experienced CW operators will also just read CW in their head, and can frequently do other tasks at the same time.
This indicates the brain has accomodated CW into its neural circuitry, just as any other second language.

Listening to CW becomes essentially effortless, even in extremely poor conditions.
Any program will read CW flawlessly "in the clear", but most CW operation is far from that ideal situation,
which is why it is vital to incorporate true intelligence and life experience into the equation.

I have had a deep and abiding interest in AI and related technologies for many decades and have watched its development
with great interest.

Do not interpret my post as a slap down of the concept of machine reading of CW, but rather an attempt to have you
avoid the AI black hole, and steer your project into more productive waters.
Were I to be doing something like this, I would produce either an expert system or markov chain to provide table referenced
rules to be executed apon the reception of CW words.
Essentially, I would provide a layered AI approach similar to the approach used in network protocols where the physical and
data layers would represent the "best" bit interpretation, and then be passed up to the application layers for further processing
and interpretation.

You should also populate your expert system or similar by talking with and observing how experienced CW operators work.
Otherwise, you are trying to simulate a system by your interpretation of how it works, which I assure you is a track to failure.

Also, I think I would use C++ or some other such language to write such a system, as Python is a nice language,
but hardly the best for such a project.
Many freeware compilers abound for C++, so the cost would be minimal.
Also, if you are going to implement neural nets, the object oriented capability intrinsic in C++, and its speed,
would serve you well.

Good luck,

Regards,

Rob

Logged
N4OI
Member

Posts: 333




Ignore
« Reply #11 on: September 06, 2014, 05:26:45 PM »

[..]
The field of AI is like the event horizon of a black hole - time gets progressively slower, until it stops altogether in the singularity.
Its mantra " real AI in another 10 years!" has been ringing in my ears for 40 years.
[...]

Yep -- I recall involvement with TI and their AI LISP systems in the 1980s.  Ten years seemed like a technological eternity back then.   I mean, who would have thought in the 1970s that all that mainframe code would still be around long enough to require a Y2K rewrite?  And by now we would all be... the Jetsons! 

Frankly, the last 35 years have produced little more than big "easy to use" programming platforms that chew up all the advancements in hardware.  It was amazing what was done with machine code (1s and zeroes) to solve real problems back in the day...

Hal, I think I want to watch "2001-A Space Odyssey" now....

73  Grin
Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #12 on: September 06, 2014, 06:53:04 PM »

A generalised machine learning algorithm is a myth.

Nature is resplendent with examples of organisms - each of which employs novel ways of adapting to its unique environment.
Even humans are in this category, put a native New Yorker in the backwoods and see how long they last.
The field of AI is like the event horizon of a black hole - time gets progressively slower, until it stops altogether in the singularity.
Its mantra " real AI in another 10 years!" has been ringing in my ears for 40 years.
There is no magic algorithm in the vein of Wolfram et al, which describes the world, the universe and everything.
...

The point I am making in sharing this information, is that CW is a LANGUAGE and is treated as such by experienced CW ops.
Most experienced CW operators will also just read CW in their head, and can frequently do other tasks at the same time.
This indicates the brain has accomodated CW into its neural circuitry, just as any other second language.

Listening to CW becomes essentially effortless, even in extremely poor conditions.
Any program will read CW flawlessly "in the clear", but most CW operation is far from that ideal situation,
which is why it is vital to incorporate true intelligence and life experience into the equation.

I have had a deep and abiding interest in AI and related technologies for many decades and have watched its development
with great interest.

Do not interpret my post as a slap down of the concept of machine reading of CW, but rather an attempt to have you
avoid the AI black hole, and steer your project into more productive waters.
Were I to be doing something like this, I would produce either an expert system or markov chain to provide table referenced
rules to be executed apon the reception of CW words.
Essentially, I would provide a layered AI approach similar to the approach used in network protocols where the physical and
data layers would represent the "best" bit interpretation, and then be passed up to the application layers for further processing
and interpretation.

You should also populate your expert system or similar by talking with and observing how experienced CW operators work.
Otherwise, you are trying to simulate a system by your interpretation of how it works, which I assure you is a track to failure.


Thanks for your insightful comments.  I agree the progress in last 40 years in AI has been disappointing. I have personally been experimenting only last 30 years so perhaps I don't have enough perspective. I did some work in early 80's with Kohonen Self Organizing Maps (SOM) which was one of the early clustering algorithms we used to recognize phonemes etc.

However, I would claim that over the last 8+ years  much progress has been made especially in the 'deep' neural networks area.
Geoffrey Hinton and other researchers working in that area have produced algorithms that are now breaking old records by large margin in multiple applications areas, ranging from handwritten text, speech and object recognition areas to natural language processing.  Multilayer 'deep' neural networks  are emerging as 'generic' architecture and as building blocks for systems where semantic meanings can be extracted from large amount of unlabeled data.  See for example the Collobert & Weston paper on "NLP from Scratch" http://arxiv.org/pdf/1103.0398.pdf  where 631 million words from Wikipedia was processed with some 2500 lines of C code and system using 150MB RAM  and this 'generic system' successfully met or exceeded benchmarks multiple task specific systems. The Google speech recognition backend systems for Android  are also using 'deep' neural network architecture trained with thousands of hours of training voice data. These systems are now getting good enough for normal consumers to start using them.

I don't think that there is an 'universal magic algorithm' that solves all the problems in the universe. However, I do believe that we are now seeing some very powerful tools  combined with a large  (or really large)  training set of data will yield more human like capabilities.  Not yet  2001 Space Odyssey HAL type features but rather ability to start predicting words or 'filling the gaps'  like you do with your own 'neural circuitry' when listening CW.  And we would be able to do this despite noise,  interference and poor conditions.

On the software languages etc. - many of publicly available NN libraries in this field have been written with C or C++ for efficiency reasons,  but there is also a lot of  Python,  Matlab (or Octave) or other scripting language front ends as these make it much easier to rapidly prototype and test ideas.  With latest multicore 3 Ghz CPUs  and 16 GB RAM machines you can handle very large datasets, and if you need more capacity it is very easy and cheap to spin up Amazon EC2 cluster to do some heavy lifting these days.

Am I being too ambitious or naive in this quest or does this seem a feasible approach?

73  Grin

Logged
N4OI
Member

Posts: 333




Ignore
« Reply #13 on: September 07, 2014, 04:01:16 AM »

[...]The Google speech recognition backend systems for Android  are also using 'deep' neural network architecture trained with thousands of hours of training voice data. These systems are now getting good enough for normal consumers to start using them.[...]

I just realized I don't remember the last time I saw anyone still using Siri on their iPhone...  It seemed pretty cool at first. Huh

73

Logged
AG1LE
Member

Posts: 149


WWW

Ignore
« Reply #14 on: September 10, 2014, 05:22:59 AM »

Morse Learning Machine Challenge made it to ARRL news pages today!

http://www.arrl.org/news/morse-learning-machine-challenge-catching-on-with-hams


PS.  Day 5 of the challenge and many more people have downloaded the competition materials, I have already 155 downloads as of this morning. It looks like it takes a few days for people to get started with this challenge. There hasn't been many challenges focused on audio specific machine learning tasks on Kaggle. 
« Last Edit: September 10, 2014, 05:28:01 AM by AG1LE » Logged
Pages: [1] 2 Next   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!