Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: [1] 2 3 Next   Go Down
  Print  
Author Topic: New Bayesian Morse decoder available for alpha testing  (Read 12265 times)
AG1LE
Member

Posts: 119


WWW

Ignore
« on: December 24, 2013, 07:01:55 PM »

I have been working on a new Bayesian Morse decoder for FLDIGI for a while.
The goal was to have a CW decoder that adapts better to different operators rhytms, sudden speed changes, signal fluctuations and noise and still is able to decode Morse code fairly accurately.

While this software is not perfect and has some bugs I am happy to release this brand new version for alpha testing.  I finished the software integration to FLDIGI v3.21.75 and spent few hours today listening bands.  Below is a small example of W1AW code bulletin at 3.58105 MHz  received with KX3 and a side by side comparison of legacy CW decoder  vs  Bayesian decoder:
http://ag1le.blogspot.com/2013/12/new-morse-decoder-part-2.html

Legacy  vs. Bayesian decoder side by side:


The full source code is available in here: http://www.w1hkj.com/ag1le

I would like to get some alpha testers who are not afraid of compiling this code from sources,  playing with it, breaking it and hopefully also helping me to find the bugs. For example  - the noise level estimator (noise.c) needs some serious work.  If you have a algorithm that does the job better this would be one place to start improving.  Also, if you know Viterbi or Kalman filter algorithms like your pockets I would like to get help fixing a bug that appears in posterior probability calculations in probp.c occasionally.

Anyways,  happy holidays for all FLDIGI users and hope to work you over the radio.

73
Mauri AG1LE
Logged
W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #1 on: December 24, 2013, 09:50:04 PM »

Looks very interesting. I'd like to play with it on this Xubuntu machine.

Although I just compiled tlf logger from source a couple weeks ago, this source wouldn't configure. Looks like I need to install some packages first. Another day though; it's bedtime here. :-)
Logged

AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #2 on: December 25, 2013, 02:06:47 PM »

Looks very interesting. I'd like to play with it on this Xubuntu machine.

Although I just compiled tlf logger from source a couple weeks ago, this source wouldn't configure. Looks like I need to install some packages first. Another day though; it's bedtime here. :-)
Thanks Mike

I am using Linux Mint 15 as my development environment. What packages are missing on your system?
Once you get this compiled let me know if you have trouble getting the decoder working. 

It seems to work better at lower signal levels - I can provide some test WAV files if you don't have radio interface to your Xubuntu machine.


73
Mauri AG1LE
Logged
W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #3 on: December 25, 2013, 02:45:39 PM »

Hi Mauri,

I installed all the missing packages. It's compiling right now. I'll keep you posted.

I have had fldigi 3.21.38 installed and working for some time. I shouldn't need any .wav files.

I will tell you that I am no C expert. I used to code a lot in Pascal, but that's been years ago. Never wrote anything in C.
Logged

W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #4 on: December 25, 2013, 03:10:13 PM »

Ok, I have it compiled and running. :-)

I think I need to change some things in the configuration. It decodes CW, but when I check the Bayes decoding, it doesn't copy CW as well. The version I had copied CW better. I was temped to change the configuration, but I don't have much more time to play with it this week. If you could suggest how best to configure it, I'll try it again.

I see the old fldigi version has the Bayes decoding box. I didn't see that before. I don't have it checked.

EDIT: I see the Help > About in both versions shows v. 3.21.75. Oops. I shouldn't have installed the new version where I did. The config is still different in both versions; I don't know what's going on.

I could post or e-mail the config files or a screenshot. Let me know what you suggest or need.
« Last Edit: December 25, 2013, 03:14:21 PM by W0BTU » Logged

W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #5 on: December 25, 2013, 06:05:52 PM »

I played with it a little more. It does not copy CW correctly at all when Bayes is checked. Unchecking it, it copies CW very well.

I tried turning the volume down as you suggested, and no change.

Perhaps if I changed the configuration?
Logged

AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #6 on: December 25, 2013, 06:32:17 PM »

I played with it a little more. It does not copy CW correctly at all when Bayes is checked. Unchecking it, it copies CW very well.

I tried turning the volume down as you suggested, and no change.

Perhaps if I changed the configuration?

Mike

Can you verify that before you switch on Bayesian decoder you have Configure / Modems / CW / General / "Matched Filter" checked and  "FFT Filter" not checked?
Also, please  check "Tracking" on.   After this is verified please click "Bayesian decoder"  on and observe  lower left corner.  If you see "CW Rx  NN"  where speed NN is changing rapidly then the new decoder is working.  As mentioned this version is still very sensitive to volume and signal/noise level.  In my system I am adjusting KX3   RF volume (I have AGC turned off) so that CW signal is barely visible on the waterfall.

There is a bug that shows up by turning CW Rx speed indicator negative  - if you get into this situation you need to restart the application. Typically this happens when for some reason the likelihood calculations in "probp.c" exceed 1.0  which on the next few sampling rounds quickly amplify pmax to a large number.  I have not found the root cause for this bug yet, but seems to be somehow related to signal strength and SNR of the signal.  The noise power estimator "noise.c" is one suspect...not sure if this part really works.

Thanks for helping out to debug this software.

73
Mauri AG1LE

 
Logged
W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #7 on: December 25, 2013, 06:50:48 PM »

Can you verify that before you switch on Bayesian decoder you have Configure / Modems / CW / General / "Matched Filter" checked and  "FFT Filter" not checked?
Also, please  check "Tracking" on.   After this is verified please click "Bayesian decoder"  on and observe  lower left corner.  If you see "CW Rx  NN"  where speed NN is changing rapidly then the new decoder is working.  As mentioned this version is still very sensitive to volume and signal/noise level. ...

That worked. :-)

EDIT: Screenshot at
http://www.w0btu.com/files/misc/software/misc/
http://www.w0btu.com/files/misc/software/misc/fldigi_beta-1.png

I wonder if Nate, N0NB would enjoy helping?
« Last Edit: December 25, 2013, 07:04:42 PM by W0BTU » Logged

AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #8 on: December 26, 2013, 06:10:24 AM »

Quote
That worked. :-)

EDIT: Screenshot at
http://www.w0btu.com/files/misc/software/misc/
http://www.w0btu.com/files/misc/software/misc/fldigi_beta-1.png

I wonder if Nate, N0NB would enjoy helping?

Thanks Mike
This is great news...your are the first person reporting successful Bayesian decode.
As you play with this version I would appreciate your feedback on things like

  • * Signal strength of successful / non-successful decoding
  • * CW speed - any limits you find when decoder stops decoding
  • * QSB - how does signal fluctuation impact decoding accuracy
  • * Waveforms - using the Scope feature of FLDIGI - any findings on better/worse decoding

Any screenshots  and WAV file captures to document findings would be welcome.

The software is still fragile, will break easily and is not fully integrated with FLDIGI user interface.
You will most likely find the "pmax overflow" bug - symptoms include  CW Rx speed indicator on bottom left corner showing a large negative number, sometimes it goes to a mode where you get only series of letter  E E E E E  etc. You need to restart FLDIGI application as it does not recover from this bug yet.

This is a fairly complex piece of software and it requires still quite a lot of work to make it really useful for the ham community.

Thanks again for all your help and support.

73
Mauri AG1LE
Logged
W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #9 on: December 26, 2013, 06:23:07 AM »

Glad to help, but I don't have as much time as I would like to. The Stew Perry contest is this weekend and I have some antennas to work on and new logging software to finish setting up.

I hope you can find others to help you as well. Perhaps if you posted a link to this thread in the Computers and Software forum here and on qrz.com, you will attract others.
Logged

AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #10 on: December 26, 2013, 11:17:15 AM »

I hope you can find others to help you as well. Perhaps if you posted a link to this thread in the Computers and Software forum here and on qrz.com, you will attract others.

Great idea  - I did a cross posting in the Computers and Software Forum.

73
Mauri AG1LE
Logged
AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #11 on: December 31, 2013, 09:00:48 PM »

Thanks to the alpha testers I have received both feedback and some test audio files to hunt down bugs in this Bayesian Decoder software.

I analyzed in details of the behavior that a few alpha testers have reported.
Here is a  summary of the problem:
http://ag1le.blogspot.com/2013/12/new-morse-decoder-part-3.html

In short the speed estimator part of the algorithm that should track and find correct WPM speed on received signals like this:


is misbehaving when decoder gets too large signals.  The threshold when this happens is very sharp and non-linear - here is error case:


I have converted all the files from C  to C++  to make it easier to re-integrate to FLDIGI.  I will continue cleaning up the code and keep working to improve the speed estimator part.  Once I have the next alpha release ready for another set of tests I will announce it in the fldigi mailing lists and eham.net CW forum.

73 and Happy New Year
Mauri AG1LE
Logged
AD0AE
Member

Posts: 78




Ignore
« Reply #12 on: January 02, 2014, 09:31:09 AM »

This is a very basic question, but in what respect is the decoder Bayesian?  What information is being used and 'updated?'

(Forgive me also, I have not fully read this thread)

73s,
Steve
AD0AE
Logged
W0BTU
Member

Posts: 1621


WWW

Ignore
« Reply #13 on: January 03, 2014, 02:01:25 PM »

I can't test this anymore. I screwed something up when I was playing with the mixer.

Maybe someone can give me some advice. I don't know where to begin to fix this.

I was trying to find the setting in the mixer that would let me adjust the signal level to running applications. In the process, I managed to "kill" the audio the signal to any running applications, including fldigi and qsstv.

The sound from my rig still comes out the computer speakers, as it always has. But that's all.

I restored the old /home/mike/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-mixer.xml but it didn't fix it, even after a reboot. (/home/ is the only backup I have.)

Any suggestions?  I'm using Xubuntu 12.04.3 LTS (Precise), which is Ubuntu with the XFCE 4.8 GUI.


I don't know exactly what Bayesian means, either. I'll concern myself with that after this audio issue is solved. :-)
« Last Edit: January 03, 2014, 02:04:16 PM by W0BTU » Logged

AG1LE
Member

Posts: 119


WWW

Ignore
« Reply #14 on: January 03, 2014, 06:52:23 PM »

This is a very basic question, but in what respect is the decoder Bayesian?  What information is being used and 'updated?'
Hi Steve
The decoder has several built-in models, such as
  •    * baseband signal channel model
  •    * keystate model
  •    * speed transition model
  •    * Morse symbol transition model
  •    * letter transition model
As the decoder receives samples of filtered audio signal after enveloper detector  (currently at rate of 200 Hz) it updates these models and calculates transition probabilities using Bayesian framework. Sequence of all possible keystate transitions are hypothesized and correlated with the incoming signal , and the most likely sequence is the output as the best estimate.

The complete description is actually much more complicated but essentially this decoder uses Bayesian methods to learn from incoming signals and update models accordingly. More details can be found from here: http://ag1le.blogspot.com/2013/09/new-morse-decoder-part-1.html



Logged
Pages: [1] 2 3 Next   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!