eHam

eHam Forums => Computers And Software => Topic started by: KY6R on December 06, 2013, 08:07:22 AM



Title: Massive Scalability and Ham Radio Logging
Post by: KY6R on December 06, 2013, 08:07:22 AM
I have loaded my logbook onto a massively scalable database that my company makes - which is called Splice Machine.

Splice Machine runs on top of Hadoop and HBase and uses the Derby SQL Interface and JDBC drivers.

Just for laughs, I loaded my logbook into Splice Machine into a 4 node AWS Cloudera Hadoop cluster, and used Pentaho PDI to load the data in milliseconds. I also used Tableau to do advanced data visualization - and its really cool - and really fun.
I just wanted to see if LOTW or Clublog could take advantage of such a platform, and the answer is, Yes.

This is much too big for even the largest DX-pedition, its in the LOTW class of logbooks!

73,

Rich
KY6R


Title: RE: Massive Scalability and Ham Radio Logging
Post by: KB4QAA on December 06, 2013, 10:05:33 AM
Quote
...Splice Machine; Hadoop; HBase; Derby SQL; JDBC; 4 node AWS Cloudera Hadoop cluster; Pentaho PDI; Tableau; LOTW; Clublog...

What can a fellow say but, "Whoop-di-doo!".   LOL. thanks.  bill


Title: RE: Massive Scalability and Ham Radio Logging
Post by: W0FM on December 17, 2013, 01:00:12 PM
Quote
Splice Machine into a 4 node AWS Cloudera Hadoop cluster, and used Pentaho PDI to load the data in milliseconds. I also used Tableau to do advanced data visualization - and its really cool - and really fun.

Huh?  I think you missed the cymbal crash right there between the bridge and the second chorus, Rich!    ;D

Terry, WØFM