Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: [1]   Go Down
  Print  
Author Topic: Massive Scalability and Ham Radio Logging  (Read 3763 times)
KY6R
Member

Posts: 3236


WWW

Ignore
« on: December 06, 2013, 08:07:22 AM »

I have loaded my logbook onto a massively scalable database that my company makes - which is called Splice Machine.

Splice Machine runs on top of Hadoop and HBase and uses the Derby SQL Interface and JDBC drivers.

Just for laughs, I loaded my logbook into Splice Machine into a 4 node AWS Cloudera Hadoop cluster, and used Pentaho PDI to load the data in milliseconds. I also used Tableau to do advanced data visualization - and its really cool - and really fun.
I just wanted to see if LOTW or Clublog could take advantage of such a platform, and the answer is, Yes.

This is much too big for even the largest DX-pedition, its in the LOTW class of logbooks!

73,

Rich
KY6R
« Last Edit: December 06, 2013, 08:10:09 AM by KY6R » Logged
KB4QAA
Member

Posts: 2412




Ignore
« Reply #1 on: December 06, 2013, 10:05:33 AM »

Quote
...Splice Machine; Hadoop; HBase; Derby SQL; JDBC; 4 node AWS Cloudera Hadoop cluster; Pentaho PDI; Tableau; LOTW; Clublog...

What can a fellow say but, "Whoop-di-doo!".   LOL. thanks.  bill
Logged
W0FM
Member

Posts: 2056




Ignore
« Reply #2 on: December 17, 2013, 01:00:12 PM »

Quote
Splice Machine into a 4 node AWS Cloudera Hadoop cluster, and used Pentaho PDI to load the data in milliseconds. I also used Tableau to do advanced data visualization - and its really cool - and really fun.

Huh?  I think you missed the cymbal crash right there between the bridge and the second chorus, Rich!    Grin

Terry, WØFM
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!