Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: Prev 1 2 3 4 [5] 6 Next   Go Down
  Print  
Author Topic: Interesting LoTW Data  (Read 13291 times)
KY6R
Member

Posts: 3291


WWW

Ignore
« Reply #60 on: December 28, 2012, 11:01:06 AM »

Oh yes, for database systems the query time (time to find a QSO match) must increase faster than the number of records to be handled. Even 10 years back a pioneer in the Internet world had to split its customer records database into 30+ pieces to maintain  reasonable processing. That was with state of the art hardware and a very large budget.

I did exactly that at Lithium Technologies on a MySQL database that supported > 400 customers world wide. Its called "sharding" or (depending how you do it) "horizontal partitioning". The common term for this is scaling out. I was hired as a contractor to save the system - which has all of companies like Cisco's on line based technical support forums on it. Comcast, HP, Nokia, Home Depot and others were also on this system.

After I did this - I told the company that even sharding in a SQL RDBMS is only a stop gap measure - that would buy them time. They listened and so we evaluated "Big Data" technologies - like Hadoop / HBase / Hive, plus several reporting technologies. The paradigm was _exactly_ like ham radio log files - its an event log which cries for an event logging framework.

The problem that I solved is scalability where data is growing faster than they could ingest and process it in a SQL RDBMS. In the old school days - you just bought bigger hardware (thats called scaling up), but that becomes cost prohibitive very fast. So the hardware upgrade is stop gap fix #1, and it can take a year to do the rest of what's needed - but I am confident that the ARRL will catch up and stay one step ahead of the devil. This episode has shown how popular (and successful) LOTW is, and what they will need to do to support things like "everybody's contests" . . . (Frankly - I was VERY surprised when they decided to support the CQ WW contest logs . . . . . that seemed like they were going to really expand their data) . . . .

Logged
PE5T
Member

Posts: 40




Ignore
« Reply #61 on: December 29, 2012, 05:01:15 AM »

Lotw seems to be stopped working, according to the latest status reports.

Kees
Logged
AF3Y
Member

Posts: 3877




Ignore
« Reply #62 on: December 29, 2012, 06:02:59 AM »

Lotw seems to be stopped working, according to the latest status reports.

Kees

Naw...... It's running dragging along slower then usual. It is now around 12 days
behind. Pitiful!  Shocked

73, Gene AF3Y
Logged
N3QE
Member

Posts: 2421




Ignore
« Reply #63 on: December 29, 2012, 06:27:01 AM »

That is the nature of the software beast; the better it is the quicker it becomes overloaded.
I like Yogi Berra's quote: "Nobody goes there anymore, it's too crowded."
Quote
Oh yes, for database systems the query time (time to find a QSO match) must increase faster than the number of records to be handled.
No, it doesn't. Effectively constructed indices can do a wonderful job making the search time grow much slower than linearly. And if carefully done... a bulk load of 10000 records need not take much longer than a bulk load of 10 records (merge techniques available since the 1960's... I doubt they teach them to kids these days!).

But... many of the older database engines reach some limit where the scaling breaks down (e.g. when the index is larger than RAM), and some gracefully slow down past that point, and others catastrophically slow down at that point.
Quote
Even 10 years back a pioneer in the Internet world had to split its customer records database into 30+ pieces to maintain  reasonable processing. That was with state of the art hardware and a very large budget.
Sharding is pretty ugly and it's a shame when it has to be done as anything other than a stopgap. Unfortunately some folks enshrine sharding as their job, instead of as a workaround to get the job done given a poor baseline. To quote Ted DziubaOh, scalability: the problem that tens of thousands of engineers yearn for, but only six actually have.
Logged
KY6R
Member

Posts: 3291


WWW

Ignore
« Reply #64 on: December 29, 2012, 07:01:51 AM »

Sharding is pretty ugly and it's a shame when it has to be done as anything other than a stopgap. Unfortunately some folks enshrine sharding as their job, instead of as a workaround to get the job done given a poor baseline.

Absolutely. I also multi-threaded a single threaded ETL process which bought way more time than sharding. In fact the nightly load went from 27 hours down to 14 hours for batch loading the previous days data. Their choices came down to either scale up / replace using something like Teradata or Exadata, or see if one of the (way over hyped but potentially useful) "NoSQL" technologies like HBase was a better choice.

HBase was the better choice - now that Cloudera has added SQL - with "Impala". But I have seen way too many data architects want to add in new technologies or do something that just adds to their resume. Call it "shiny penny syndrome".
Logged
AB8MA
Member

Posts: 765




Ignore
« Reply #65 on: December 30, 2012, 07:21:53 AM »

I see that in the last 8 hours, fewer than 100 (93) User files have been processed, but 34824 QSO records have been entered into the system.
Logged
W9KEY
Member

Posts: 1165




Ignore
« Reply #66 on: December 30, 2012, 09:50:33 AM »

I see that in the last 8 hours, fewer than 100 (93) User files have been processed, but 34824 QSO records have been entered into the system.

when I sent QRL? today to LoTW here was the response:

LoTW: QRU -- pse QRX 6 to 8 weeks -- pile-up too big, must QRT

Wink
Logged
AJ4RW
Member

Posts: 568




Ignore
« Reply #67 on: December 30, 2012, 12:38:54 PM »

This might have been mentioned before but I couldn't find it. The only problem I'm concerned with this LOTW debacle is that it will deter other stations/countries/rare DXCC entities from using the LOTW service.  LOTW is a very efficient means of QSLing when it's operating correctly which is not the case at present.
Randy
Logged
AF3Y
Member

Posts: 3877




Ignore
« Reply #68 on: December 30, 2012, 04:18:09 PM »


 The only problem I'm concerned with this LOTW debacle is that it will deter other stations/countries/rare DXCC entities from using the LOTW service. 

Randy

Might deter them? Hell, I dont see a lot of LOTW use from around the globe right now. Seems like a mostly NA thing... I wonder what percantage of users is NA.... Huh

73, Gene AF3Y
Logged
AA6YQ
Member

Posts: 1851


WWW

Ignore
« Reply #69 on: December 30, 2012, 04:57:01 PM »


 The only problem I'm concerned with this LOTW debacle is that it will deter other stations/countries/rare DXCC entities from using the LOTW service. 

Randy

Might deter them? Hell, I dont see a lot of LOTW use from around the globe right now. Seems like a mostly NA thing... I wonder what percantage of users is NA.... Huh


Since 2012-01-01, 29369 different callsigns have uploaded QSOs to LotW. Of those, 14,189 were from the continental US, Alaska, or Canada -  48.3%.

The majority of stations who uploaded to LotW so far in 2012 were from outside of North America.

    73,

          Dave, AA6YQ

Logged
N3QE
Member

Posts: 2421




Ignore
« Reply #70 on: December 30, 2012, 05:20:03 PM »

Might deter them? Hell, I dont see a lot of LOTW use from around the globe right now. Seems like a mostly NA thing... I wonder what percantage of users is NA.... Huh

Funny, while others here gripe about LOTW not being used around the world, I just got Bahrain and Israel confirmed on 80M CW by LOTW. Bahrain was ATNO, Israel was a bandfill. 80M has been very, very good to mideast past few weeks.

On LOTW popularity, I think Yogi Berra said best, "Nobody goes there anymore. It's too crowded."

Tim.
Logged
NU1O
Member

Posts: 2762




Ignore
« Reply #71 on: December 30, 2012, 06:15:27 PM »

I have no complaints with LoTW with regards to DX stations.  Try getting West Virginia confirmed. That is harder than any DX station.

73,

Chris/NU1O

Logged
W9KEY
Member

Posts: 1165




Ignore
« Reply #72 on: December 30, 2012, 09:32:40 PM »


Since 2012-01-01, 29369 different callsigns have uploaded QSOs to LotW. Of those, 14,189 were from the continental US, Alaska, or Canada -  48.3%.

The majority of stations who uploaded to LotW so far in 2012 were from outside of North America.

    73,

          Dave, AA6YQ


That's good to hear, thanks for the numbers.  Does this mean North American hams are less virtually inclined than hams from other continents, or is it reflective of the dispersion of hams by continent?
Logged
N3QE
Member

Posts: 2421




Ignore
« Reply #73 on: December 31, 2012, 06:03:37 AM »

Since 2012-01-01, 29369 different callsigns have uploaded QSOs to LotW. Of those, 14,189 were from the continental US, Alaska, or Canada -  48.3%.
The majority of stations who uploaded to LotW so far in 2012 were from outside of North America.
That's good to hear, thanks for the numbers.  Does this mean North American hams are less virtually inclined than hams from other continents, or is it reflective of the dispersion of hams by continent?

It's entirely consistent with the distribution of NA vs international logs submitted to major DX contests. Roughly 50/50. Check out ARRL or CQ "logs received" pages.

It's also entirely consistent with the distribution of NA vs international logs in the major DXpeditions. Roughly 50/50. Check out e.g. HK0NA's clublog statistics: http://www.clublog.org/charts/?c=hk0na#r

I would be very surprised if the distribution of LOTW users was radically different than the distribution of hams that participate in DX contests or DXpeditions.

I would be very surprised if LOTW users represented a good cross section of those who only participate in 2-meter nets, or who only do Emcomm stuff. There would be some overlap of course (the activities are not mutually exclusive!) but hardly representative.

Tim N3QE
Logged
AB8MA
Member

Posts: 765




Ignore
« Reply #74 on: December 31, 2012, 08:10:10 AM »

Good news:

Quote
Dec 31, 2012: LoTW Update --
The new LoTW hardware has been delivered and is in the process of being configured and tested. We hope to have it up and running within a few weeks. In the meantime, please do not upload your log more than once. This only causes more processing delays.

The LoTW Team
Logged
Pages: Prev 1 2 3 4 [5] 6 Next   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!