eHam

eHam Forums => DXing => Topic started by: AF3Y on November 05, 2012, 03:51:51 PM



Title: LOTW
Post by: AF3Y on November 05, 2012, 03:51:51 PM
Down AGAIN?  >:(

I guess every Monday after a contest, LOTW is not working?

Sounds like a pisspoor way to run an operation.

73, Gene AF3Y


Title: RE: LOTW
Post by: NU1O on November 05, 2012, 04:07:25 PM
Down AGAIN?  >:(

I guess every Monday after a contest, LOTW is not working?

Sounds like a pisspoor way to run an operation.

73, Gene AF3Y

I have a charge on this month's AMEX statement for about $150 for LoTW credits, some certificates and endorsements.  I'm sure many of you have spent similar amounts. We should at the very least have a system which is quick, reliable, and user-friendly.  The current LoTW system is anything but user-friendly.  Many overseas stations tell me it is much too complicated.  Even Stateside stations say the same thing. We have about 53,000 users worldwide. I think it would be 5 times that if it was more user-friendly.

73,

Chris/NU1O


Title: RE: LOTW
Post by: K2QB on November 06, 2012, 06:43:12 AM
Complicated??? How complicated is it...I click one button to upload my QSO's to LoTW and one button to download them.. Then I sit back and look at the confirmations. People who complain it is too complicated in my opinion are really saying I am too lazy to learn anything new. And yes, after a contest maybe it does take a few days for the system to catch up. Let's see thousands of logs being uploaded with thousands upon thousands of QSO's. And if it takes two or three days for your confirmations to show up...so what? They eventually do. As far as the $150 charge.....what would it have cost to mail all those out? Envelopes, stamps, cards, time, cost of the award....sorry I've been chasing DX for over 40 years so I'm not a newbie by any means and LoTW is one of the best things ever to come along.


Title: RE: LOTW
Post by: NK7Z on November 06, 2012, 07:03:12 AM
I've been chasing DX for over 40 years so I'm not a newbie by any means and LoTW is one of the best things ever to come along.
I wholeheartedly agree...  LOTW has saved me thousands in QSL card fees over the years...  Yes it takes a few days, but wow...  I remember the old days when it might take a YEAR to get a single card.  That and the price is right, free...  Use eQSL to do the awards that they support, and you can save a lot of cash...  My WPX cost me something less than 20 bucks, (at most, using eQSL).  If you are uncomfortable with the fees, then don't chase the award...


Title: RE: LOTW
Post by: K3STX on November 06, 2012, 07:28:40 AM
Complicated??? How complicated is it...I click one button to upload my QSO's to LoTW and one button to download them..

I love LOTW as much as anyone, but for MOST of us who do NOT use computer logging programs it is not a simple push of a button. From a contest logger like N1MM I have to generate an ADIF file, save it on my desktop, then open TQSL, open the ADIF file, LOG IN with my password, and then a .tq8 file is created and saved on my desktop. Then go to the ARRL website, log into LOTW, and upload my .tq8 file.

Certainly not impossible, but certainly not one click. Maybe for people who keep computer logs with some program that automatically links to LOTW, but I don't know about computer logging programs.

Still, I have a cheat sheet telling me what to do, it takes only 5 minutes a few times a year. The money I have saved in stamps for 5BDXCC, WOW!!

paul


Title: RE: LOTW
Post by: NI0C on November 06, 2012, 07:48:14 AM
K3STX wrote:
Quote
From a contest logger like N1MM I have to generate an ADIF file, save it on my desktop, then open TQSL, open the ADIF file, LOG IN with my password, and then a .tq8 file is created and saved on my desktop. Then go to the ARRL website, log into LOTW, and upload my .tq8 file.

Certainly not impossible, but certainly not one click. Maybe for people who keep computer logs with some program that automatically links to LOTW, but I don't know about computer logging programs.

It's incredibly easy to setup the free DXLab DXKeeper program to do all this for you, so you can upload a file or an individual QSO to LoTW with a couple of mouse clicks.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: NK7Z on November 06, 2012, 08:03:26 AM
Complicated??? How complicated is it...I click one button to upload my QSO's to LoTW and one button to download them..

I love LOTW as much as anyone, but for MOST of us who do NOT use computer logging programs it is not a simple push of a button. From a contest logger like N1MM

Hi Paul,
But you are using a computer logging program...  N1MM is a "Contest Logger" as per their site...  The fault lies with N1MM, not LOTW.  LOTW has provided an API, (Applications Programmers Interface), to the community; this is a standardized way of letting programs, (or Apps), talk to the LOTW servers.  Of course LOTW needs to be running for it to work, which it is not as of this moment.  N1MM must not be using the APIs...  N1MM is supported enough that I would expect someone to add it soon...  There is a feature request link on their page.  You should request a direct upload to LOTW.  The N1MM team is a great set of guys, and they are very responsive to user needs.  

I use N3FJP's logging software, and MixW in tandem.  During a contest, I use MixW, and after the contest I import the log into N3FJP's logging software.  If interested, I have an extensive review of N3FJP's Amateur Contact Log, at: http://nk7z.net/review-of-amateur-contact-log-3-0/ with screenshots etc.  Also, a review of MixW at: http://nk7z.net/review-of-mixw-multimode-software/  As well as many other reviews etc. on site...


Title: RE: LOTW
Post by: K3STX on November 06, 2012, 08:31:28 AM
The "problem" is not N1MM, the problem is me. I do not WANT to use a computer logging program for everyday use. I only use the computer for the 4-6 contests/year I operate in. I like my paper logs.

Having said all this, I still think LOTW is great and the slight hassle is offset 1000-fold by the convenience of confirmations with no cards. Now for 5BWAZ credits to count , THAT will be great.

Paul


Title: RE: LOTW
Post by: NK7Z on November 06, 2012, 09:26:57 AM
The "problem" is not N1MM, the problem is me. I do not WANT to use a computer logging program for everyday use. I only use the computer for the 4-6 contests/year I operate in. I like my paper logs.


Hi again Paul,

It took me a year or more to embrace an electronic log...  Once done though I am a much happier person for it...  Hopefully N1MM will add direct upload to LOTW soon.  That program has an amazing amount of support. 

Gud DX!


Title: RE: LOTW
Post by: NU1O on November 06, 2012, 10:01:59 AM
Complicated??? How complicated is it...I click one button to upload my QSO's to LoTW and one button to download them.. Then I sit back and look at the confirmations. People who complain it is too complicated in my opinion are really saying I am too lazy to learn anything new. And yes, after a contest maybe it does take a few days for the system to catch up. Let's see thousands of logs being uploaded with thousands upon thousands of QSO's. And if it takes two or three days for your confirmations to show up...so what? They eventually do. As far as the $150 charge.....what would it have cost to mail all those out? Envelopes, stamps, cards, time, cost of the award....sorry I've been chasing DX for over 40 years so I'm not a newbie by any means and LoTW is one of the best things ever to come along.

I have a problem with N3FJP's AC Log interfacing with LoTW which nobody in the support forum has been able to solve so I must use the work-around of saving my log, using TQSL to create a T-8 file and then going to the ARRL site to upload my file. One day I will call the ARRL for help since the guys who write these logging problem have enough on their table fixing their own bugs w/o worrying about LoTW problems.  It was much easier when it was all streamlined into the program, however.  You are correct, though, it was hitting just one button.

As for the cost savings, you are inflating them by assuming I'd have sent out a QSL for every LoTW confirmation.  I have 4,500 LoTW confirmations but I never would've sent QSL cards out for each and every LoTW confirmation and neither would most of you.  For example, how many times do I need Germany or Italy confirmed on 10 meter phone?  I probably have a couple hundred LoTW confirmations for each of those countries while one QSL card would suffice per band/mode.  If I were to estimate I'd guess I would have sent out about 200 to 300 cards but certainly not the 4,500 I now have via LoTW. There is a savings but it is not $2 for each LoTW confirmation.

Please don't misunderstand, I really like LoTW and I think it is a wonderful innovation.  I'm just passing on what many foreign and US hams have told me and describing my own problem with LoTW.  

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on November 06, 2012, 10:06:54 AM

I have a problem with N3FJP's AC Log interfacing with LoTW which nobody in the support forum has been able to solve so I must use the work-around of saving my log, using TQSL to create a T-8 file and then going to the ARRL site to upload my file.

Chris,

Can you expand on the issue you are having...  I had one as well, just like that.  Write me off list, as this is now a support question...  There may be a fix.


Title: RE: LOTW
Post by: W5DQ on November 06, 2012, 10:14:59 AM
The "problem" is not N1MM, the problem is me. I do not WANT to use a computer logging program for everyday use. I only use the computer for the 4-6 contests/year I operate in. I like my paper logs.

Having said all this, I still think LOTW is great and the slight hassle is offset 1000-fold by the convenience of confirmations with no cards. Now for 5BWAZ credits to count , THAT will be great.

Paul

Good morning Paul,

Not to belittle your choice of paper logs but if (or when) you get a stack of QSLcards, say 100+ in a single pkg, from the buro to update confirmations for in your paper logs and return QSL cards to, how long does that take you to do. If it was one or two then I would say it is a moot point. But if you were to receive buro cards like I do (and many of the hams I handle cards for a W6 Incoming DX QSL buro volunteer) at the rate of 50-200+ per quarter, I would think you would find that employing an electronic logbook would greatly simplify your logging and return QSL efforts. I can update my log and print a return QSL card label for each QSL in just a matter of a couple of seconds each and once I am done, the stack of QSL cards go back into the box to be shipped out in the next shipment (usually 3 to 4 times a year).

I too keep a binder with printed pages of my electronic logbook for that 'if all else fails' moment (along with seperate electronic backups of the log data files stored apart in case of fire, etc.) but I find having the ability to 'near instantly' being able to find a QSO or group of QSOs indespensible in my ham activities.

I think you might find that you would like electronic logging if you were to give it a whirl if you haven't before. Run an electronic logbook along side of your paper one for a while and see if you don't start leaning more towards the electronic one. Just some personal observations.

Either way, have fun and good luck in your DXing and QSLing.

Gene W5DQ



Title: RE: LOTW
Post by: M0TTB on November 06, 2012, 10:24:34 AM
Hopefully the LoTW issue isn't as bad as some are saying.
http://dx-world.net/2012/lotw-lost-data/


Title: RE: LOTW
Post by: K3STX on November 06, 2012, 10:48:58 AM
But if you were to receive buro cards like I do (and many of the hams I handle cards for a W6 Incoming DX QSL buro volunteer) at the rate of 50-200+ per quarter, I would think you would find that employing an electronic logbook would greatly simplify your logging and return QSL efforts.

Yep, and I go back through maybe 4 logbooks each time I get the package. It does indeed take time, but I think that is part of the fun, rummaging through my ARRL logbooks to check off the QSL RCVD box and hand-write a card to check off the QSL SENT box! I guess I am too old fashioned. I don't like the idea of booting-up a computer simply to listen on my radio and maybe call a station or two.

This is my deficiency, not LOTW!!! For new 5BDXCC fills I simply enter them in N1MM "DX" contest file, edit the dates, and send them in. My T6MO QSO was confirmed in less than 24 hr on LOTW that way.

p


Title: RE: LOTW
Post by: N3QE on November 06, 2012, 11:09:35 AM
I understand your desire to be "a guy with a radio" and not "a guy with a radio and a computer for logging".

I operated a few contests during power outages. Radio and a few vital accessories (including a small 12V LED lamp) run off solar charged batteries. Computer is not a vital accessory.

After each such incident... the big hassle was not LOTW. The big hassle was typing the QSO's into the computer after power came back. Doing this with N1MM or ARRL's TrustedQSL program is possible but very very awkward. So many mouse clicks... so many places where the right number has to be typed into the correct field (but not into the wrong field!). I found that some of the WA7BNM web-to-cabrillo forms were far superior to the drudgery of N1MM or TrustedQSL for making a cabrillo file, that I then imported and re-exported as ADI, which I then ran through TrustedQSL and

For me the computer conversions are fairly painless. The boring drudgery is the point-and-drool user interfaces of many of the loggers for converting a paper log. A free text window for typing works great for me.



Title: RE: LOTW
Post by: NU1O on November 06, 2012, 11:58:36 AM

I have a problem with N3FJP's AC Log interfacing with LoTW which nobody in the support forum has been able to solve so I must use the work-around of saving my log, using TQSL to create a T-8 file and then going to the ARRL site to upload my file.

Chris,

Can you expand on the issue you are having...  I had one as well, just like that.  Write me off list, as this is now a support question...  There may be a fix.

Sure, when I try to upload my new contacts using AC Log I get an error which says, "Invalid Procedure Call or Argument". I asked what this meant in the AC Log user's forum and I did not get one reply.  I can still download my confirmations but I can not upload using AC Log so I must create a .TQ8 file and go to the ARRL's site to upload my data.  This computer is about a year old and everything worked fine until one day when I started getting that error message.  I did not change any paths or settings.  I am no computer whiz and do not have a clue what that message means.  I was very surprised when nobody in the forum answered.  Usually, Scott Davis, who wrote the program, is very good at answering questions but even he did not chime in although he makes no attempt to pass himself off as an expert on LoTW.  If this was an AC Log problem I think Scott or some other user would've answered my call for help.

If anybody knows what's causing my problem or if it is a LoTW problem I'd really appreciate the help. 

73,

Chris/NU1O


Title: RE: LOTW
Post by: KH6DC on November 06, 2012, 01:00:08 PM
Sounds like choke to me after they merged CQ WW and WPX awards to LOTW.  I love and use LOTW over eQSL for the awards chasing.  I confirm to LOTW daily and eQSL maybe once per quarter if I'm lucky or if someone emails me to request eQSl confirmation.

I don't use a logger program so input everything using TQSL then upload to LOTW-nothing difficult about that.

73, Delwyn KH6DC


Title: RE: LOTW
Post by: W4VKU on November 06, 2012, 01:11:44 PM
There was talk about ADIF file to Lotw upload and the convoluted process involved.
Yes, it can be a bit complicated for some. Would have been nice if other logging programs had a plugin that did
it direct.

In addition to the Adif->lotw upload, i get the ADIF loaded into DxKeeper at the same time. Dxkeeper will
Sync up with Lotw and will mark QSL's that you have received. It is cool in the sense that it plugs in
to DxView and i can use Dxview to get beam headings, gray line and also see the band fills for
the given callsign. Pretty nifty tool if you need to chase Dxpeds and not make Dupe contacts.

Also make sure you keep the digital certificate safe for Lotw, or else if the computer craps out, you are out of luck.

LOTW is definitely down, since the website says so when one tries to log in.

Krish
w4vku


Title: RE: LOTW
Post by: AF3Y on November 06, 2012, 01:29:44 PM
My original post was not about the complicity or lack of, while using LOTW.   I use a computer logger and have NO problems uploading files, UNLESS the damned system is down from a contest, or whatever, which seems to happen more and more lately.

I am NOT a computer guru, but it seems to me that if the LOTW system needs upgrading, tweaking, or enlarging, or whatever to make it handle the loads, then ARRL should do that. I suppose its a $$ thing? Well, Hell, they are charging us, right ???? Not enough money? Not enough people?  I dont know, I just know if you need more resources of whatever kind to make your system work, that is what you need to do.

This is not a first time problem.  Happens OFTEN....... Not very impressive, IMHO.  As someone said, if the amount of hams on LOTW now are shutting it down, heaven forbid an increase in users.  I Like LOTW, I just dont like the down time, which should be fixed by ARRL, sooner rather than later if they want more users..

73, Gene AF3Y



Title: RE: LOTW
Post by: AJ4RW on November 06, 2012, 01:38:34 PM
It's my fault that LOTW is down.  I uploaded about 500 Q's from the contest plus my contacts for last 3 weeks.  I also use the HRD LOTW upload feature which might contribute to the LOTW problem but the uploading process is as smooth as silk thanks to HRD.
Randy


Title: RE: LOTW
Post by: AF3Y on November 06, 2012, 02:03:25 PM
It's my fault that LOTW is down.  I uploaded about 500 Q's from the contest plus my contacts for last 3 weeks.  I also use the HRD LOTW upload feature which might contribute to the LOTW problem but the uploading process is as smooth as silk thanks to HRD.
Randy

YEP.... I thought it might have been you. STOP THAT!!! hi

73, Gene AF3Y


Title: RE: LOTW
Post by: K7KB on November 06, 2012, 02:57:53 PM
"DX World.net has learned that up to “45% of [recently] uploaded logs and delivered awards / DXCC certificates have been lost due to corrupted database / backups”.

It’s currently unknown if the “missing” data can be retrieved. An official update from LoTW developers would, of course, be beneficial – least not to put minds at rest.

As can be seen from the latest message on the ARRL LoTW homepage, the Logbook team is currently investigating the issue."

Wonderful, just wonderful :P If this turns out to be true then it seems to me the LOTW team needs to completely revamp their DB Server/backup system so this doesn't happen in the future. There have been too many problems with system slowdowns, lost LOTW uploads, etc. If they expect the user community to use LOTW then it needs to be reliable, period.

John K7KB


Title: RE: LOTW
Post by: AA6YQ on November 06, 2012, 04:40:18 PM

I have a problem with N3FJP's AC Log interfacing with LoTW which nobody in the support forum has been able to solve so I must use the work-around of saving my log, using TQSL to create a T-8 file and then going to the ARRL site to upload my file.

Chris,

Can you expand on the issue you are having...  I had one as well, just like that.  Write me off list, as this is now a support question...  There may be a fix.

Sure, when I try to upload my new contacts using AC Log I get an error which says, "Invalid Procedure Call or Argument".

"Invalid Procedure Call or Argument" is without question a defect in AC Log.

     73,

          Dave, AA6YQ


Title: RE: LOTW
Post by: NU1O on November 06, 2012, 05:25:26 PM

"Invalid Procedure Call or Argument" is without question a defect in AC Log.

     73,

          Dave, AA6YQ

Thanks, Dave.  I'll send Scott a private email and ask him what's going on.  There is a lot of volume in the help forum and Scott does not always catch them all.  If he can't help me I can always switch to your program.

What I find odd is when I asked the question in the forum - about 5 months ago - not one person said they were having the same problem.

73,

Chris/NU1O


Title: RE: LOTW
Post by: K3STX on November 06, 2012, 06:41:11 PM
I Like LOTW, I just dont like the down time, which should be fixed by ARRL, sooner rather than later if they want more users..

Even if it takes two weeks to confirm your contacts instead of 12 hours, who cares? The system is "crashing" due to too many logs being submitted, that is a GOOD thing! And I bet the reason they don't "upgrade" is that upgrading will cost MONEY, which would mean ARRL would have to raise money by increasing member dues.

Hams are cheap; we can either tolerate a slow "server" (or whatever it is) OR we pony-up and chip-in to get new ones. I think we all know what the choice will be.

paul


Title: RE: LOTW
Post by: AB8MA on November 06, 2012, 07:16:36 PM
I think that when this incident is resolved, somebody on the Logbook team is going to be a whole lot more knowledgeable.

I don't envy them.


Title: RE: LOTW
Post by: K3STX on November 06, 2012, 07:22:25 PM
Didn't exactly the same thing happen last year after either CQWW or ARRL DX with day long delays and lost logs? I know I had to re-send MY .tq8 file that was lost back then.

I doubt any head will roll and I doubt they will learn anything.

paul


Title: RE: LOTW
Post by: K6UJ on November 06, 2012, 08:03:03 PM
John, W7KH is at the top of the list for ARRL Mixed DXCC with 398 confirmed.

What makes this especially noteworthy is that he doesn't have a computer in his shack.  Still uses
paper logs only.  He doesn't use the DX spotting clusters, (doesn't use LOTW   :D)

73,
Bob
K6UJ



Title: RE: LOTW
Post by: KH6DC on November 06, 2012, 11:00:08 PM
My original post was not about the complicity or lack of, while using LOTW.   I use a computer logger and have NO problems uploading files, UNLESS the damned system is down from a contest, or whatever, which seems to happen more and more lately.

I am NOT a computer guru, but it seems to me that if the LOTW system needs upgrading, tweaking, or enlarging, or whatever to make it handle the loads, then ARRL should do that. I suppose its a $$ thing? Well, Hell, they are charging us, right ???? Not enough money? Not enough people?  I dont know, I just know if you need more resources of whatever kind to make your system work, that is what you need to do.

This is not a first time problem.  Happens OFTEN....... Not very impressive, IMHO.  As someone said, if the amount of hams on LOTW now are shutting it down, heaven forbid an increase in users.  I Like LOTW, I just dont like the down time, which should be fixed by ARRL, sooner rather than later if they want more users..

73, Gene AF3Y



I agree Gene, LOTW seems a bit archaeic compared to some of the databases out there.

73, Delwyn, KH6DC


Title: RE: LOTW
Post by: SV1XV on November 07, 2012, 02:36:15 AM
I does not seem archaic, it seems dead.


Title: RE: LOTW
Post by: NN3W on November 07, 2012, 03:34:48 AM
This is a serious problem that ARRL and its BOD need to address.  LoTW is not only a log repository, it is also a revenue generator for the League.  Part of the ability of the League to generate revenue is grounded in faith and trust that the system works, works well, and is bulletproof.

This latest little "glitch" really starts to cast doubt as to if the ARRL really knows what its doing vis-a-vis LoTW and has the gravitas to manage such a system.  

I'll note that given what happened in late October/early November 2011, the League should have been prepared for any contingency that would come up.  

It wasn't.


Title: RE: LOTW
Post by: SV1XV on November 07, 2012, 03:49:33 AM
This latest little "glitch"...  
We don't even know if it is a little glitch or a major catastroph and we can hope for the best outcome while fearing for the worst. I would expect some news on the ARRL web site (http://www.arrl.org), but there is nothing there. I hope they shall post an update in the morning (EST).



Title: RE: LOTW
Post by: W4VKU on November 07, 2012, 05:33:54 AM
If this corruption has resulted in data loss, it is going to result in loss of faith in this system. How do we chase down all
those QSL's, now that they don't show up?

May be the business model will need to be changed. Perhaps something to the order of, download the QSL credits and
pay when you cashin for a certificate. That way, one can download it offline and then still deal with such
data loss by showing the proof of those QSL's( from our own backups). The file could be signed digitally prior to
downloading, to avoid any sort of tamper.
 
krish
w4vku


Title: RE: LOTW
Post by: NK7Z on November 07, 2012, 05:39:17 AM

"Invalid Procedure Call or Argument" is without question a defect in AC Log.

     73,

          Dave, AA6YQ

Hi Dave (AA6YQ),
That error was from the LOTW site, not ACLog.  It showed up when using a browser only...  We are were all getting that error from the LOTW site prior to the current error message displayed at the LOTW site...  I am not sure I see how that is ACLog, as opposed to LOTW.  Could you expand?

EDIT:
Note, the error message LOTW had was:
Fatal error: initializeLOTWDB: -709 – CONNECT: (cannot connect to server socket): General database error [initializeLOTWDB: -709 – CONNECT: (cannot connect to server socket)

So you may be right it may be an error in ACLog or how ACLog reacts to a failure to log in...


Title: RE: LOTW
Post by: AB8MA on November 07, 2012, 07:29:08 AM
Hopefully the LoTW issue isn't as bad as some are saying.
http://dx-world.net/2012/lotw-lost-data/

If this information is correct, no data was lost.

http://www.arrl.org/news/logbook-of-the-world-experiencing-maintenance-issues

11/07/2012

Quote
ARRL Information Technology Manager Michael Keane, K1MK, is reporting that the Logbook of The World (LoTW) system is currently down for maintenance. “With the system temporarily down, radio amateurs are not able to access LoTW or upload their logs,” Keane explained. “No data has been lost, and everyone’s records are intact.


Title: RE: LOTW
Post by: N7SMI on November 07, 2012, 07:33:34 AM
I've designed and maintained critical, high-traffic online databases (10s of thousands of transactions per minute and billions of records). LoTW is stupid simple in comparison, size, and processing. In addition to the continual delays and down-time, the fact that they've now been down for 3 days strongly confirms that they have no idea what they're doing. They either need to pull the plug or get a DB person that is competent and invest in some basic, reliable infrastructure.

Even if data is lost, it will be easy to re-upload logs - though I highly doubt they'll be able to handle everyone uploading the last several months of contacts once they get back online.

What a mess!


Title: RE: LOTW
Post by: AF3Y on November 07, 2012, 10:58:00 AM
I've designed and maintained critical, high-traffic online databases (10s of thousands of transactions per minute and billions of records). LoTW is stupid simple in comparison, size, and processing. In addition to the continual delays and down-time, the fact that they've now been down for 3 days strongly confirms that they have no idea what they're doing. They either need to pull the plug or get a DB person that is competent and invest in some basic, reliable infrastructure.


Perhaps a couple more Commodore 64s and a few more tape decks.  ::)

Their system really does stink.

73, Gene AF3Y


Title: RE: LOTW
Post by: W5DQ on November 07, 2012, 11:16:14 AM
I've designed and maintained critical, high-traffic online databases (10s of thousands of transactions per minute and billions of records). LoTW is stupid simple in comparison, size, and processing. In addition to the continual delays and down-time, the fact that they've now been down for 3 days strongly confirms that they have no idea what they're doing. They either need to pull the plug or get a DB person that is competent and invest in some basic, reliable infrastructure.

Even if data is lost, it will be easy to re-upload logs - though I highly doubt they'll be able to handle everyone uploading the last several months of contacts once they get back online.

What a mess!

Send them your resume. Perhaps they can make you an offer you can't refuse.  ;)

I agree. Something has to be done about the current status of development and maintenance. I thought we had problems in getting real time avionics to work right but this borders on ridiculous!!


Title: RE: LOTW
Post by: K3VAT on November 08, 2012, 02:53:22 AM
As of almost 1100Z on Thursday, LoTW still down.
ARRL says "no data loss".  Just need some patience while they do their thing.
73, Rich, K3VAT


Title: RE: LOTW
Post by: W2IRT on November 08, 2012, 04:48:41 AM
In the meantime, if anybody needs to submit a paper DXCC application I'll be glad to check your cards (no limit) and do it the old-fashioned way. :D


Title: RE: LOTW
Post by: NK7Z on November 08, 2012, 05:19:14 AM
As of almost 1100Z on Thursday, LoTW still down.
ARRL says "no data loss".  Just need some patience while they do their thing.
73, Rich, K3VAT

Hi Rich,
I agree...  Here is how I suspect this will shake out:

1.  ARRL will fix the problem.
2.  ARRL will address the reasons it happened.
3.  Things will be better than prior to this issue, after it is back up.

I suspect this is a big project for the ARRL, as their first steps into the digital age, things will not be perfect, thus this sort of thing will happen.  I am sure the IT staff is feeling the heat right now, if not from the ARRL itself, then from themselves...  Having worked in IT for a number of decades, most IT people are really dedicated, and want things to work.  So far the only real issue I see here is the lack of clarity for the current status of LOTW, when the systems breaks. 

How often a system breaks is almost a direct correlation to how much money one spends on them.  The ARRL is not Target, or Walmart, so they are not going to have that level of funding to spend to make LoTW five nines up time, so expecting it to behave like it has five nines is not reasonable.  I would be ecstatic with three nines, that's eight hours and forty six minutes of downtime a year...  I would expect that somewhere close to 10 to 20 hours a year of downtime is acceptable, at least to me.   

LoTW will come back up, and it will probably be better, or if not now, then this incident will be used to vector more funds/work towards making it better.  We all see that there are clear problems with the current setup, and I am sure the ARRL also see this.  Being a responsible organization, I expect they will fix it as soon as they can and then use this opportunity to make it better.  I am willing to wait until is is fixed, and I am still glad it is here, and I will continue to use it after it is fixed. 

The one thing I would recommend to the ARRL is to get a light on this and put some transparency to it, start posting daily updates so the users don't wonder what is happening.  Start posting them in the forums both here and at QRZ, and on their web site... 

For those folks that say this should never have happened, it did, and it will again, and hopefully it will get better each time it happens...  This is a big project in a bright spotlight...

73's
Dave


Title: RE: LOTW
Post by: NU1O on November 08, 2012, 06:15:58 AM
This is a serious problem that ARRL and its BOD need to address.  LoTW is not only a log repository, it is also a revenue generator for the League.  Part of the ability of the League to generate revenue is grounded in faith and trust that the system works, works well, and is bulletproof.

This latest little "glitch" really starts to cast doubt as to if the ARRL really knows what its doing vis-a-vis LoTW and has the gravitas to manage such a system.  

I'll note that given what happened in late October/early November 2011, the League should have been prepared for any contingency that would come up.  

It wasn't.

I once spent about an hour reading everything on the LoTW site and the ARRL claims they do not charge the full cost for LoTW expenses.  It's not a profit center according to them. That maybe part of the problem.

The ARRL prides itself on technology and all the advances which have occurred in radio over the past century.  "From Spark to Space" was the motto on the QSL cards they gave me when I once operated W1AW.  I believe that motto may be or was once the motto for the League. The problems with LoTW are a great embarrassment to the ARRL

We have database experts with very impressive qualification in this forum who have claimed the LoTW database is not overly complex and it should be a fairly easy to create a system which is much more robust.  I can't personally make that claim but the system needs to be fixed so it doesn't go down after every contest.  If the present help isn't capable of fixing the problem they need to be replaced.  Also, if they are selling their product for less than cost they should reprice so there are enough resources to hire high quality help and maintain the system properly.  If my recent bill had been $200 instead of $150 I would still use LoTW as the benefits are too great.

73,

Chris/NU1O


Title: RE: LOTW
Post by: NU1O on November 08, 2012, 06:36:22 AM

"Invalid Procedure Call or Argument" is without question a defect in AC Log.

     73,

          Dave, AA6YQ

Hi Dave (AA6YQ),
That error was from the LOTW site, not ACLog.  It showed up when using a browser only...  We are were all getting that error from the LOTW site prior to the current error message displayed at the LOTW site...  I am not sure I see how that is ACLog, as opposed to LOTW.  Could you expand?

EDIT:
Note, the error message LOTW had was:
Fatal error: initializeLOTWDB: -709 – CONNECT: (cannot connect to server socket): General database error [initializeLOTWDB: -709 – CONNECT: (cannot connect to server socket)

So you may be right it may be an error in ACLog or how ACLog reacts to a failure to log in...

In fairness to Dave I have been receiving the "Invalid Procedure Call or Argument" error for months. It did not appear recently when LoTW went down after SS. I've received the error when LoTW was working fine for everybody else.

Dave has written a wonderful piece of software but at the time it was more than I needed.  I just wanted something very simple. I have no doubt Dave knows what he's talking about.

73,

Chris/NU1O



Title: RE: LOTW
Post by: AB8MA on November 08, 2012, 06:45:00 AM
Here is an interesting e-mail: (Highlights are mine)

Quote
A few members have asked about Logbook of the World (LoTW) being
inoperable.  Here are the facts about this situation:

The department manager over Information Technology at HQ has confirmed
that no data were lost.  The problem is that storage capacity has been
filled
. Apparently, the fact that storage for LoTW was rapidly being
filled due to the rapid expansion of data input into it was not caught
in time to allow capacity to be expanded before the program shut down.
Staff advised the directors that capacity is being added and that
appropriate alarms are being built into the program to prevent this
situation from happening in the future.  LoTW should resume full
operations essentially momentarily.

I apologize for the inconvenience this situation has caused.

73,

Jim

Jim Weaver, K8JE
Director, Great Lakes Division


Title: RE: LOTW
Post by: 4O4A on November 08, 2012, 06:50:44 AM
Quote
Logbook of the World should be returned to service by 9AM EST 09 Nov 2012 (2012-11-09 1400Z)


Title: RE: LOTW
Post by: K4JK on November 08, 2012, 06:51:44 AM
Here's another update:

http://www.arrl.org/news/logbook-of-the-world-expected-to-be-back-online-on-november-9


Title: RE: LOTW
Post by: NU1O on November 08, 2012, 08:27:20 AM
Here is an interesting e-mail: (Highlights are mine)

Quote
A few members have asked about Logbook of the World (LoTW) being
inoperable.  Here are the facts about this situation:

The department manager over Information Technology at HQ has confirmed
that no data were lost.  The problem is that storage capacity has been
filled
. Apparently, the fact that storage for LoTW was rapidly being
filled due to the rapid expansion of data input into it was not caught
in time to allow capacity to be expanded before the program shut down.
Staff advised the directors that capacity is being added and that
appropriate alarms are being built into the program to prevent this
situation from happening in the future.  LoTW should resume full
operations essentially momentarily.

I apologize for the inconvenience this situation has caused.

73,

Jim

Jim Weaver, K8JE
Director, Great Lakes Division

I am in the Financial Services industry so I am no expert when it comes to software or database management.

Can somebody in the field please tell me how one can be unaware that storage capacity was at its limit?  To use a simplistic analogy I know when a CD, DVD, or my Hard Drive is nearing its capacity and I expand capacity when I'm nearing the limit. 

Shouldn't a DB manager know he/she is about to run out of storage capacity?

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on November 08, 2012, 08:38:22 AM

Can somebody in the field please tell me how one can be unaware that storage capacity was at its limit?  To use a simplistic analogy I know when a CD, DVD, or my Hard Drive is nearing its capacity and I expand capacity when I'm nearing the limit. 

Shouldn't a DB manager know he/she is about to run out of storage capacity?
Chris/NU1O


I am not sure how they missed that one, but it is good that they admitted it!  My guess is that the monitoring software failed...  I can't imagine the ARRL not monitoring disk space.  We used to have a pair of machines monitoring our sites, (three TV stations), if one monitoring system failed, the other should catch the failed monitor, and/or failed web site, or vice-versa. 

My guess is that the ARRL had one system monitoring the LoTW server, and it failed at some time in the past, prior to the disk full problem...  Although someone mentioned that the slowdown should have been a sign, and it should have been, it was missed... 

Beyond the above, I have no clue how the ARRL missed a full disk, assuming that is what it was. 

I am just glad there is no data loss, and I am happy as heck the ARRL is now providing timely updates...  Of course this is all speculation with almost no data, so it is probably way off base...

73's
Dave


Title: RE: LOTW
Post by: W4VKU on November 08, 2012, 10:35:55 AM
You assume there was a paid DB manager involved?


Title: RE: LOTW
Post by: NK7Z on November 08, 2012, 11:11:09 AM
You assume there was a paid DB manager involved?

I do...  Hope I am right!


Title: RE: LOTW
Post by: KF6ABU on November 08, 2012, 11:50:09 AM
my guess is back ups have been failing and not truncating transaction logs.


Title: RE: LOTW
Post by: NK7Z on November 08, 2012, 01:25:41 PM
my guess is back ups have been failing and not truncating transaction logs.

Given that there was no loss of data, my guess is backups have been working, and something else blowed up...

73's
Dave


Title: RE: LOTW
Post by: AD9DX on November 08, 2012, 05:47:24 PM
my guess is back ups have been failing and not truncating transaction logs.

Given that there was no loss of data, my guess is backups have been working, and something else blowed up...

73's
Dave


I have it on good authority that a mouse chewed through a critical  cable in their einac. The real problem came when someone insisted that cloth covered wire had to be used or it just wouldn't work "right"


Title: RE: LOTW
Post by: NI0C on November 09, 2012, 05:04:40 AM
LoTW is back online now, chugging along processing its backlog.  I checked my awards accounts and everything looks okay, except that WPX is still not right.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: NU1O on November 09, 2012, 09:09:28 AM
LoTW is back online now, chugging along processing its backlog.  I checked my awards accounts and everything looks okay, except that WPX is still not right.

73,
Chuck  NI0C


They are still not up to contest QSOs.  My last QSO is from November 2nd and I upload daily.

I hope they learned something from this and changes will be made if needed.  I hope they didn't just apply a Band-Aid to a major wound.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AB8MA on November 09, 2012, 05:44:48 PM
I hope they didn't just apply a Band-Aid to a major wound.

73,

Chris/NU1O

Lets hope they fixed something. Down again?

Quote
Fatal error: initializeLOTWDB: -709 - CONNECT: (protocol error): General database error [initializeLOTWDB: -709 - CONNECT: (protocol error)]


Title: RE: LOTW
Post by: AD9DX on November 09, 2012, 05:52:22 PM
I hope they didn't just apply a Band-Aid to a major wound.

73,

Chris/NU1O

Lets hope they fixed something. Down again?

Quote
Fatal error: initializeLOTWDB: -709 - CONNECT: (protocol error): General database error [initializeLOTWDB: -709 - CONNECT: (protocol error)]

After being down all week the eniac shit its pants again.  This is frustrating. 


Title: RE: LOTW
Post by: NI0C on November 09, 2012, 06:07:38 PM
Getting same error message here.  Also, the ARRL Online DXCC Application appears to be down.  I just tried entering a QSL and got a "browser sync error."


Title: RE: LOTW
Post by: NI0C on November 09, 2012, 06:44:17 PM
LoTW seems to be working again, as I just uploaded a new QSO.  I suspect it is receiving heavy usage now.  I'm still waiting for it to process QSO's made on 04 November.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: NU1O on November 09, 2012, 11:21:47 PM
I have read K1ZZ's columns for over 24 years.  According to Dave we members are the League and the League prides itself on transparency and state of the art technology. Is this really true or is it just a nice slogan?

Where is the transparency?  I would like to see a QST article on exactly what kind of software they are using to maintain the LoTW database, what the hardware is that is being used to maintain the database, the annual revenues LoTW generates along with its annual expenses, and the personal working in the the LoTW department along with their qualifications.

As I wrote in a previous post, Database Management is not my field of expertise, so you guys and ladies who are experts in the field should chime in and add what other information ARRL members should be privy to so that members can decide if the proper software and hardware that is being used is sufficient for the job at hand and if they have hired an adequate number of qualified employees.

As I also wrote in another post, although my current month's AMEX bill for LoTW was about $150 I would be willing to pay more if that's what it takes to have a reliable system that doesn't get backlogged after every large contest.

As soon as feasible I feel K1ZZ needs to write a QST column specifically about LoTW which informs us ARRL members the information I outlined above plus the information the experts in the field conclude membership needs to know so we may make an informed judgement as to whether the problems causing these outages and delays need a completely revamped system or whether some short term fixes are sufficient.

All but the most ardent ARRL sycophants should now realize there are big problems with LoTW and the status quo is totally unacceptable.

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on November 09, 2012, 11:57:17 PM

...ARRL members should be privy to so that members can decide if the proper software and hardware that is being used is sufficient for the job at hand and if they have hired an adequate number of qualified employees.


Hi Chris,

Something about that statement makes me vaguely uncomfortable...  The members should not be making that level of decision, they should make decisions on who the BOD, (and friends), are and who to elect, based on a members positions and performance...  In my experience, (20 years in the data business), having members specifying software, and staffing levels is a sure recipe for disaster.  Follow the chain of command upstream, and push there...  Also I really like your idea of an article on exactly what and how LoTW is built and works. 

73's
Dave


Title: RE: LOTW
Post by: NU1O on November 10, 2012, 12:42:43 AM

...ARRL members should be privy to so that members can decide if the proper software and hardware that is being used is sufficient for the job at hand and if they have hired an adequate number of qualified employees.


Hi Chris,

Something about that statement makes me vaguely uncomfortable...  The members should not be making that level of decision, they should make decisions on who the BOD, (and friends), are and who to elect, based on a members positions and performance...  In my experience, (20 years in the data business), having members specifying software, and staffing levels is a sure recipe for disaster.  Follow the chain of command upstream, and push there...  Also I really like your idea of an article on exactly what and how LoTW is built and works.  

73's
Dave


Dave, I am just very frustrated and your are correct: my suggestion is just micro-managing.  My suggestion would no doubt lead to too many opinions from too many different people.  That's the good thing about this forum.  We all can toss around ideas and those with problems can be dismissed with a sensible rebuttal.

I do feel strongly we deserve a better explanation as to what's going on with LoTW than the current one sentence explanations we are currently receiving on the LoTW home page.  We are paying for this system and we have a right to know whether this will be a common occurrence or if there are plans to scrap the current system and build something much more reliable.  I do know LoTW only has about 53,000 users and these outages aren't going to add users.  We will most likely lose current users and hams who were thinking of using LoTW and that is heading in the opposite direction we need to be going.

I  feel very strongly that Dave, K1ZZ, should write a whole column in QST which explains the system and what went wrong.  If he is serious about transparency that would be a great way to deal with this problem.  In order to lobby our respective Representatives we do need to know what the system is, what went wrong, and whether this is just a temporary fix or a long term solution. My feeling, without knowing the current system, is based upon similar occurrences in the past and I feel they've just applied another  Band-Aid.

73,

Chris/NU1O





Title: RE: LOTW
Post by: NK7Z on November 10, 2012, 05:09:41 AM

Dave, I am just very frustrated and your are correct: my suggestion is just micro-managing.  My suggestion would no doubt lead to too many opinions from too many different people.  That's the good thing about this forum.  We all can toss around ideas and those with problems can be dismissed with a sensible rebuttal.

I do feel strongly we deserve a better explanation as to what's going on with LoTW than the current one sentence explanations we are currently receiving on the LoTW home page.  We are paying for this system and we have a right to know whether this will be a common occurrence or if there are plans to scrap the current system and build something much more reliable.  I do know LoTW only has about 53,000 users and these outages aren't going to add users.  We will most likely lose current users and hams who were thinking of using LoTW and that is heading in the opposite direction we need to be going.

I  feel very strongly that Dave, K1ZZ, should write a whole column in QST which explains the system and what went wrong.  If he is serious about transparency that would be a great way to deal with this problem.  In order to lobby our respective Representatives we do need to know what the system is, what went wrong, and whether this is just a temporary fix or a long term solution. My feeling, without knowing the current system, is based upon similar occurrences in the past and I feel they've just applied another  Band-Aid.


Hi Chris,

I agree totally with what you are saying, my hope is that this incident will, or has forced a re-think of the current system.  It is clear that something is being pushed to limits, and beyond.  The concept of transparency is such a good thing, but so very elusive in practice.  It really does not matter to me that I am paying for it, my dues did not go up as a result of LoTW being added, what does matter to me is that the act of creating the system, promoting the system, the fact that the ARRL has it's name all over this project, and that it appears to be having some significant issues, which appear to be unaddressed is the real issue.  It tarnishes the ARRL's name, and appears to be half thought out.  You are absolutely correct, a single article in QST covering how it works, and what happened, with a plan to stop it from happening again would go a very long way to mitigate LoTW in my book.

This is very much like the digital edition of QST...  I came very close to writing a scathing review of the digital edition because of the implementation, I didn't because the act of moving forward, towards digital publication is good, and to hammer it so soon would have been a net negative.  It should have been done as an e-book, not whatever propitiatory system they used.  From my point of view the digital QST is not nearly as easy to use as any e-book I own.  It is virtually unreadable on my iPad, and I refuse to sit and read it on my computer.  I see that they have just added an iTunes application for accessing content, so maybe that is better...  I will check it out...

I have always been one of those hams that has been active on air, and a member of ARRL, but inactive in watching the ARRL, and the politics of ham radio.  I have worked my entire life up until last November, and intend on becoming active beyond just getting on the air now that I am retired.

As an aside it is a pleasure swapping ideas with you... 

73's
Dave


Title: RE: LOTW
Post by: N7SMI on November 10, 2012, 06:45:38 AM
We will NOT tolerate such civil and courteous dialogue here. How dare one of you kindly rebut the other and the first simply accept the rebuttal and apologize!?! It's just not normal!

But seriously, I think dialogue and openness from the ARRL is needed regarding LoTW. Many of us are heavily invested in it - more with time, QSOs, DXPedition funding and commitments, software functionality, etc. than in ARRL dues. It's becoming increasingly clear that the infrastructure is not future-proof and is unlikely to handle such sustained growth for much longer, and the present is barely functional when we need it the most.


Title: RE: LOTW
Post by: NK7Z on November 10, 2012, 07:19:59 AM

We will NOT tolerate such civil and courteous dialogue here. How dare one of you kindly rebut the other and the first simply accept the rebuttal and apologize!?! It's just not normal!


I know...  I felt bad not knee jerking, and screaming...  :)  I'll try and do better next time!  :)

73's
Dave


Title: RE: LOTW
Post by: NK7Z on November 10, 2012, 07:27:08 AM
I think dialogue and openness from the ARRL is needed regarding LoTW. Many of us are heavily invested in it - more with time, QSOs, DXPedition funding and commitments, software functionality, etc. than in ARRL dues. It's becoming increasingly clear that the infrastructure is not future-proof and is unlikely to handle such sustained growth for much longer, and the present is barely functional when we need it the most.

I had not even thought about the cost of donations to DXpeditions...  That is an interesting and useful way to monetize LoTW.  I just hope that the ARRL actually fixes the problems, and does not just patch them.  If eQSL can keep a system running, then I would expect ARRL can keep a system running, if they don't, it speaks volumes for their internal organization practices. 

You are correct, if this is not corrected, the continued growth of LoTW will cease...

73's
Dave
PS:  I forgot to yell again...  Sorry!


Title: RE: LOTW
Post by: NU1O on November 10, 2012, 07:37:11 AM

Hi Chris,

I agree totally with what you are saying, my hope is that this incident will, or has forced a re-think of the current system.  It is clear that something is being pushed to limits, and beyond.  The concept of transparency is such a good thing, but so very elusive in practice.  It really does not matter to me that I am paying for it, my dues did not go up as a result of LoTW being added, what does matter to me is that the act of creating the system, promoting the system, the fact that the ARRL has it's name all over this project, and that it appears to be having some significant issues, which appear to be unaddressed is the real issue.  It tarnishes the ARRL's name, and appears to be half thought out.  You are absolutely correct, a single article in QST covering how it works, and what happened, with a plan to stop it from happening again would go a very long way to mitigate LoTW in my book.

This is very much like the digital edition of QST...  I came very close to writing a scathing review of the digital edition because of the implementation, I didn't because the act of moving forward, towards digital publication is good, and to hammer it so soon would have been a net negative.  It should have been done as an e-book, not whatever propitiatory system they used.  From my point of view the digital QST is not nearly as easy to use as any e-book I own.  It is virtually unreadable on my iPad, and I refuse to sit and read it on my computer.  I see that they have just added an iTunes application for accessing content, so maybe that is better...  I will check it out...

I have always been one of those hams that has been active on air, and a member of ARRL, but inactive in watching the ARRL, and the politics of ham radio.  I have worked my entire life up until last November, and intend on becoming active beyond just getting on the air now that I am retired.

As an aside it is a pleasure swapping ideas with you... 

73's
Dave
Hi Dave,

I do not think I was as active the first two years I was licensed as the past two years but I never got involved with clubs or the politics of amateur radio at anytime and I now think that was a big mistake.

I only started to participate in this forum about a year to a year and a half ago and I'm very glad I decided to become an active participant. I've learned a lot that did not pertain solely to DX. Perhaps there should be a separate forum for amateurs to debate ARRL policies. It might be a way for an amateur with some fresh ideas to get elected to office and then communicate with the members he represents and even those he does not.  I do not currently see that dialogue and I think more transparency is needed. I strongly feel ARRL members are entitled to an explanation of the current LoTW problems, the fixes, and where we are heading.

Now that you are retired I think it would be wonderful if you got involved in League politics.  

When I was much younger I followed local, state, and national politics very closely. I still follow national politics very closely but all the negativity has turned me off. I would never get involved in politics but someday I may get involved in League matters.  You could be my mentor!  :)

73 and best wishes,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on November 10, 2012, 07:49:19 AM

I only started to participate in this forum about a year to a year and a half ago and I'm very glad I decided to become an active participant. I've learned a lot that did not pertain solely to DX. Perhaps there should be a separate forum for amateurs to debate ARRL policies. It might be a way for an amateur with some fresh ideas to get elected to office and then communicate with the members he represents and even those he does not.  I do not currently see that dialogue and I think more transparency is needed. I strongly feel ARRL members are entitled to an explanation of the current LoTW problems, the fixes, and where we are heading.

Chris/NU1O

The idea of a new forum for discussion of ARRL politics would be handy for this sort of discussion...  I hope someone at eHam is watching this and creates one...  I am not a mentor, nor shall I ever be, but thank you.  Much the reverse is how it should work! 

73's
Dave


Title: RE: LOTW
Post by: NI0C on November 10, 2012, 08:09:55 AM
The 42 hour processing delay currently reported on the LoTW website is obsolete.  They were reporting a 42 hr delay shortly before the big system crash a few days ago.  I doubt that any processing of the backlog took place while the system was being fixed, and meanwhile people had Sweepstakes logs to upload, not to mention daily QSO's. 

As of this morning, I'm still waiting for QSO's uploaded last Sunday to show up.  So, we're about six days behind now.  The overall numbers seem to be cranking away pretty fast, though, so maybe the system will catch up before CQ WW CW. 

73,
Chuck  NI0C


Title: RE: LOTW
Post by: WS3N on November 10, 2012, 09:52:40 AM
As I also wrote in another post, although my current month's AMEX bill for LoTW was about $150 I would be willing to pay more if that's what it takes to have a reliable system that doesn't get backlogged after every large contest.

Having seen this $150 figure twice, I must say that I wonder about its relevance. How much would you have paid without LoTW. I don't mean in terms of stamps, etc., but just in what you would have sent to the ARRL? In other words, how much of that actually goes toward the costs of LoTW?

That having been said, I'm not disagreeing with your main points. I'm sure everyone would like to see greater reliability, transparency, etc.

Personally, I can't really complain. As a DXLab user, accessing LoTW couldn't be much simpler. And, since uploads are so easy, I usually do so very often, resulting in small files that are quickly processed. Also, I don't provide any financial support, unless they take something from membership dues, although I would gladly pay directly for use. My biggest concern is the impact these problems have on the expansion of the user pool.

Jack


Title: RE: LOTW
Post by: SV1XV on November 10, 2012, 10:46:20 AM
I doubt that any processing of the backlog took place while the system was being fixed, ...
Yes, it did. When the system came up I found 3 QSLs from ON5UR with QSL timestamps around 2012-11-09 03:48:16.


Title: RE: LOTW
Post by: N9KX on November 14, 2012, 11:20:51 AM
The current notice dated 11/13/2012 says LoTW is approx. 48 hours behind but is still processing log data ok. I hope that is right, something tells me it is getting worse.  Given that each new single log submission requires cross-checking that entry with every other entry in the system (right?) -- if there are problems now it will only get MUCH worse as users and log submissions increase daily.

Hopefully there is a BIG upgrade to the system in the works -- if not this year then next.  Anyone know if ARRL has budgeted for that?  Or how much of LoTW credit fees goes directly toward system upkeep?

 


Title: RE: LOTW
Post by: SV1XV on November 14, 2012, 11:38:38 AM
Given that each new single log submission requires cross-checking that entry with every other entry in the system (right?) --
Why? Isn't the QSO database indexed at least by callsign?


Title: RE: LOTW
Post by: NU1O on November 14, 2012, 12:52:12 PM
We will NOT tolerate such civil and courteous dialogue here. How dare one of you kindly rebut the other and the first simply accept the rebuttal and apologize!?! It's just not normal!
Jared,

From one amateur astronomer to another you have produced some very beautiful pictures of the Heavens and the ISS in transit.  I think your photo of M51 is your Magnum Opus to date but all your Messier work is beautiful.  It must be nice not having the light pollution of the East Coast Corridor.

73,

Chris/NU1O



Title: RE: LOTW
Post by: AB8MA on November 14, 2012, 09:14:48 PM
My upload from 22:51Z has been processed already. The back log has decreased considerably.

edit ... Sorry, it was an earlier upload. My latest QSO's were not included.


Title: RE: LOTW
Post by: N9KX on November 15, 2012, 10:22:24 AM
Given that each new single log submission requires cross-checking that entry with every other entry in the system (right?) --
Why? Isn't the QSO database indexed at least by callsign?

 :-[ ok, i stand corrected ...

The site is now saying the system is 48 hours behind and has a large queue to process...  sorry if this info is somewhere i've missed, but is there any info on upgrades that have taken place in the past and will take place in the future and how well they will speed up the processing power and capacity of the system?


Title: RE: LOTW
Post by: V47JA on November 15, 2012, 04:30:16 PM
Hi,

I Uploaded about 6000 Q's to LoTW today. Anybody have any idea as to the current processing time for LoTW?

73 and DX,

John  V47JA  -  W5JON


Title: RE: LOTW
Post by: W1VT on November 15, 2012, 04:46:27 PM
Nov 15, 2012: LoTW Log Processing is About 4 Days Behind -- The system is processing log data a little slower than usual, but is working.  There is a very large number of logs in the queue -- thanks for your patience and for using LoTW, and we will continue to work to streamline the system.

From the LoTW page


Title: RE: LOTW
Post by: NI0C on November 16, 2012, 06:12:49 AM
Quote
LoTW Log Processing is About 4 Days Behind

I don't believe their estimates anymore; they are always obsolete and optimistic.  I'm still waiting for my 80m QSO with PT0S to show up in my QSO's list, let alone QSL's.  I uploaded that QSO early Sunday morning, so it's been over five days now.   

The PT0S team has uploaded some QSO's several days ago, but when you search for PT0S, the LoTW website returns "No log data." 

I've been a fan of LOTW since its inception.  However, LoTW has rapidly deteriorated from a fine running machine to a sputtering, stalling jalopy.  It needs to be fixed very soon.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: AB8MA on November 16, 2012, 07:17:17 AM
Quote
thanks for your patience and for using LoTW, and we will continue to work to streamline the system.

I would be interested to hear more details on what is meant by this statement, and what steps are planned. As others have mentioned, an article in a future QST regarding LoTW implemented and planned changes would go a long ways toward recovering any lost confidence in LoTW.


Title: RE: LOTW
Post by: N1AM on November 16, 2012, 11:12:01 AM
This is certainly turning into a real embarassment for the League. Now processing is 4 days behind and Sweepstakes SSB is on this weekend.  I agree that an in depth explanation should appear in QST however, in addition, the League should immediately post an extensive explanation on their web site regarding what is really going on with LOTW.  Hey ARRL please hire a full time
DB Manager.
John N1AM 


Title: RE: LOTW
Post by: N2RJ on November 16, 2012, 01:19:27 PM
This is certainly turning into a real embarassment for the League. Now processing is 4 days behind and Sweepstakes SSB is on this weekend.  I agree that an in depth explanation should appear in QST however, in addition, the League should immediately post an extensive explanation on their web site regarding what is really going on with LOTW.  Hey ARRL please hire a full time
DB Manager.
John N1AM 


I don't know if the league has enough money to pay for one. I think that is a big holdback. Maybe if they started charging a monthly fee for LoTW it would be better. I would chip in a few bucks if they did that. But I can see how it would discourage some people who already have a gripe about the system. So I think what you're seeing here is you get what you pay for.


Title: RE: LOTW
Post by: W2IRT on November 16, 2012, 02:24:07 PM
I don't know if the league has enough money to pay for one. I think that is a big holdback.
Maybe if they stopped funneling a huge percentage of their resources into the Hamsexy/Shack-on-a-belt  Emcomm crowd they'd have a little something left in the kitty for a program that adds to their international stature. They've built the DXCC program up to be the pinnacle of lifetime achievement DXing, yet they keep treating it like the red-headed stepchild of ham radio.

Maybe if they started charging a monthly fee for LoTW it would be better. I would chip in a few bucks if they did that. But I can see how it would discourage some people who already have a gripe about the system. So I think what you're seeing here is you get what you pay for.
If they did that you'd see participation fall like Wile E. Coyote off a cliff. Especially internationally, which is the avenue where it needs the most promotion. LoTW currently has just north of 50,000 contributors and 455M individual QSO records. Is that a big enough database to warrant a full-time DBA? Especially given that it really only bogs down severely between the end of October and the end of November and again after the spring contests. Beyond that it's just the odd upload throughout the day.

Still, it does need more resources than it's receiving at the moment. IMO, what it really needs is a ground-up rebuild. Let the shack-on-a-belters go elsewhere and funnel some cash to one part of the hobby that's got a proven track record of being extremely active, will be around for more than a year or two and has money to spend.


Title: RE: LOTW
Post by: K4JK on November 16, 2012, 03:55:02 PM
LOTW was a solid system 3-4 years ago but I agree, it's being overwhelmed and wasn't built to withstand the demand it currently undergoes. Having had to manage SQL databases on limited budgets, it's doable but very hard to make them scalable. As long as you have limited demand and routine usage it's not a big deal to do something like that on the cheap. But once your system grows like LOTW has it takes more horsepower on both the front and back ends to keep everything running smoothly.

So I can empathize with whoever is running the technical side of things, as they obviously only have so many resources to keep the system going. They are probably doing as good of a job they can with what has been provided to them. I'd be great if the league would agree to rebuild the system on one of the cloud-based providers so that extra resources can be given to it based on demand, but that costs $$$$. While they did that, they could also look into overhauling the current technical aspects of the "paper" DXCC system so that it integrates more seamlessly with LOTW. Might as well make it part of the same project.

IMO the League did a great thing with LOTW, but they need to keep following through to make sure it stays the defacto logging/award system for DXers, full-time paper chasers and casual operators alike. It would be a shame if it became a victim of it's own success.



Title: RE: LOTW
Post by: N2RJ on November 16, 2012, 05:55:05 PM
I hear talk about hardware upgrades. That's all well and good but there's so much that can be done with software. I deal with terabytes of data all the time. Have been for the better part of 10 years, from state Government to large financial companies, and now with high traffic websites and CMS systems... you'd be surprised what I do with little hardware.


Title: RE: LOTW
Post by: K3VAT on November 16, 2012, 11:07:28 PM
LoTW definitely working and I'm now seeing more and more confirmations come in.  Latest confirmations are still ~6 days ago, but this is progress.  They'll probably get down to a 1 or 2 day delay before another 2.5 million (guestimation) QSOs hit from the ARRL SS.  73, Rich, K3VAT


Title: RE: LOTW
Post by: AJ4RW on November 17, 2012, 09:36:21 AM
Maybe this has been covered before but I don't feel like looking around for it.  A while back I read somewhere that LOTW has 50,000 users +/- but does anyone really know how many users there are currently with LOTW?  And if so, how many?
Randy


Title: RE: LOTW
Post by: SV1XV on November 17, 2012, 12:39:01 PM
Have a look in the site of HB9BZA:  http://www.hb9bza.net/lotw-users-list



Title: RE: LOTW
Post by: AJ4RW on November 18, 2012, 01:03:31 AM
Thanks Costas for the info.  I guess I'm trying to figure out why eQSL has so many unique users (203,566) but yet has a lot fewer interruptions in service.  I know they're 2 completely different services but still have a common goal, providing QSL to amateurs.  When looking at LOTW, what comes to mind is the computer I used when going to college.  In order to boot it up, you had to use a 5 1/4" floppy disk and it had no hard drive.  Something is wrong with the current system used at LOTW.  ARRL, quit being so cheap and update/upgrade the system or whatever else needs to be done!!!


Title: RE: LOTW
Post by: NU1O on November 18, 2012, 08:11:13 AM
So there are 58,093 calls in 339 current DXCC entities.  I don't think I need to tell anybody the only entity/person (he's is really both a DXCC entity and individual user) who is not using LoTW is Apollo on Mt. Athos.

73,

Chris/NU1O



Title: RE: LOTW
Post by: N2RJ on November 18, 2012, 11:57:57 AM
Thanks Costas for the info.  I guess I'm trying to figure out why eQSL has so many unique users (203,566) but yet has a lot fewer interruptions in service.  I know they're 2 completely different services but still have a common goal, providing QSL to amateurs.  When looking at LOTW, what comes to mind is the computer I used when going to college.  In order to boot it up, you had to use a 5 1/4" floppy disk and it had no hard drive.  Something is wrong with the current system used at LOTW.  ARRL, quit being so cheap and update/upgrade the system or whatever else needs to be done!!!

eQSL is not without its problems. What I notice is that they tend to have safeguards in place like timeouts for long running transactions. This prevents a total crash of the system but when doing some long running stuff (like submitting a ton of QSOs for WPX award) it times out. eQSL also has realtime upload I think with a SOAP query. With LoTW they only accept uploads in batches of whole log files. I think if they adopted a mechanism for realtime uploads that would help spread out the load.


Title: RE: LOTW
Post by: W2IRT on November 19, 2012, 10:56:07 AM
Four days my feathered fanny. The last upload of mine that's showing on LoTW was on Nov. 9th. I've made 2 or 3 incremental uploads since that time and not one is showing yet.


Title: RE: LOTW
Post by: K3VAT on November 19, 2012, 11:35:56 AM
Four days my feathered fanny. The last upload of mine that's showing on LoTW was on Nov. 9th. I've made 2 or 3 incremental uploads since that time and not one is showing yet.

Now ARRL says 6 Days.  My last uploaded Q that shows up is from 5 days ago.   Nearly 2 Million (guestimation) new Qs over the last 24 hours doesn't help.  Ugh!  


Title: RE: LOTW
Post by: NI0C on November 19, 2012, 11:39:25 AM
On a positive note, I just received my first LoTW confirmation from PT0S!  As Murphy's law for QSL's dictates (Least needed QSL's arrive first), it was for my 40m QSO.  Somehow LoTW never processed my uploads for my other QSO's, so I resent them.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: K7KB on November 19, 2012, 11:50:13 AM
Two of my PT0S confirmations came through this morning on LOTW, 17M CW and 40M SSB. So at least those are beginning to process and it's nice to get that one confirmed!

John K7KB


Title: RE: LOTW
Post by: AA5VU on November 19, 2012, 12:41:11 PM
On a positive note, I just received my first LoTW confirmation from PT0S!  As Murphy's law for QSL's dictates (Least needed QSL's arrive first), it was for my 40m QSO.  Somehow LoTW never processed my uploads for my other QSO's, so I resent them.

73,
Chuck  NI0C


Thanks for the posting. It is good to know the PT0S operation are uploading to LoTW.  The PT0S QRZ page says NO to LoTW; however, the operation web page indicated they will upload when they can. TNX again for letting us know they are trying.

My PT0S QSO's have been uploaded to LoTW but have not been processed. The last upload processed was 8-Nov-2012 as of 19-Nov nothing new has been processed. Hopefully, the log jam will clear soon.

73 Dick AA5VU


Title: RE: LOTW
Post by: NI0C on November 19, 2012, 01:03:28 PM
Dick, AA5VU,
You're welcome.  I got an e-mail today from Chris, HA5XA, saying that about a third of the PT0S QSO's have been processed by LoTW thus far.
73,
Chuck  NI0C


Title: RE: LOTW
Post by: W5RDW on November 19, 2012, 01:07:26 PM
I received my 17, 20 and 40 meter CW LOTW confirmations today!


Title: RE: LOTW
Post by: AA5VU on November 19, 2012, 01:45:38 PM


[/quote]

Thanks for the posting. It is good to know the PT0S operation are uploading to LoTW.  The PT0S QRZ page says NO to LoTW; however, the operation web page indicated they will upload when they can. TNX again for letting us know they are trying.

My PT0S QSO's have been uploaded to LoTW but have not been processed. The last upload processed was 8-Nov-2012 as of 19-Nov nothing new has been processed. Hopefully, the log jam will clear soon.

73 Dick AA5VU
[/quote]




Title: RE: LOTW
Post by: LA9XSA on November 19, 2012, 02:58:30 PM
Maybe if they stopped funneling a huge percentage of their resources into the Hamsexy/Shack-on-a-belt  Emcomm crowd they'd have a little something left in the kitty for a program that adds to their international stature.
It wouldn't exactly do any good for international stature and good will for the ARRL or ham radio as a whole if they shut down ARES, the NTS, Ham Aid, etc. to turn LotW into Fort Knox. "Sorry, earthquake victims in the Caribbean, California or wherever - we shut down that emcomm stuff to buy a 99.999999% uptime quadruple-redundant system to manage our hobby diploma program."

Also, realize that contesting, DX and Emcomm is not opposing goals. A good contest station will probably be a good traffic handling station; a good DXer probably knows the right mode and frequency to reach a particular disaster-stricken area in the 3rd world (and might even know every active ham who lives there); political representatives probably find it easier to let us use valuable frequencies to tell each other that we're 59 years old, talk about antennas and complain about the president all day, when they understand that amateur operating skills can be used for organized emergency response from time to time.

Even with a few days of downtime in a year, LotW doesn't really have to have better uptime than it does already - it just needs to catch up on the backlog when it does go down.


Title: RE: LOTW
Post by: NI0C on November 19, 2012, 03:24:49 PM
LA9SXA wrote:
Quote
Even with a few days of downtime in a year, LotW doesn't really have to have better uptime than it does already - it just needs to catch up on the backlog when it does go down. 
 

What happened to LoTW recently was probably a "perfect storm," as it went down during a very busy contesting and Dx'ing time.  What we need now is some assurance that uploaded QSO's will eventually make it into the system, as well as assurance that the backlog will indeed go down.  So far, it's just been going up.  It would also help if ARRL would give more frequent and accurate estimates of the backlog until the situation improves.

73,
Chuck  NI0C



Title: RE: LOTW
Post by: W2IRT on November 19, 2012, 03:52:04 PM
What happened to LoTW recently was probably a "perfect storm," as it went down during a very busy contesting and Dx'ing time.

So true. If only there was a way for the League to know ahead of time just when these busy "contests" would take place so they could have the resources in place to handle the load.

Quote from: LA9SXA
It wouldn't exactly do any good for international stature and good will for the ARRL or ham radio as a whole if they shut down ARES, the NTS, Ham Aid, etc. to turn LotW into Fort Knox. "Sorry, earthquake victims in the Caribbean, California or wherever - we shut down that emcomm stuff to buy a 99.999999% uptime quadruple-redundant system to manage our hobby diploma program."

I'm not saying shut down ARES (does ARES even need the League to function?). I'm saying look at the overall tone of QST, the official voice of the League. They pay lip service to serious DXers and contesters at best. We're the crazy uncle to their way of thinking. Their focus is probably 70% emcomm, with the rest split between build-it-yourself projects that were once taught at the elementary-school level, handing out PR press releases for new gear (please, tell me the last time you say a truly, brutally-critical review in QST) and bedwetting over arcane spectrum issues. I feel that the League knows that DXers will continue to flock to the DXCC program so they have no incentive to throw any money at it. So long as it exists it will make them money with little need to do anything.

Yet go to Dayton and look at their vast exhibit. Very little of it is aimed above the Technician or General level of knowledge. It's as if they see the pinnacle of annual HF operation being Field Day....which relates to emergency preparedness, naturally! There's precious little on the topic of 160m operation in QST (but loads of stuff for 6m and up), and their SSB and CW DX contests get perfunctory mention but hardly the top billing that they both deserve; namely as a venue for the world's best contest operators to strut their stuff on the world stage.

Trust me, if there were any component of LoTW that had to do with giving PR to emcomm, the League would be on it like white on rice and they'd have their platinum-plated quadruply-redundant bulletproof system in place.


Title: RE: LOTW
Post by: N9KX on November 20, 2012, 05:05:48 AM

So true. If only there was a way for the League to know ahead of time just when these busy "contests" would take place so they could have the resources in place to handle the load.

thanks for pinning my sarcasm meter and making me smile

Trust me, if there were any component of LoTW that had to do with giving PR to emcomm, the League would be on it like white on rice and they'd have their platinum-plated quadruply-redundant bulletproof system in place.

so, would a DB manager and budget be easier to get if we use emcomm as the hook, line and sinker? can some one more creative than me in this sphere please get on that like it is an ATNO?   ;D



Title: RE: LOTW
Post by: V47JA on November 20, 2012, 05:37:43 AM
Hi,

I Uploaded about 6000 Q's to LoTW today. Anybody have any idea as to the current processing time for LoTW?

73 and DX,

John  V47JA  -  W5JON


Hi,

To answer my own question, it took a little less then 5 days.  All 6000 Q's were on LoTW this morning.

73 and DX,

John   V47JA


Title: RE: LOTW
Post by: NU1O on November 20, 2012, 09:04:20 AM
(please, tell me the last time you say a truly, brutally-critical review in QST)


I have been reading QST long before I was a licensed amateur.  I'd guess I have been reading it for 35 to 40 years now and I have NEVER read a brutally honest review which is why except for lab specs I don't rely on QST for any objective product reviews. Their main concern is to make sure the advertising dollars keep rolling in so the last thing they are going to do is offend a client by saying their product is junk and should not be bought.  When I want honest reviews on products I use the review section on Eham.  I really don't pay ARRL dues to read QST.  

I have a question for somebody who is willing to be brutally honest. During Hurricane Sandy I listened to some of the weather nets where various hams where checking in with barometric, temp., and wind data. I've also been at my place on Cape Cod and they have a daily morning net on 2 meters where hams from all over the Cape and Islands check in with weather data. I have a good ham friend who used to be very big into weather nets and he still has a nice weather station mounted on his tower. During the storm I asked him why all that weather data is needed and he never gave me a good reason except to keep saying it's important. My retort was with NOAA, the various WX stations at military and general aviation airports, and all the TV and radio stations that have weather stations, that much, or all, of that data is not used by any professional.

Is the purpose of the weather nets to have a pool of operators who are trained at delivering accurate information for emergencies or is that weather information actually put to good use by professionals?  

I have a pretty decent station and this spring I installed a backup generator which runs on natural gas and will power the whole house plus the station and amplifier. It kicks in automatically after about 30 seconds w/o grid power. There is a local ham who is on the police force and I told him if I could ever be of service during an emergency I would naturally be willing to help.  Other than the occasional rescue when amateur radio is used are we actually going to be of any use in an emergency or is this just a big dog and pony show?  

I am asking these questions in all sincerity (and probably naively as I was never involved with Emcomm) and I know they are very controversial questions but I am a born cynic and when we were hit hard here last Halloween and without power for a whole week most hams were without power and of no use. Personally, I don't see hams coming to the rescue in a great emergency such as when the old CW keys were brought out in the movie Independence Day but, OTOH, I would like to read a reply which convinces me I am wrong.

My thanks to whomever steps up to the plate and if a number reply maybe this should be moved to its own topic.

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on November 20, 2012, 10:39:43 AM
Hi Chris,

The WX nets may be putting into practice a training process...  In MARS, we run traffic, which is in essence traffic about how much traffic we have run...  i.e. a report on the number and type of messages which are sent and on which nets.  This is done to promote training in the procedures which may someday be needed.  A good number of folks will run this traffic, they are net control stations, so they report net status almost daily.  That traffic, gives them a practice on running traffic.  At least that is all I can think of...


Title: RE: LOTW
Post by: W2IRT on November 20, 2012, 08:10:29 PM
I run a weather station at home. A Davis Vantage PRO-2. It's calibrated against the precision instruments at KCDW, less than two miles away. I upload my data to Weather Underground ever 2.75 seconds and it in turn is broadcast everywhere. I also send every 5 minutes to the FINDU system. Most serious weathergeeks do the same and the NWS automatically gathers and verifies the (relative) accuracy of the data.

About all I can think of is that the data sent over weather nets could be of use within a storm's circulation when it knocks out power to the region. For example, during and after Sandy the official weather station at KCDW was down for over a week, as was Fairfield OEM's private station. I was one of the very few in western Essex County able to generate accurate reports of temperature, humidity, pressure, precipitation and wind direction (and to a lesser degree wind speed, which is hampered by trees).

I also upload official precipitation reports to a volunteer network called CoCoRAHS, which collects NWS-level accurate reports from around the country.

If you're curious, my WX station data is http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KNJWESTC2 (http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KNJWESTC2) and my precipitation reports can be found at http://data.cocorahs.org/cocorahs/maps/?country=USA&state=NJ&lat=40.879633&lon=-74.227041&date= (http://data.cocorahs.org/cocorahs/maps/?country=USA&state=NJ&lat=40.879633&lon=-74.227041&date=).


Title: RE: LOTW
Post by: NI0C on November 21, 2012, 05:27:32 AM
Back to LoTW processing times, for anyone interested.  Based on a couple of recent QSO's in my log, the processing time is roughly 107 hours, about four and a half days. 

73,
Chuck  NI0C


Title: RE: LOTW
Post by: W2IRT on November 21, 2012, 05:44:48 AM
Back to LoTW processing times, for anyone interested.  Based on a couple of recent QSO's in my log, the processing time is roughly 107 hours, about four and a half days. 

Very odd. I'm still not getting anything after CQWW, and I know I've uploaded at least three valid .TQ8 files since.


Title: RE: LOTW
Post by: NK7Z on November 21, 2012, 05:47:05 AM
Back to LoTW processing times, for anyone interested.  Based on a couple of recent QSO's in my log, the processing time is roughly 107 hours, about four and a half days. 

Very odd. I'm still not getting anything after CQWW, and I know I've uploaded at least three valid .TQ8 files since.

I have had a single confirm in the past week, got it last night...  I uploaded maybe 20 new contacts 4 days ago, and no validation that they have been accepted yet...


Title: RE: LOTW
Post by: N9KX on November 21, 2012, 06:07:24 AM
The LoTW 'Find Call' tool shows: Last upload for K9AIM: 2012-11-19 00:37:10Z

That seems odd given that the LoTW site has a post on 11/19/2012 that suggests the system is 6 days behind and given that uploads by me from before 2012-11-19 at 00:37:10Z do not seem to have been processed yet ...   ???


Title: RE: LOTW
Post by: AB8MA on November 21, 2012, 06:31:08 AM
The LoTW 'Find Call' tool shows: Last upload for K9AIM: 2012-11-19 00:37:10Z

That seems odd given that the LoTW site has a post on 11/19/2012 that suggests the system is 6 days behind and given that uploads by me from before 2012-11-19 at 00:37:10Z do not seem to have been processed yet ...   ???

I think their use of language dates back to when a 4 hour delay in processing was news.

I show "Last upload for AB8MA: 2012-11-19 21:27:24Z". This isn't an actual upload, but rather the date the file I uploaded  ‎November ‎14, ‎2012, ‏‎20:50:54Z was finally processed.


Title: RE: LOTW
Post by: AF3Y on November 21, 2012, 07:20:40 AM
My last upload date shows as 10-25, but I have uploaded a couple small files since then. Does this mean, they never got the files.  When I uploaded them, it said they were accepted.

Either way, LOTW kinda sucks. They either need to fix/upgrade it, or just discontinue it ???

Why am I getting the feeling that they dont give a damn? They must have some kind of plan in mind
to overhaul or at least tweak the thing.

73, Gene AF3Y


Title: RE: LOTW
Post by: AJ4RW on November 21, 2012, 08:42:53 AM
Gene, you might want to get any uploads done before this weekends CQ WW CW contest.
Randy


Title: RE: LOTW
Post by: N9KX on November 21, 2012, 08:44:02 AM

Why am I getting the feeling that they dont give a damn? They must have some kind of plan in mind
to overhaul or at least tweak the thing.

73, Gene AF3Y

So far it seems they are dealing with the bottlenecks by kicking the can farther down the road a little.  And, they are not communicating the details of the problem(s) nor the proposed solution and timeline.  That pattern can lead one to believe they don't give a damn...

And that leads me to wonder about who originally came up with the LoTW system and whether they are not as frustrated as anyone?  What was once shiny and had a wow factor now has started limping along. 

LoTW has saved me hundreds of dollars I don't have for QSLing and thus allowed me to attain DXCC and also helps me enthusiastically continue to fish for DX on a daily basis.  So while I do appreciate it, I would love to see the ARRL make it state of the art...


Title: RE: LOTW
Post by: NU1O on November 21, 2012, 09:27:57 AM
I upload just about everyday that I am on the rig and today's LoTW confirmations are for QSOs on November 12th. We will have a real mess if the backlog from Sweepstakes is not cleared out by the time the uploads for CQWW start coming in.

I believe I read LoTW generated $900,000 in revenue and the claim was made they are not charging market rates. I have invested in small local banks for years and that sounds like more than some spend on data processing. I'm talking about banks with $500 million to a $1 Billion in assets.

I can't recall when and where I got that $900K number from but it had to be somewhere on the ARRL's site.

73,

Chris/NU1O


Title: RE: LOTW
Post by: ND4X on November 21, 2012, 11:01:09 AM
Can anyone explain how uploads from some stations (PT0S for instance), show upload times of today - while normal users have multiple uploads that are not showing up yet from days ago?

Paul - ND4X


Title: RE: LOTW
Post by: AF3Y on November 21, 2012, 11:05:04 AM
Gene, you might want to get any uploads done before this weekends CQ WW CW contest.
Randy

Thanks..... I am uploaded thru last nite as of this morning.  73, Gene AF3Y


Title: RE: LOTW
Post by: AB8MA on November 21, 2012, 11:30:46 AM
Can anyone explain how uploads from some stations (PT0S for instance), show upload times of today - while normal users have multiple uploads that are not showing up yet from days ago?

Paul - ND4X


I do not think LoTW knows when a file is actually uploaded. The "upload times of today" was actually the timestamp for the last time a file was Processed, not Uploaded.

again,

I show "Last upload for AB8MA: 2012-11-19 21:27:24Z". This isn't an actual upload, but rather the date the file I uploaded  ‎November ‎14, ‎2012, ‏‎20:50:54Z was finally processed.


Title: RE: LOTW
Post by: K3VAT on November 22, 2012, 03:46:07 AM
Just received from the LoTW several confirmations - the latest one confirmed occurred ~100 hours ago (I upload once or twice a day depending on the number of QSOs that I make).

Re: PT0S - I've received confirmation (2 so far) for my later QSOs - in other words, my earliest QSO isn't yet confirmed by LoTW (but it is listed on the PT0S site).

So who knows exactly the mechanism of why certain stations' QSOs appear before others when it is supposed to be a FIFO (first in, first out) [rhetoric question].

GL 73, Rich, K3VAT


Title: RE: LOTW
Post by: NK7Z on November 22, 2012, 03:59:43 AM
Just received from the LoTW several confirmations - the latest one confirmed occurred ~100 hours ago (I upload once or twice a day depending on the number of QSOs that I make).

Re: PT0S - I've received confirmation (2 so far) for my later QSOs - in other words, my earliest QSO isn't yet confirmed by LoTW (but it is listed on the PT0S site).

So who knows exactly the mechanism of why certain stations' QSOs appear before others when it is supposed to be a FIFO (first in, first out) [rhetoric question].

GL 73, Rich, K3VAT

Thank you for posting that... I am waiting for a contact with them to show on both LoTW and on their web site...  What you are saying leads me to believe they are uploading in parts, and not in the monotone increasing time intervals...  I have hope again!!


Title: RE: LOTW
Post by: K3VAT on November 22, 2012, 04:13:00 AM
Very good Dave,

So you might want to revise your QRZ.COM page and perhaps change that wording "instant QSLs" hi hi. ;D

GL, 73, Rich, K3VAT


Title: RE: LOTW
Post by: NK7Z on November 22, 2012, 04:24:32 AM
Very good Dave,

So you might want to revise your QRZ.COM page and perhaps change that wording "instant QSLs" hi hi. ;D

GL, 73, Rich, K3VAT

That was put in when I could upload, and wait a few moments, then get a confirm back, not days, or weeks like now...  It has been changed...  Thanks for catching that.  I also added eQSL to the list.


Title: RE: LOTW
Post by: LA9XSA on November 22, 2012, 04:45:31 AM
About all I can think of is that the data sent over weather nets could be of use within a storm's circulation when it knocks out power to the region.
That's a good point, but trained weather observers can also provide something that radar and automated weather stations can't: Actual human observation to see if that hook on the radar image is actually a tornado, size of hail, damage reports, and readings from non-automated weather stations that happen to be in a particularly interesting part of the storm (like the eye for example).

I'm not saying shut down ARES (does ARES even need the League to function?).
Yes, even though it uses a bottom-up approach, they need the local coordination and sometimes they also need help from neighboring sections or HQ, whether that be coordination of mutual aid, resources, or legal and regulatory issues.

They pay lip service to serious DXers and contesters at best.
We shouldn't fall into the trap of thinking that our niche of amateur radio should be the only one. If there's a big cover story about DXing, do the ARDF enthusiasts grumble about that? If an SSB contest is featured, will PSK fanatics be cutting their keyboard cables in protest?

It's not like the ARRL stays on the same topic all the time either: They used to have a campaign about digital communications, then it was "when all else fails" (emcomm), and now it's "The DIY magic of amateur radio" (hacking and buidling stuff). Maybe the next promotion could appeal to radio orienteering or international friendships? The general public, politicians and media are understandably more responsive to information about emcomm than many other niches though, so don't expect that to go away.

Trust me, if there were any component of LoTW that had to do with giving PR to emcomm, the League would be on it like white on rice and they'd have their platinum-plated quadruply-redundant bulletproof system in place.
Of course they should, but no component of LotW should be part of emcomm anyway. If the system was something like the Red Cross "Safe and Well" system or another system handling emergency and disaster welfare communications, it would be especially important that it stayed up in an emergency.

LotW is not critical infrastructure, nor was it meant to be. It's just a neat way to confirm a QSO for awards. And I've got QSLs ticking in here. Faster and cheaper than paper (though I like paper too).


Title: RE: LOTW
Post by: AB8MA on November 22, 2012, 06:11:37 AM
In the past 10 hours 21 minutes on LoTW,

53,559 QSO records have been entered into the system.
906 User files have been processed.

In the last 10 minutes, there has been no activity.

I think what this means to me is that I'd best quit looking. :)





Title: RE: LOTW
Post by: W2IRT on November 22, 2012, 06:12:28 AM
For the most part, however, amateur radio itself isn't part of the critical infrastructure, outside of tornado alley and other sparsely-populated regions. There are a great many emergency organizations that have little to no use for hams, especially in urban areas. I don't have a problem with local groups coordinating with local agencies and providing service if requested by said agencies. I have a problem with the disproportionate share of the pie that emcomm gets today. Read QST today, then look through the archives and see what it was like 15, 25, 30, 40, 50 years ago or more. It was more of an advanced technical hobby back then, and required a certain dedication and intelligence level to get in the door. The League was focused on what I consider should be the core competencies of the hobby -- sound fundamental electronics knowledge, good HF operating practice, design, construction and component-level repair of gear and so on.

Now it's the opposite. Populist "Oh, anybody can do it" touchy-feely nonsense that caters to non-technical hams (a contradiction of terms if ever there was one), glorifies emcomm, field day and selling books/merchandise. I'm not saying DX/contesting should be their focus. I'm saying the League is focusing on the segment of the hobby who may not be quite so serious in the hobby. I have to wonder, how many shack-on-a-belters are still heavily active after the first couple of years, and what is their participation level in the hobby beyond chatting on the local repeater coming home from work every day.

To my way of thinking, DX and contesting themselves shouldn't be the focus of the League, but rather reaching out and bending over backwards to cater to those with serious stations or who wish to someday have them. LoTW is one such way. It provides a great service to a higher calibre of hams.


Title: RE: LOTW
Post by: WS3N on November 22, 2012, 09:56:29 AM
To my way of thinking

Exactly.



Title: RE: LOTW
Post by: N2RJ on November 22, 2012, 12:49:44 PM
To my way of thinking, DX and contesting themselves shouldn't be the focus of the League, but rather reaching out and bending over backwards to cater to those with serious stations or who wish to someday have them. LoTW is one such way. It provides a great service to a higher calibre of hams.

If the league were to adopt this strategy, its membership would quickly drop to zero, or close to it.

The league by its nature needs to be a "big tent." It needs to cater to everyone, both the beginner and the veteran. It needs to cater to everyone, from the guys on 2m repeaters to worked 'em all on every band, mode and power level.

As for LoTW, it provides a service to everyone it seems. DXCC for me was partially LoTW credits, and it was great not to have to go and send cash through the mail to get a QSL card to confirm that I worked a common entity like Germany or Italy. It also helps contesters (and others) who view QSLing as a chore. Honestly if it were just the elites in LoTW, it would fall flat on its face.

Now all of this said, there's really not much of a reason to leave the system let it languish. The CQWW organizers used cloud computing to process all of the logs for CQWW in 45 minutes. This includes, I believe, more matching of QSOs than there would be with LoTW.

The digital edition of QST is also pretty much done "halfway" versus CQ magazine who has an excellent digital product. I am not strictly talking about features, but stability. With CQ I can view the magazine on the web or my iPad with little trouble. This probably boils down to cost, given that the league has to do more with the dues money than just the magazine.

It's not even these recent things. The website redesign crashed for days and they even had to roll back.

I don't really know what's up with ARRL but I think technology wise they need to do things differently.


Title: RE: LOTW
Post by: W2IRT on November 23, 2012, 07:12:58 AM
I'm not so sure it would kill membership. You can be "big tent" without the complete dumbing down that they embarked on when emcomm became their BigDeal. Take a look at a QST from the 60s or 70s or 80s and the calibre of material and its presentation were aimed at a different "level" of ham than it is currently. At least the few I perused last week when this first came up.

I'm not saying become an engineering journal, but certainly higher than it is now. But therein lies the problem. They need to keep growing revenues, and J. Random Ham has become less savvy than those of a generation ago. I don't know the answer--hell, I wish I did--but I'm convinced that when they sold their soul to emcomm, everything changed.

Now with all that said, I do strongly support the League in their mission of lobbying and representing our interests before the ITU, etc. They do good work there. As to QST, reduce my subscription rate and let me have just the digital version of QST. I have ZERO interest in dead-tree magazines. I'd like my dues to be uses less for layout and production of a glossy magazine than into the core issues that need our help.

They absolutely need to ditch their IT personnel, I suspect. I think they need not just new tech, but new ideas and a new way of looking at their problems. They need to realize that they're just putting band-aids on an arterial gash.


Title: RE: LOTW
Post by: WF2S on November 23, 2012, 07:35:35 AM
I fully agree with K2QB. I have 40+ years of dxing and started in high school. QSLing has become expensive for dxers all over the world. The League came up with LoTW, and despite the occasional slowdown, I prefer it to spending $3 o rmore for a paper card that has to be mailed to ARRL for credit. If anyone feels that the LoTW program is so poor and lacking, fine, use an alternative !
Many  have had confusion setting up their account but staff, paid for by ARRL members, are available as well as a Power Point presentation to walk you through.
I was a complainer until I went into management, then I learned to not complain until I was prepared to offer a solution to the problem. Yes, it is not perfect, but I could not do better.

                                                 ;)


Title: RE: LOTW
Post by: SV1XV on November 23, 2012, 07:47:08 AM
Take a look at a QST from the 60s or 70s or 80s and the calibre of material and its presentation were aimed at a different "level" of ham than it is currently...  I'm not saying become an engineering journal, but certainly higher than it is now.
Back then some articles from QST and The Radio Amateurs Handbook were used as reading material in colleges. Some really good QST articles of the period got citations (were listed as references) in scientific papers and symposia presentations.





Title: RE: LOTW
Post by: W2IRT on November 23, 2012, 10:31:48 PM
Take a look at a QST from the 60s or 70s or 80s and the calibre of material and its presentation were aimed at a different "level" of ham than it is currently...  I'm not saying become an engineering journal, but certainly higher than it is now.
Back then some articles from QST and The Radio Amateurs Handbook were used as reading material in colleges. Some really good QST articles of the period got citations (were listed as references) in scientific papers and symposia presentations.

Exactly. Some of the material they're presenting today are, I swear, aimed at those who ride the short school bus and wear helmets walking to the playground. "This is how you cut a dipole. This is how to solder."  :-X


Title: RE: LOTW
Post by: AB8MA on November 24, 2012, 06:03:47 AM
I wonder if there is a way to query LoTW to see what the queue size is.


Title: RE: LOTW
Post by: N1UK on November 24, 2012, 06:30:32 AM
Quote
Exactly. Some of the material they're presenting today are, I swear, aimed at those who ride the short school bus and wear helmets walking to the playground. "This is how you cut a dipole. This is how to solder." 

If you get on 2m these days most of them don't even know how to do that...land of the might J-pole...hi hi .

The conversations go something like this...how are you doing...I am doing OK...how are you doing?...I am doing OK....and then they talk about food !!


Mark N1UK


Title: RE: LOTW
Post by: AB8MA on November 24, 2012, 06:41:18 AM
Quote
Exactly. Some of the material they're presenting today are, I swear, aimed at those who ride the short school bus and wear helmets walking to the playground. "This is how you cut a dipole. This is how to solder." 

If you get on 2m these days most of them don't even know how to do that...land of the might J-pole...hi hi .

The conversations go something like this...how are you doing...I am doing OK...how are you doing?...I am doing OK....and then they talk about food !!


Mark N1UK


At least they knew how to work SPLIT!! :)


Title: RE: LOTW
Post by: N1UK on November 24, 2012, 07:34:35 AM
Quote
At least they knew how to work SPLIT!! Smiley

Not all of them..


Title: RE: LOTW
Post by: WS3N on November 24, 2012, 08:14:38 AM
Quote
Exactly. Some of the material they're presenting today are, I swear, aimed at those who ride the short school bus and wear helmets walking to the playground. "This is how you cut a dipole. This is how to solder." 

If you get on 2m these days most of them don't even know how to do that...land of the might J-pole...hi hi .

The conversations go something like this...how are you doing...I am doing OK...how are you doing?...I am doing OK....and then they talk about food !!


Mark N1UK


I don't know why anyone would want to ragchew or be on a 2m repeater, but if that's what some choose then I'd say they can talk about anything they like. Who gets to decide what topics are worthy?


Title: RE: LOTW
Post by: AF3Y on November 24, 2012, 08:44:31 AM

I don't know why anyone would want to ragchew or be on a 2m repeater.


Lets see now, HOW can I say/answer this without............  Oh Hell, I cant......... I dont want to get yelled at again.

73, Gene AF3Y


Title: RE: LOTW
Post by: WS3N on November 24, 2012, 01:18:46 PM
Oh, come on, Gene. That's never stopped you before.  ;)

My point (poorly made) is that it's just an opinion, not TRUTH. I'm not interested, but so what? There is no wrong or right way to pursue the hobby, so enjoy.

I've often seen it stated, and there seems to be general agreement, that DXers are at the top of the ham-operator heap. Of course, it was stated by DXers and agreed with by DXers. It's like right-wing talk radio and the left-wing equivalent. Both sides are preaching to the choir, and believe they are right because everybody else they ever hear agrees with them.


Title: RE: LOTW
Post by: KA5N on November 25, 2012, 05:44:58 AM
LOTW     

At the rate LOTW is processing VS uploads surely the system is more than 6 days
behind.  From the index of input vs output about 300,000 uploads vs  3000 processed
(yesterday to today) is a factor of 100X and will never get caught up unless the
numbers begin to tend closer together.   About a week ago I made a contact and I
had barely entered it before the confirmation had appeared.

Allen  KA5N


Title: RE: LOTW
Post by: AF3Y on November 25, 2012, 06:35:51 AM
LOTW     

At the rate LOTW is processing VS uploads surely the system is more than 6 days
behind.  From the index of input vs output about 300,000 uploads vs  3000 processed
(yesterday to today) is a factor of 100X and will never get caught up unless the
numbers begin to tend closer together.   About a week ago I made a contact and I
had barely entered it before the confirmation had appeared.

Allen  KA5N

Allen...... FYI,  I uploaded a small file (25 - 30 QSOs) on the 20th.  It did not show up last night, but it is there today.  Looks like 5 to 6 days is the norm now. BTW, I remember uploading files and having them active on LOTW in a few hours myself (In the good old days).

73, Gene AF3Y


Title: RE: LOTW
Post by: W5RDW on November 25, 2012, 07:47:43 AM
Just noticed this morning my QSO's are current now, being 5 days behind up until this morning.

Now if I could get a few more QSL's coming in, I would be happy. I have all the PT0S Q's I worked confirmed as of yesterday.


Title: RE: LOTW
Post by: AB8MA on November 25, 2012, 03:42:34 PM
LOTW     

At the rate LOTW is processing VS uploads surely the system is more than 6 days
behind.  From the index of input vs output about 300,000 uploads vs  3000 processed
(yesterday to today) is a factor of 100X and will never get caught up unless the
numbers begin to tend closer together.   About a week ago I made a contact and I
had barely entered it before the confirmation had appeared.

Allen  KA5N

I can see how you can determine the number of user files processed, but how do you determine the number of uploads?


Title: RE: LOTW
Post by: W5JON on November 25, 2012, 05:39:48 PM
It will be interesting to see how all the Uploads from the CQWW CW Contest this weekend adds to the LoTW backlog, and processing time.

73 and DX,

John W5JON


Title: RE: LOTW
Post by: KA5N on November 25, 2012, 05:45:59 PM
Look at logbook status and it tells the number of QSO Records entered.

and the QSL's that resulted and

number of user files that have been processed.

I think one can get a good idea from those numbers and by watching how they
change from day to day.  Since I don't know anything about the code used my
estimate could be off a large amount.  But there is obviously  more data going in
than comming out.
Allen KA5N


Title: RE: LOTW
Post by: AB8MA on November 26, 2012, 07:03:17 AM
LoTW began operation in 2003. Not sure what month.

01:08 11/11/2011
375,158,555 QSO records have been entered into the system.
46,044,077 QSL records have resulted.
45,223 Users are registered in the system
67,933 Certificates are active
2,576,705 User files have been processed

14:55 11/26/2012
457,664,730 QSO records have been entered into the system.
62,405,682 QSL records have resulted.
53,759 Users are registered in the system
80,790 Certificates are active
3,708,023 User files have been processed

I know that is only two points of data at the end of the timeline, but I would say that maybe it's time for an upgrade?

on 11/15/2011 the following was news on the ARRL site:

Quote
Currently, the processing delay is at 45 hours, down from 60 hours last week.


Title: RE: LOTW
Post by: N2RJ on November 26, 2012, 09:27:35 AM
I see 9Y4D is uploading to LoTW now. This is great for anyone who wanted to confirm 9Y but did not want to deal with paper cards, lost mail etc.

I have also uploaded all of my 9Y4RAJ logs to LoTW from past operation.


Title: RE: LOTW
Post by: AF3Y on November 26, 2012, 04:41:46 PM
Ha Ha....... Just tried LOTW..... ARRL.org is down, and obviously so is LOTW.  Why am I not surprised?

73, Gene AF3Y


Title: RE: LOTW
Post by: KF6ABU on November 26, 2012, 04:47:52 PM
Ha Ha....... Just tried LOTW..... ARRL.org is down, and obviously so is LOTW.  Why am I not surprised?

73, Gene AF3Y

LoTW app, and website are working fine here, currently.


Title: RE: LOTW
Post by: NU4B on November 27, 2012, 04:09:14 AM
I uploaded on 11/22 after I worked PT0S and then again after the CQWWCW contest on the 26th. Neither is showing. I'm not so surprised about the 26th upload - but the 22nd upload isn't showing and I noticed PT0S has uploaded on the 26th.

Does anybody know if there is a way to check if I'm still in the queue? Or should I just send in the missing upload again?

I haven't had any problems in the past.


Title: RE: LOTW
Post by: NI0C on November 27, 2012, 04:35:13 AM
NU4B wrote:
Quote
I uploaded on 11/22 after I worked PT0S and then again after the CQWWCW contest on the 26th. Neither is showing. I'm not so surprised about the 26th upload - but the 22nd upload isn't showing and I noticed PT0S has uploaded on the 26th.

Does anybody know if there is a way to check if I'm still in the queue? Or should I just send in the missing upload again?

Larry, I'm still waiting for a one QSO upload made on 11/22 to appear in my LoTW QSO list.  The last QSO in my list was uploaded on 11/20 and processed on 11/24.

I think this may have been mentioned before in this discussion, but a good way to see which uploads have been accepted by the system is to click "Your Account" on the main page, then click on "your Activity" at the left.  However, there doesn't seem to be a way for us to check files that have been uploaded, but not yet processed. That's where the waiting seems to take place. 

So far, all of my uploaded files seem to have gotten through, and a couple of times that I repeated an upload, I was embarrassed to find a warning of a "duplicate QSO record" under "Your Activity."

The good news about the system is, that once your QSO's are processed and you do get QSL's, the applications are processed as quickly as ever.  I made a small application this past Friday (when I'm sure ARRL staff were off work), and it was processed already yesterday morning.

73,
Chuck  NI0C   


Title: RE: LOTW
Post by: AB8MA on November 27, 2012, 04:57:07 AM
I uploaded on 11/22 after I worked PT0S and then again after the CQWWCW contest on the 26th. Neither is showing. I'm not so surprised about the 26th upload - but the 22nd upload isn't showing and I noticed PT0S has uploaded on the 26th.

Does anybody know if there is a way to check if I'm still in the queue? Or should I just send in the missing upload again?

I haven't had any problems in the past.

The thing is, all you know is that one of the PT0S uploads was actually processed on the 26th. Who knows how long that file was sitting in the queue. The wording on the LoTW site is very misleading.

I would give it a few more days before sending anything you already sent.


Title: RE: LOTW
Post by: KC8LTL on November 27, 2012, 05:10:47 AM
All I know is that the log I uploaded on 9 Nov still hasn't been processed.  This sounds like there are major problems they are not acknowledging.


Title: RE: LOTW
Post by: NU4B on November 27, 2012, 05:13:48 AM
Hmmmm, interesting guys.
The note on LoTW left on the 26th says:

"Nov 26, 2012: LoTW Log Processing is About 4 Days Behind -- The system is processing log data a little slower than usual, but is working.  Now that CQWW CW is over we expect a large number of logs to flow through the system.  Thanks for your patience and for using LoTW."

I guess we wait a couple more days, but I would think our 11/22 uploads would have been processed by now if they were 4 days behind. Which is not a problem, I was just worried my upload file got lost. And since a new one was involved, well you know how it goes... I'm all about patience... until its my new one.  ;D


Title: RE: LOTW
Post by: KA5N on November 27, 2012, 07:23:29 AM
I also thought that a couple of my uploads had been lost, but they just chugged out
this morning with a QSL for PT0S (input the 20th) after 7 days and a QSO for AH0I
(input the 19th) after 8 days.   This is surely slower than normal but there is no indication that uploads are being lost. 

Of course the CLUB LOG online log is so much faster that everything else seems glacial.
It is sooo nice to see one's call in a log the same day you make the contact (often
sooner) so you don't have to make a bunch of insurance dupes.
Be thankful for what you get and "don't worry, be happy"
Allen KA5N


Title: RE: LOTW
Post by: NK7Z on November 27, 2012, 07:51:38 AM
Found on the LOTW mail list:

===CUT===
The Delta Division Director's update this morning addressed LOTW.  Here is part of what Dave, K5UZ had to say.  Hopefully you are getting the same news from your representatives.

"At last week's Administration and Finance Committee
meeting, staff was authorized to replace the old magnetic drives with
faster digital units.  The change-over and upgrade will take a few
weeks as staff wants to 1.)keep the old units for back-up data and 2.)
Allow for systems testing and comparison. This process should take
approximately eight weeks; therefore I now pose the question:  Isn't
it time for a software upgrade to go along with the new LOTW hardware?
Please let your Director know what you think on this issue as it
certainly is an opportune time to get this fine membership service
adjusted and running the way it should!" 
===CUT===


Title: RE: LOTW
Post by: N2RJ on November 27, 2012, 08:00:20 AM
So they replaced the old hard drives with SSDs.

I wouldn't be surprised if they still have problems. It doesn't seem like a hardware issue.


Title: RE: LOTW
Post by: N9KX on November 27, 2012, 09:33:54 AM
Who knows how long that file was sitting in the queue. The wording on the LoTW site is very misleading.

QSL.  I had a confirmation processed on 11/24, but the uploads on both ends were probably 7 days ago (or more).  The danger with their way of wording it is that people will re-upload a lot of data that is still in the queue thinking it did not take.  

I am still waiting for uploads to be processed that happened a lot more than just 4 days ago...


Title: RE: LOTW
Post by: W2IRT on November 27, 2012, 10:58:48 AM
Replacing spinning iron with SSDs may be a great idea for short-term performance improvements, but one of the big downsides of SSDs are that they have a finite write capability. Over the long term I have to wonder if they're prepared for more frequent drive failures. This definitely strikes me as a chewing gum and bailing wire approach to database/hardware administration.


Title: RE: LOTW
Post by: N2RJ on November 27, 2012, 11:26:05 AM
They could solve all of this by moving the whole thing to the cloud. No hardware to worry about and scale up/down as needed. For contest time, for example, they could scale all the way up. During low periods they scale down. Saves $$.



Title: RE: LOTW
Post by: N9KX on November 27, 2012, 11:34:13 AM
I just did a "find call" search to see what LoTW says is the last time I uploaded.  Here is the result:

Last upload for K9AIM: 2012-11-24 08:47:54Z

That is the last time I had a confirmation of something I had uploaded well before Nov. 24...
I am sure it is off just as much when it comes to suggesting when PT0S last uploaded...

which makes me a little less apprehensive about seeing confirmation for my Nov 20th PT0S contact sometime in the future...


Title: RE: LOTW
Post by: K4BNC on November 27, 2012, 02:09:36 PM
The LOTW page says 4 days for processing.  That matches just what I experienced.  A log that I uploaded on 11/22 was processed on 11/26.  It did contain PT0S contacts from 11/21 that were confirmed.


Title: RE: LOTW
Post by: N1UK on November 27, 2012, 03:16:07 PM
Quote
The LOTW page says 4 days for processing.  That matches just what I experienced.  A log that I uploaded on 11/22 was processed on 11/26


You are lucky, mine was uploaded on 21 Nov 2012 and I am still waiting!

Mark N1UK


Title: RE: LOTW
Post by: AB8MA on November 27, 2012, 04:44:42 PM
A file I uploaded on the 24'th at 9:00 AM just got processed today at 7:00 PM.


Title: RE: LOTW
Post by: NU4B on November 28, 2012, 11:10:09 AM
ARRL update"

http://www.arrl.org/news/arrl-s-logbook-of-the-world-bug-fix-is-on-the-way

Apparently some logs in queue were lost?


Title: RE: LOTW
Post by: AB8MA on November 28, 2012, 12:11:05 PM
That sounds like a description of the fallout, not the original issue(s).


Title: RE: LOTW
Post by: N9KX on November 28, 2012, 12:24:51 PM
Quote from: David Sumner K1ZZ Chief Executive Officer ARRL November 28, 2012
<snip>
When LoTW was designed more than a decade ago -- long before the present IT staff was here -- an assumption was made as to how many logs could possibly be in the queue at a given time. The assumption was based on users uploading their most recent QSOs perhaps once a week or once a month. The environment in which LoTW now operates is quite different from that assumption, in that many users now upload logs with small numbers of QSOs in them, almost in real time. This creates a much larger number of separate logs
When a log is uploaded, it is identified by a file name that is assigned by the user. Because there is no way to avoid duplication of file names that are assigned in this fashion, the LoTW system renames each file. Because of the unusual processing delay -- combined with the dramatic increase in the number of submitted logs -- the system began to run out of unique identifiers for the log files. This resulted in a file sometimes being renamed with an identifier that had already been assigned to a log that was still in the queue, causing the earlier log to be overwritten.

Once the problem was identified, designing a fix was relatively easy. It should be in place by 2359 UTC November 28. Because the number of overwritten logs is relatively small, we have decided to keep the system available for use, even though this may result in a few more logs being lost until the fix is in place. <snip>

kudos to K1ZZ for offering some transparency regarding the issue.
I am surprised the system wasn't proactively updated by someone who knew the ins and outs of how the system worked every year or so to keep pace with usage and technology...  am i missing something  ???


Title: RE: LOTW
Post by: NK7Z on November 28, 2012, 12:46:28 PM

kudos to K1ZZ for offering some transparency regarding the issue.


Yes, this is a step in the right direction...


Title: RE: LOTW
Post by: W2IRT on November 28, 2012, 06:45:44 PM
My CQWW-CW log from Sunday evening just got processed and a few matches have trickled in so I suspect it's catching up pretty quickly now.


Title: RE: LOTW
Post by: KE8G on November 29, 2012, 02:21:14 AM
My CQWW-CW log from Sunday evening just got processed and a few matches have trickled in so I suspect it's catching up pretty quickly now.

So has my CQWW/CW log from Sunday evening... at this point ALL of my logs are now processed. 


Title: RE: LOTW
Post by: WS4T on November 29, 2012, 03:37:46 AM
It's a big relief to hear they have figured out the "lost log" problem.

I'm a big fan of LotW and I keep track of what logs I have uploaded and reload them if necessary. But there's another class of user (mostly rare DX!) that only uploads logs to do the rest of us a favor. That type of user is not checking every day to see if his log was processed, so if it's lost, there's a chance it won't get uploaded again.


Title: RE: LOTW
Post by: KB2FCV on November 29, 2012, 10:39:54 AM
I think my logs were amongst the lost. I submitted some in the middle of last week that didn't make it, so I re-submitted my logs last night after 2359.. so I guess maybe check next week sometime.


Title: RE: LOTW
Post by: NU1O on November 29, 2012, 01:33:54 PM
Just for the hell of it I think I'll upload my 20,000 contacts in one big file.  I must have uploads that got lost along the way and he did say it was the small uploads which were the pain in the ass, right?

73,

Chris/NU1O


Title: RE: LOTW
Post by: AB8MA on November 29, 2012, 02:14:54 PM
Just for the hell of it I think I'll upload my 20,000 contacts in one big file.  I must have uploads that got lost along the way and he did say it was the small uploads which were the pain in the ass, right?

73,

Chris/NU1O

I just e-mailed each of my 20,000 contacts requesting that they re-upload data from the dates I worked them. Just for insurance, you see?


Title: RE: LOTW
Post by: NU4B on November 30, 2012, 09:17:06 AM
Well, still nothing. No PT0S upload processed, no CQWWCW upload processed. I guess I was also one of the very few who had their uploads overwritten even after the 25th. Instead of a Power Ball winner, it looks like I'm a LoTW wiener!


Title: RE: LOTW
Post by: NK7Z on November 30, 2012, 11:42:53 AM
Hi,
Whats odd is I did much the same as you...  I uploaded a large portion of my log after the affected dates, and have not seen a return on them yet...  I also have not seen a single confirmation in about a week...  This is not normal...  I normally get one or two a week easily...  I am tempted to upload my entire log, (20K contacts), and see if that flushes things out...  Not getting any confirmations for a week or longer is not normal...
73's
Dave


Title: RE: LOTW
Post by: WW3QB on November 30, 2012, 11:45:25 AM
I am tempted to upload my entire log, (20K contacts), and see if that flushes things out...

Please do not. That will just make the backlog worse for the rest of us.


Title: RE: LOTW
Post by: NK7Z on November 30, 2012, 11:55:05 AM
Tempted is the operative word here... 

73's
Dave


Title: RE: LOTW
Post by: N9KX on December 01, 2012, 09:29:09 AM
Well, still nothing. No PT0S upload processed, no CQWWCW upload processed. I guess I was also one of the very few who had their uploads overwritten even after the 25th. Instead of a Power Ball winner, it looks like I'm a LoTW wiener!

I am not seeing anything new in the last week either ...  ???  The LoTW page has an update today saying they are 4 days behind now (not 3)...

I wouldn't be surprised if a lot of us uploaded our PT0S contact since we knew they were uploading to LoTW and that created a large number of small files -- the kind evidently that are especially prone to being lost by the system  :-\


Title: RE: LOTW
Post by: AD9DX on December 01, 2012, 08:27:50 PM
LoTW is now off line for maintenance  ???


Title: RE: LOTW
Post by: AD9DX on December 01, 2012, 08:28:32 PM
ANd now seconds later back up  ???


Title: RE: LOTW
Post by: N9KX on December 02, 2012, 09:07:59 AM
If I go to the https://p1k.arrl.org/lotwuser/qsostats (https://p1k.arrl.org/lotwuser/qsostats) (LoTW statistics page) to find the date of my last uploaded QSO, it lists my newest QSO upload as having occurred 2012-11-19 23:48:00  (while my most recent confirmation was on 11/24).

Anyone able to verify QSO's more recent than 11/19 are being processed???



Title: RE: LOTW
Post by: AB8MA on December 02, 2012, 09:42:38 AM
PT0S Date/Time       2012-11-22 18:52:00

QSL       2012-12-02 11:33:31


Title: RE: LOTW
Post by: NN4RH on December 02, 2012, 09:56:34 AM
If I go to the https://p1k.arrl.org/lotwuser/qsostats (https://p1k.arrl.org/lotwuser/qsostats) (LoTW statistics page) to find the date of my last uploaded QSO, it lists my newest QSO upload as having occurred 2012-11-19 23:48:00  (while my most recent confirmation was on 11/24).

Anyone able to verify QSO's more recent than 11/19 are being processed???



My CQWWDX CW log was uploaded on the 26th and finally was processed on the 30th, and a few QSLs have occurred.


Title: RE: LOTW
Post by: W1VT on December 02, 2012, 01:07:45 PM
Uploaded half the log during the contest, which has been processed, but am still waiting on the part uploaded at 1AM the day after the contest.

TT8TT just came through for number 244!


Title: RE: LOTW
Post by: N9KX on December 03, 2012, 10:51:32 AM
My two TT8TT QSO's were confirmed yesterday...

I had mistakenly read "the problem is fixed" statement to apply to the backlog and not just the lost upload issue, but evidently the system is now 5 days behind... Hopefully a fix for that is in the works (and budget) as well.  Here are two recent snapshots of LoTW status:

Logbook Status 12/2/2012 16:58z
459,279,601   QSO records have been entered into the system.
62,614,920   QSL records have resulted.
53,980   Users are registered in the system
81,079   Certificates are active
3,723,991   User files have been processed


Logbook Status 12/3/2012 18:44z
459,523,126   QSO records have been entered into the system.
62,682,961   QSL records have resulted.
54,017   Users are registered in the system
81,125   Certificates are active
3,725,404   User files have been processed



Title: RE: LOTW
Post by: NU4B on December 03, 2012, 11:41:30 AM
I still haven't had any of my uploads show up.  >:(


Title: RE: LOTW
Post by: NI0C on December 03, 2012, 02:26:07 PM
Hmm...
"The system is catching up."  Not to quibble, but over the past weekend the backlog was four days-- now it's five.  Where is the evidence that it's catching up?

73,
Chuck  NI0C 


Title: RE: LOTW
Post by: N9KX on December 03, 2012, 03:15:06 PM
Hmm...
"The system is catching up."  Not to quibble, but over the past weekend the backlog was four days-- now it's five.  Where is the evidence that it's catching up?

73,
Chuck  NI0C 

Thank you!!! finally some resonant feedback on where we and LoTW are at!  8) 



Title: RE: LOTW
Post by: AB8MA on December 03, 2012, 03:48:00 PM
Hmm...
"The system is catching up."  Not to quibble, but over the past weekend the backlog was four days-- now it's five.  Where is the evidence that it's catching up?

73,
Chuck  NI0C 

Easy. It caught up those 4 days at a cost of 5 days. It is a totally different backlog now. :)


Title: RE: LOTW
Post by: N9KX on December 03, 2012, 06:23:35 PM
They could solve all of this by moving the whole thing to the cloud. No hardware to worry about and scale up/down as needed. For contest time, for example, they could scale all the way up. During low periods they scale down. Saves $$.

and if there is a disinclination for transparency regarding the problems with the present LoTW system -- all the more reason to Cloud up  ;)


Title: RE: LOTW
Post by: N9KX on December 03, 2012, 06:53:06 PM
Well, still nothing. No PT0S upload processed, no CQWWCW upload processed. I guess I was also one of the very few who had their uploads overwritten ...

I just had the QSO's I uploaded on the 28th get processed.  And my PT0S QSO from the 20th is confirmed.  LoTW confirmed it today (12-4-2012 at 0205z).  So for me it is 6 days behind, which may cheer you up if you are waiting less than 6 days.  (Looks like my original upload was lost, but the one I re-did on the 28th has now been processed)


Title: RE: LOTW
Post by: NU4B on December 03, 2012, 07:03:58 PM
Well, still nothing. No PT0S upload processed, no CQWWCW upload processed. I guess I was also one of the very few who had their uploads overwritten ...

I just had the QSO's I uploaded on the 28th get processed.  And my PT0S QSO from the 20th is confirmed.  LoTW confirmed it today (12-4-2012 at 0205z).  So for me it is 6 days behind, which may cheer you up if you are waiting less than 6 days.  (Looks like my original upload was lost, but the one I re-did on the 28th has now been processed)

That's good news!
I did a small upload near the end of last week so maybe that one will show up. That's the one that has PT0S. I haven't re-uploaded  ??? my CQWWCW logs yet.


Title: RE: LOTW
Post by: AF3Y on December 04, 2012, 03:51:27 AM
I uploaded a small file on the 29th.  Nothing yet as of 7 AM local time this morning. ::)

73, Gene AF3Y


Title: RE: LOTW
Post by: AC4RD on December 04, 2012, 03:56:14 AM
I uploaded a small file on the 29th.  Nothing yet as of 7 AM local time this morning. ::)

I uploaded a file on the afternoon of Tues 27 Nov 2012, and it still hasn't shown up--about 6.5 days later.


Title: RE: LOTW
Post by: NU1O on December 04, 2012, 08:44:58 AM
Found on the LOTW mail list:

===CUT===
The Delta Division Director's update this morning addressed LOTW.  Here is part of what Dave, K5UZ had to say.  Hopefully you are getting the same news from your representatives.

"At last week's Administration and Finance Committee
meeting, staff was authorized to replace the old magnetic drives with
faster digital units.  The change-over and upgrade will take a few
weeks as staff wants to 1.)keep the old units for back-up data and 2.)
Allow for systems testing and comparison. This process should take
approximately eight weeks; therefore I now pose the question:  Isn't
it time for a software upgrade to go along with the new LOTW hardware?
Please let your Director know what you think on this issue as it
certainly is an opportune time to get this fine membership service
adjusted and running the way it should!" 
===CUT===

I don't even know who my director is and that's partly my fault, but the ARRL has a massive database and communicating with constituents should not be a big problem.  I get an ARRL letter every week via Email.  Why can't my director get the Email addresses for his/her constituents from the ARRL and send emails when appropriate?  Should we really be chasing them, or is it their job to inform us members? I don't believe I have ever received an Email from my director.  That's why I was so surprised to see an Email from a director which admits they need to make some investments and spend some money.  Where are the Emails from the other directors? 

When my brother ran for State Representative in 2008 I typed in the zip codes on QRZ for the communities his district represented and we mailed a letter to all those with amateur licenses which stated his brother (me) was an amateur radio operator and my brother was familiar with the problems facing our hobby.  Nobody with an R next to their name won any elections up and down the East Coast that year so there is no telling if it helped or not.

I think what happens in most charitable organizations is the CEO runs the day-to-day operations and all the other non-paid positions just fall into line and don't rock the boat. What's needed in any non-profit organization are people who are not afraid to hold an opinion different from the manager's position and who are not afraid to let their differing opinions known.

Two days before Sweepstakes around November 15th I was about 5 or 6 days behind in my LoTW confirmations and I didn't think there was a snowball's chance in hell they'd clear out the Sweepstakes confirmations before they got hit with the CQWW uploads.  To my surprise they did get out most of the SS confirmations but I think they had to play some games to do so.  Maybe the SS files got moved to the front and the common files were delayed or lost.

The delay has actually got longer than before SS.  It has now been 9 days since CQWW ended and I have yet to receive a confirmation for any QSO dated on the 26th or later.

What we really need is a Whistleblower (and I normally don't like them but this is not national defense we're talking about) to tell us what is actually going on.

73,

Chris/NU1O


Title: RE: LOTW
Post by: N2RJ on December 04, 2012, 11:39:51 AM
Chris, most ARRL elected officials run unopposed. You're not exactly going to get the cream of the crop there.

We recently elected a new director here, who actually had to campaign and work for his seat. He's actually a good friend of mine - Mike, N2YBB. From what I know of him I think he'll do a good job. He's already reaching out.

In Hudson division we already were getting emails from our division reps though, so they communicate with us.

But I think that some of them in other divisions get elected in an uncontested race then sit and do nothing.


Title: RE: LOTW
Post by: NU1O on December 04, 2012, 02:45:01 PM
Chris, most ARRL elected officials run unopposed. You're not exactly going to get the cream of the crop there.

We recently elected a new director here, who actually had to campaign and work for his seat. He's actually a good friend of mine - Mike, N2YBB. From what I know of him I think he'll do a good job. He's already reaching out.

In Hudson division we already were getting emails from our division reps though, so they communicate with us.

But I think that some of them in other divisions get elected in an uncontested race then sit and do nothing.

Ryan, other than seeing K1ZZ's name on every editorial for as long as i can recall, I don't know beans about the politics of the ARRL or how it it governed.

I do know it is generally not a good thing if people run unopposed or hold a job for a very long period of time. 

If your friend, Mike, had to fight for his position I have a feeling he's liable to take it more seriously than somebody who had no opposition. 

I hope he does a good job for you guys and maybe he will want to play a bigger role someday.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AD9DX on December 04, 2012, 04:11:26 PM
I have had some of my first CQWW QSL's show up.  I got 4 today.  I have not had my upload I put in on the 29th show up yet.  HMM


Title: RE: LOTW
Post by: AF3Y on December 04, 2012, 05:23:57 PM
All of my ZK2C QSOs just showed up, but my upload of the 29th is not there yet.

73, Gene AF3Y


Title: RE: LOTW
Post by: N6ORB on December 04, 2012, 09:18:44 PM
I just had an upload (re-upload) sent Nov 28 processed this evening. Elapsed time was six days plus a few hours. This was the second lost file that I had to re-upload.

Dave, N6ORB


Title: RE: LOTW
Post by: NU1O on December 05, 2012, 12:49:55 AM
Rather than going forward in time my confirmations are actually going backwards in time.  I really think many of those smaller files were held back so the SS backlog could be taken care of before they got swamped with CQWW uploads.

Gene, I also received the QSL for ZK2C but that QSO was in February of this year.

Has anybody received a LoTW confirmation for a QSO after November 25th?

73,

Chris/NU1O


Title: RE: LOTW
Post by: N9KX on December 05, 2012, 01:25:44 AM

Has anybody received a LoTW confirmation for a QSO after November 25th?

73,

Chris/NU1O


I have one uploaded QSO that took place on the 26th that has been processed by LoTW (no QSL yet though):
Station
Call Sign       K9AIM
DXCC       UNITED STATES OF AMERICA
ITU Zone       4
Grid       EM69tx
State       IN
County       BOONE
Worked Station
Worked       ZF2AH
Date/Time       2012-11-26 16:40:00
Mode       CW (CW)
Band       12M
Frequency       24.897
Record ID 459597872 Received: 2012-12-04 02:05:38


Title: RE: LOTW
Post by: NU4B on December 05, 2012, 02:14:33 AM
I, too, received my ZK2C. As far as any of my recent uploads - NADA!


Title: RE: LOTW
Post by: K5JZ on December 05, 2012, 04:15:29 AM
Found on the LOTW mail list:

===CUT===
The Delta Division Director's update this morning addressed LOTW.  Here is part of what Dave, K5UZ had to say.  Hopefully you are getting the same news from your representatives.

"At last week's Administration and Finance Committee
meeting, staff was authorized to replace the old magnetic drives with
faster digital units.  The change-over and upgrade will take a few
weeks as staff wants to 1.)keep the old units for back-up data and 2.)
Allow for systems testing and comparison. This process should take
approximately eight weeks; therefore I now pose the question:  Isn't
it time for a software upgrade to go along with the new LOTW hardware?
Please let your Director know what you think on this issue as it
certainly is an opportune time to get this fine membership service
adjusted and running the way it should!" 
===CUT===

I don't even know who my director is and that's partly my fault, but the ARRL has a massive database and communicating with constituents should not be a big problem.  I get an ARRL letter every week via Email.  Why can't my director get the Email addresses for his/her constituents from the ARRL and send emails when appropriate?  Should we really be chasing them, or is it their job to inform us members? I don't believe I have ever received an Email from my director.  That's why I was so surprised to see an Email from a director which admits they need to make some investments and spend some money.  Where are the Emails from the other directors? 

When my brother ran for State Representative in 2008 I typed in the zip codes on QRZ for the communities his district represented and we mailed a letter to all those with amateur licenses which stated his brother (me) was an amateur radio operator and my brother was familiar with the problems facing our hobby.  Nobody with an R next to their name won any elections up and down the East Coast that year so there is no telling if it helped or not.

I think what happens in most charitable organizations is the CEO runs the day-to-day operations and all the other non-paid positions just fall into line and don't rock the boat. What's needed in any non-profit organization are people who are not afraid to hold an opinion different from the manager's position and who are not afraid to let their differing opinions known.

Two days before Sweepstakes around November 15th I was about 5 or 6 days behind in my LoTW confirmations and I didn't think there was a snowball's chance in hell they'd clear out the Sweepstakes confirmations before they got hit with the CQWW uploads.  To my surprise they did get out most of the SS confirmations but I think they had to play some games to do so.  Maybe the SS files got moved to the front and the common files were delayed or lost.

The delay has actually got longer than before SS.  It has now been 9 days since CQWW ended and I have yet to receive a confirmation for any QSO dated on the 26th or later.

What we really need is a Whistleblower (and I normally don't like them but this is not national defense we're talking about) to tell us what is actually going on.

73,

Chris/NU1O

Dave Norris K5UZ is a fine gentleman and my club, the Magnolia DX Association helped to elect Dave as Director of the Delta Division... because we knew that Dave would stay engaged with the DX community as he is one of us. With K5UR Rick as Executive VP, the DX community will be in good hands for some time to come. I appreciate your kind words for Dave and I will forward your comments to him.

73,
George K5JZ


Title: RE: LOTW
Post by: W1VT on December 05, 2012, 04:44:25 AM
Quote from: NU1Olink=topic=86250.msg637878#msg637878 date=1354697395

Has anybody received a LoTW confirmation for a QSO after November 25th?

Chris/NU1O

5X1NH  11/27  Processed 12/3 18Z


Title: RE: LOTW
Post by: NK7Z on December 05, 2012, 05:56:04 AM
Quote from: NU1Olink=topic=86250.msg637878#msg637878 date=1354697395

Has anybody received a LoTW confirmation for a QSO after November 25th?


My last confirmed QSO is on the 21st, of Nov...  Odd, as I have a few in the stack, accepted by LoTW.  I have been able to upload, and get confirms of the upload with the normal 3 to 4 day delay.  However, as you noted, no QSO confirmations though...

73's


Title: RE: LOTW
Post by: NU4B on December 05, 2012, 07:30:58 AM
OMG! FINALLY!

My little 4 QSO file got processed.

And I had to wait a half a month to get PY0/S confirmed! This is unheard of - 20 years ago!  ;D

(I am afraid to resend my CQWW log, I might break something.  ;D)


Title: RE: LOTW
Post by: NU1O on December 05, 2012, 04:31:30 PM
Today is the 10th day since my latest LoTW QSL on 11/25.  I'm still getting some QSLs from SS and CQWW, plus some things from 2011.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AB8MA on December 05, 2012, 05:45:19 PM
Lately, my "Most Recent QSLs" is showing old qsl's I have from years ago, and my QSL records count has been a constant.


Title: RE: LOTW
Post by: K4JK on December 05, 2012, 05:48:46 PM
Lately, my "Most Recent QSLs" is showing old qsl's I have from years ago, and my QSL records count has been a constant.
That most likely means people you have worked and confirmed are uploading duplicate copies of their logs.


Title: RE: LOTW
Post by: NK7Z on December 05, 2012, 05:49:22 PM
Lately, my "Most Recent QSLs" is showing old qsl's I have from years ago, and my QSL records count has been a constant.

That's odd...  I have been getting a mix of old QSL's as well...  Some 2 years old, and one from 2006...  I just figured that the persons uploaded finally...  I have not been watching my total counts though...

73's


Title: RE: LOTW
Post by: AF3Y on December 06, 2012, 06:28:27 AM
Well, I knew I would lose my temper eventually.  My upload of the 29th is STILL not processed.  ARRL, IF you are reading this (which I doubt), either FIX THE G.D. THING OR TAKE IT OFF LINE AND JUST START OVER. Kinda like a cell phone company using two tin cans and a string/

Gene AF3Y


Title: RE: LOTW
Post by: N4OGW on December 06, 2012, 07:24:00 AM
One of the problems is that very little status is given on LOTW about just what the system is doing other than a generic "x days behind" news. A display showing the number of unprocessed files would be great.

Uploaded files should also be shown in your "Log Book Activity" as soon you upload them. Right now no status on a file is given until after it is processed.

Tor
N4OGW


Title: RE: LOTW
Post by: NU4B on December 06, 2012, 09:26:11 AM
Well, I knew I would lose my temper eventually.  My upload of the 29th is STILL not processed.  ARRL, IF you are reading this (which I doubt), either FIX THE G.D. THING OR TAKE IT OFF LINE AND JUST START OVER. Kinda like a cell phone company using two tin cans and a string/

Gene AF3Y

From the last (Dec. 3rd) update on the LoTW web site:

"Thanks for your patience."    ;D ;D ;D ;D


Title: RE: LOTW
Post by: N9KX on December 06, 2012, 09:33:27 AM
I just had the QSO's I uploaded on the 28th get processed.  And my PT0S QSO from the 20th is confirmed.  LoTW confirmed it today (12-4-2012 at 0205z).  So for me it is 6 days behind, which may cheer you up if you are waiting less than 6 days.  (Looks like my original upload was lost, but the one I re-did on the 28th has now been processed)

upon further review, I am fairly sure I was a day off and that my QSO's that the system processed on the 4th were uploaded on the 27th not the 28th which means the system was 7 days behind (at least) 2.5 days ago...

I keep checking to see if the LoTW status gets updated to reflect it being 7 to 10 days behind, but so far it still says 5...


Title: RE: LOTW
Post by: N9KX on December 06, 2012, 09:43:04 AM
Kinda like a cell phone company using two tin cans and a string/

[sarcasm mode:] cheer up Gene, they have already put in process a plan to upgrade the string  ;D



Title: RE: LOTW
Post by: AF3Y on December 06, 2012, 01:12:10 PM
Kinda like a cell phone company using two tin cans and a string/

[sarcasm mode:] cheer up Gene, they have already put in process a plan to upgrade the string  ;D



I KNOW, I KNOW..... (we used to use candle wax :D)

Gene AF3Y


Title: RE: LOTW
Post by: N9KX on December 06, 2012, 01:59:17 PM
I keep checking to see if the LoTW status gets updated to reflect it being 7 to 10 days behind, but so far it still says 5...

sought LoTW status update now posted:

Quote from: LoTW home page
Dec 6, 2012: LoTW Log Processing is About 7 Days Behind -- For more information on what has been happening with LoTW lately, please see this news item.  NEW... The bug that was causing the problem has been fixed.  But there is still a large queue of logs steadily being processed.  Thanks for your patience.


Title: RE: LOTW
Post by: KH6DC on December 06, 2012, 05:14:40 PM
ARRL blames it on people submitting batch logs and they claim the program was written for users to upload daily or weekly.  I've been an off and on user for 7 years and a serious user within the past 5 years and don't beleive the system sufferred tis kind of issue.  Only after ARRL incorporated the WPX program since the ARRL are only after money.


Title: RE: LOTW
Post by: N4CR on December 06, 2012, 05:48:38 PM
I have found many times over the years that something stupid that a programmer does works fine on a small quantity of data and grinds to a halt with massive data. This is where systems analysis comes in, identifies the bottleneck and recommends a fix.

I work on systems with massive amounts of data at work every day. Two years ago, our business partners came to us and asked us to look at some software that was running slowly, the programmer that was contracted to write it was long gone. We did data analysis, performed some advanced changes to the SQL code (still did the same exact thing, though) and got a 15000% speed increase. Now what took hours takes seconds. We changed the batch timer that used to run every 5 minutes (and was skipping 3 to 10 of those 5 minute periods before the next one started) to running once a minute with 90% of the minute to spare.

So either the LoTW server side software is well written and tuned and the hardware is not up to the task or there is some combination of software and hardware bottleneck that is not being addressed.

This is definitely a solvable problem. All it takes is the desire, backing and talent to get it done.


Title: RE: LOTW
Post by: W6DXO on December 06, 2012, 05:50:55 PM
Unbelievable...The ARRL LOTW site now states that processing is running about 7 days behind:

"Dec 6, 2012: LoTW Log Processing is About 7 Days Behind -- For more information on what has been happening with LoTW lately, please see this news item."


If this were happening in a mission critical environment in Silicon Valley someone's
head would be on a pike!

73 de harry, W6DXO




Title: RE: LOTW
Post by: NU4B on December 07, 2012, 11:23:00 AM
Unbelievable...The ARRL LOTW site now states that processing is running about 7 days behind:

"Dec 6, 2012: LoTW Log Processing is About 7 Days Behind -- For more information on what has been happening with LoTW lately, please see this news item."


If this were happening in a mission critical environment in Silicon Valley someone's
head would be on a pike!

73 de harry, W6DXO

But its not - its about QSO confirmations for a bunch of hams.  ;D

Still waiting on CQWWCW processing. 10 meter contest about to start in about 4.5 hours.





Title: RE: LOTW
Post by: WA0CRI on December 07, 2012, 01:01:02 PM
They're running a lot more than 7 days behind.  My last successful log update was November 24 .  .  .


Title: RE: LOTW
Post by: AF3Y on December 07, 2012, 01:42:11 PM
Unbelievable...The ARRL LOTW site now states that processing is running about 7 days behind:

"Dec 6, 2012: LoTW Log Processing is About 7 Days Behind -- For more information on what has been happening with LoTW lately, please see this news item."


If this were happening in a mission critical environment in Silicon Valley someone's
head would be on a pike!

73 de harry, W6DXO

But its not - its about QSO confirmations for a bunch of hams.  ;D

Still waiting on CQWWCW processing. 10 meter contest about to start in about 4.5 hours.




"Houston - - -   WE HAVE A PROBLEM"   ;D

Gene AF3Y


Title: RE: LOTW
Post by: AC4RD on December 08, 2012, 05:55:26 AM
If this were happening in a mission critical environment in Silicon Valley someone's
head would be on a pike!

Yeah, true.  But consider how much the LOTW system has grown over the last few years.  And consider how much we PAY for the LOTW service.  ARRL is running this without asking American hams to chip in; they don't even require you to be an ARRL member to use the service.

I for one would be glad to pay $10 a year or a $20 one-time fee to help fund LOTW.  The discussion of how database systems work fine with small amounts of data but bog down with large amounts--that's true as can be.  But a good systems analysis costs money. 

Considering that we aren't paying *anything* for LOTW, I'm not about to gripe about slow processing, not me.


Title: RE: LOTW
Post by: NU4B on December 08, 2012, 09:54:32 AM
Quote
Considering that we aren't paying *anything* for LOTW, I'm not about to gripe about slow processing, not me.

And how long have you been a ham?  ;D ;D


Title: RE: LOTW
Post by: AC4RD on December 08, 2012, 11:08:45 AM
And how long have you been a ham? 

Since 1991.  And I've been an Extra since 1992.  And I guess you meant that as a joke; hams love to complain.  But it's serious to me:  LOTW is a GREAT service and considering we're getting it for free, it's irrational to blame ARRL for a processing slowdown.  Anybody who is upset can send $20 or $50 to Newington--what you'd spend getting a few direct DX cards or what you'd donate to a DXpedition, and designate the money to LOTW improvements, and if enough of us DO that, THEN people might have some grounds for complaining about it. 


Title: RE: LOTW
Post by: N9KX on December 08, 2012, 12:04:13 PM

Yeah, true.  But consider how much the LOTW system has grown over the last few years.  And consider how much we PAY for the LOTW service.  ARRL is running this without asking American hams to chip in; they don't even require you to be an ARRL member to use the service.

I for one would be glad to pay $10 a year or a $20 one-time fee to help fund LOTW.  The discussion of how database systems work fine with small amounts of data but bog down with large amounts--that's true as can be.  But a good systems analysis costs money. 

Considering that we aren't paying *anything* for LOTW, I'm not about to gripe about slow processing, not me.

Well, there are fees charged for award credits, and you have to be a member to apply for an award, so it does generate some revenue.  It is also a great way for the ARRL to net hams the world over. 

Asking if hams complain too much, may be like asking if water is too wet, but still.  If you put the ARRL name on a service, it reflects upon the ARRL and amateur radio as a whole.  I am not an IT expert, but surely it is a little embarrassing to have it limping down the road in its present state.  Hopefully the hardware upgrade stops all of our squeaking, but I would not bet the farm on it.  Probably the software is in need of some good tweaks.  N4CR pointed out a simple change in the software can sometimes make a system 15,000 times faster.  If that is true, getting that done would be money well spent. 


Title: RE: LOTW
Post by: NU4B on December 08, 2012, 12:28:22 PM
And how long have you been a ham? 

Since 1991.  And I've been an Extra since 1992.  And I guess you meant that as a joke; hams love to complain.  But it's serious to me:  LOTW is a GREAT service and considering we're getting it for free, it's irrational to blame ARRL for a processing slowdown.  Anybody who is upset can send $20 or $50 to Newington--what you'd spend getting a few direct DX cards or what you'd donate to a DXpedition, and designate the money to LOTW improvements, and if enough of us DO that, THEN people might have some grounds for complaining about it. 


Yes, I was just joking with you. And I certainly didn't mean to belittle the seriousness of your post. (Even in the most serious of times, events, or such I can still recognize the irony or humor of a situation. Maybe its what keeps me going when things get rough.)

Actually I think K9AIM made one of the most stunning statements when you think about it:

"If you put the ARRL name on a service, it reflects upon the ARRL and amateur radio as a whole.  I am not an IT expert, but surely it is a little embarrassing to have it limping down the road in its present state."

We are talking about one of (if not the) premier amateur radio organizations in the world.


Title: RE: LOTW
Post by: AC4RD on December 08, 2012, 12:58:14 PM
"If you put the ARRL name on a service, it reflects upon the ARRL and amateur radio as a whole.  I am not an IT expert, but surely it is a little embarrassing to have it limping down the road in its present state."

We are talking about one of (if not the) premier amateur radio organizations in the world.

It's a valid point, absolutely.  You and AIM are correct that it certainly doesn't look good for the ARRL to have slowdowns like this.  But maybe it's time for ARRL to start charging American hams, or North American hams, some small fee to use the system?  LOTS of us are using it, and it's a great thing not to have to wait for a card that may or may not arrive.  And a small annual (or one-time) fee might fund more hardware and the services of an IT professional to tweak the system.


Title: RE: LOTW
Post by: AJ4RW on December 08, 2012, 01:36:21 PM
Quote
And consider how much we PAY for the LOTW service. ARRL is running this without asking American hams to chip in; they don't even require you to be an ARRL member to use the service.

Considering that we aren't paying *anything* for LOTW, I'm not about to gripe about slow processing, not me.

If this is true, what do my membership dues go to?  I know ARRL derives its revenues from several sources and dues are just one source but still dues contribute.  Granted, to use LOTW you don't have to be a member.  But still my membership dues do contribute to LOTW no matter how small an amount.
Randy


Title: RE: LOTW
Post by: NU1O on December 08, 2012, 06:23:04 PM
"If you put the ARRL name on a service, it reflects upon the ARRL and amateur radio as a whole.  I am not an IT expert, but surely it is a little embarrassing to have it limping down the road in its present state."

We are talking about one of (if not the) premier amateur radio organizations in the world.

It's a valid point, absolutely.  You and AIM are correct that it certainly doesn't look good for the ARRL to have slowdowns like this.  But maybe it's time for ARRL to start charging American hams, or North American hams, some small fee to use the system?  LOTS of us are using it, and it's a great thing not to have to wait for a card that may or may not arrive.  And a small annual (or one-time) fee might fund more hardware and the services of an IT professional to tweak the system.


Why are you saying hams do not pay to use LoTW. They most certainly do! They charge for each confirmation you use as a credit. I think it's 12 cents a credit. Then there is an application fee and various award or endorsement fees. I was charged about $155 last month for using LoTW. 

I have also read, but I can no longer find the article, that LoTW generated $900,000 in revenue last year.

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on December 08, 2012, 06:33:31 PM


I have also read, but I can no longer find the article, that LoTW generated $900,000 in revenue last year.

73,

Chris/NU1O

I so hope that is correct...  If so, then LoTW will be fixed instantly...

73's


Title: RE: LOTW
Post by: AJ4RW on December 08, 2012, 07:14:40 PM
Quote
If so, then LoTW will be fixed instantly...

I think a more appropriate word to use is "should".  Don't forget, ARRL has ARES to consider giving money to hi hi.
Randy


Title: RE: LOTW
Post by: NU1O on December 08, 2012, 08:40:21 PM
Quote
If so, then LoTW will be fixed instantly...

I think a more appropriate word to use is "should".  Don't forget, ARRL has ARES to consider giving money to hi hi.
Randy

What the hell is ARES?  It sounds like a disease.

73,

Chris/NU1O


Title: RE: LOTW
Post by: NK7Z on December 08, 2012, 09:03:33 PM

What the hell is ARES?  It sounds like a disease.

73,

Chris/NU1O

Go here to see what it is:
http://bit.ly/VWe7mD

73's


Title: RE: LOTW
Post by: N9KX on December 09, 2012, 01:34:42 AM

Actually I think K9AIM made one of the most stunning statements when you think about it:

"If you put the ARRL name on a service, it reflects upon the ARRL and amateur radio as a whole.  I am not an IT expert, but surely it is a little embarrassing to have it limping down the road in its present state."

We are talking about one of (if not the) premier amateur radio organizations in the world.

The more I reflect upon it, the more it seems to me that it is in the ARRL's financial interest to spend money making LoTW state of the art.  One of the neat things about ham radio when I was a teen (and novice) in 1976, was that ham radio operators were cool and a lot of ham radio was innovative and on the cutting edge.  I'm not sure what it would take for the ARRL to make ham radio look as appealing to kids today as things like OSCAR looked to me when I was a kid, but I doubt technolgically savvy teens today are impressed by our LoTW system in its present state.

Hopefully that changes, because LoTW can be something that puts the ARRL in a very favorable light. 

Whoever came up with the idea of LoTW and got it going was visionary.  It deserves a full tune up  ;)


Title: RE: LOTW
Post by: WA0CRI on December 09, 2012, 09:42:55 AM
Does anybody have a LOTW QSO update more recent than November 24th ?


Title: RE: LOTW
Post by: WW3QB on December 09, 2012, 09:52:06 AM
Does anybody have a LOTW QSO update more recent than November 24th ?

2012-12-05 08:30:42 LOTW_QSO: Processing file: 20121129032832.10147
2012-12-05 08:30:42 LOTW_QSO: User file: ww3qb76a.tq8
2012-12-05 08:30:42 LOTW_QSO: Certificate found for WW3QB - UNITED STATES OF AMERICA (291)
2012-12-05 08:30:47 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:47 LOTW_QSO:    QSO: WW3QB ZD9UW 2012-10-01 20:38:00Z 15M SSB
2012-12-05 08:30:48 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:48 LOTW_QSO:    QSO: WW3QB TT8TT 2012-10-08 03:18:00Z 40M CW
2012-12-05 08:30:49 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:49 LOTW_QSO:    QSO: WW3QB PT0S 2012-11-22 19:52:00Z 17M CW
2012-12-05 08:30:49 LOTW_QSO: Successfully processed 9 QSO records in 6.315923 seconds
2012-12-05 08:30:49 LOTW_QSO: 3 QSL records entered
2012-12-05 08:30:49 LOTW_QSO: No errors encountered

It took six days in the queue to process. My last QSL is from D3AA 2012-12-09 04:50:20


Title: RE: LOTW
Post by: AJ4RW on December 09, 2012, 10:19:49 AM
Quote
It took six days in the queue to process. My last QSL is from D3AA 2012-12-09 04:50:20

Wow, that was fast!  The last upload for me was on Dec 2, 2012 and still nada.  It's a good thing I'm not in a big hurry. ::)
Randy


Title: RE: LOTW
Post by: N9KX on December 09, 2012, 12:32:55 PM
It took six days in the queue to process. My last QSL is from D3AA 2012-12-09 04:50:20

what is the QSO date of your most recent LoTW QSL?


Title: RE: LOTW
Post by: WW3QB on December 09, 2012, 12:42:20 PM
It took six days in the queue to process. My last QSL is from D3AA 2012-12-09 04:50:20

what is the QSO date of your most recent LoTW QSL?

The QSO date of my most recent LoTW QSL (D3AA) is 2012-08-08. QSLed on 2012-12-09.
The QSO date of my most recent QSO that was QSLed on LoTW (PT0S) is 2012-11-22. QSLed on 2012-12-05.


Title: RE: LOTW
Post by: N9KX on December 09, 2012, 01:36:45 PM
The QSO date of my most recent QSO that was QSLed on LoTW (PT0S) is 2012-11-22. QSLed on 2012-12-05.

My most recent QSO date of a LoTW QSL is Nov. 25

I too had some D3AA confirmations yesterday, but the QSO's were from long ago...

Has anyone had LoTW QSL a QSO that took place after Dec 1?


Title: RE: LOTW
Post by: WA0CRI on December 09, 2012, 07:35:36 PM
I appreciate the LOTW service but wish they would be more frank about the actual situation.  Don't state that the system is running 7 days behind when some accounts such as mine is are 16 days behind - I've had no update in submitted QSO's since November 24.


Title: RE: LOTW
Post by: WW3QB on December 09, 2012, 07:44:45 PM
I appreciate the LOTW service but wish they would be more frank about the actual situation.  Don't state that the system is running 7 days behind when some accounts such as mine is are 16 days behind - I've had no update in submitted QSO's since November 24.

You may be one of the lost files from before Nov. 28 2103Z. http://www.arrl.org/news/update-to-logbook-of-the-world-bug-fix


Title: RE: LOTW
Post by: WA0CRI on December 09, 2012, 08:04:13 PM
I waited until after December 3 as their notice requested and have resubmitted my logs numerous times but no updates have ever appeared after November 24.  They obviously have much more serious problems than they are disclosing.  As far as I can determine my account in the LOTW database ended on November 24th.


Title: RE: LOTW
Post by: W5DQ on December 09, 2012, 08:56:03 PM
Does anybody have a LOTW QSO update more recent than November 24th ?

2012-12-05 08:30:42 LOTW_QSO: Processing file: 20121129032832.10147
2012-12-05 08:30:42 LOTW_QSO: User file: ww3qb76a.tq8
2012-12-05 08:30:42 LOTW_QSO: Certificate found for WW3QB - UNITED STATES OF AMERICA (291)
2012-12-05 08:30:47 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:47 LOTW_QSO:    QSO: WW3QB ZD9UW 2012-10-01 20:38:00Z 15M SSB
2012-12-05 08:30:48 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:48 LOTW_QSO:    QSO: WW3QB TT8TT 2012-10-08 03:18:00Z 40M CW
2012-12-05 08:30:49 LOTW_QSO: QSO record QSLs a contact
2012-12-05 08:30:49 LOTW_QSO:    QSO: WW3QB PT0S 2012-11-22 19:52:00Z 17M CW
2012-12-05 08:30:49 LOTW_QSO: Successfully processed 9 QSO records in 6.315923 seconds
2012-12-05 08:30:49 LOTW_QSO: 3 QSL records entered
2012-12-05 08:30:49 LOTW_QSO: No errors encountered

It took six days in the queue to process. My last QSL is from D3AA 2012-12-09 04:50:20

My last upload was 11-27 and the results below show it was processed on 11-29

Date/Time:          2012-11-29 06:02:12
User:          w5dq
File:          LotWUpload.tq8
Messages:      

2012-11-29 06:02:12 LOTW_QSO: Processing file: msg-22022-1.tq8
2012-11-29 06:02:12 LOTW_QSO: User file: LotWUpload.tq8
2012-11-29 06:02:12 LOTW_QSO: Certificate found for W5DQ - UNITED STATES OF AMERICA (291)
2012-11-29 06:02:18 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:18 LOTW_QSO:    QSO: W5DQ 5W1SA 2012-11-25 03:52:07Z 20M CW
2012-11-29 06:02:25 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:25 LOTW_QSO:    QSO: W5DQ DL4CF 2012-11-25 07:52:00Z 40M CW
2012-11-29 06:02:29 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:29 LOTW_QSO:    QSO: W5DQ JE4CIL 2012-11-25 23:58:24Z 15M CW
2012-11-29 06:02:31 LOTW_QSO: WARNING: QSO duplicates existing record
2012-11-29 06:02:31 LOTW_QSO:    QSO: W5DQ JJ4CDW 2011-11-27 01:43:59Z 15M CW
2012-11-29 06:02:44 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:44 LOTW_QSO:    QSO: W5DQ R0FA 2012-11-25 01:27:51Z 15M CW
2012-11-29 06:02:47 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:47 LOTW_QSO:    QSO: W5DQ UA0AGI 2012-11-25 02:06:47Z 20M CW
2012-11-29 06:02:52 LOTW_QSO: QSO record QSLs a contact
2012-11-29 06:02:52 LOTW_QSO:    QSO: W5DQ V51YJ 2012-11-24 20:24:07Z 15M CW
2012-11-29 06:02:59 LOTW_QSO: Successfully processed 212 QSO records in 47.157208 seconds
2012-11-29 06:02:59 LOTW_QSO: 1 record was a duplicate
2012-11-29 06:02:59 LOTW_QSO: 6 QSL records entered
2012-11-29 06:02:59 LOTW_QSO: No errors encountered

Gene W5DQ


Title: RE: LOTW
Post by: NI0C on December 10, 2012, 07:02:04 AM
The last of three batches I uploaded to LoTW during the ARRL 160m contest has been processed by the system.  The time from upload to accepting my QSO's in the system was about 191 hours (almost exactly eight days).  I'm continuing to upload small files as I make QSO's, with confidence they will not be lost. 

I did lose a couple of small files that were uploaded prior to the "fix," but those have been taken care of by repeat uploads.

I understand (through reading the DXLab Reflector) that the ARRL is acquiring additional resources, and hopes to have those in place several weeks from now.  In the meantime, it appears that LoTW is acting as a delay line, but is not losing anything in the process.

73,
Chuck  NI0C


Title: RE: LOTW
Post by: WS4T on December 10, 2012, 07:26:04 AM
WA0CRI: You should upload your log(s) again. They were clearly lost.

Same happened to me with my CQ WW CW log (1100+ QSOs). I uploaded it again and it was finally processed.

73,
Gary, ES1WST


Title: RE: LOTW
Post by: N9KX on December 10, 2012, 08:15:13 AM
Quote from: LoTW news and notes
Dec 10, 2012: LoTW Log Processing is About 8 Days Behind -- For more information on what has been happening with LoTW lately, please see this news item http://www.arrl.org/news/arrl-s-logbook-of-the-world-bug-fix-is-on-the-way (http://www.arrl.org/news/arrl-s-logbook-of-the-world-bug-fix-is-on-the-way).  NEW... The bug that was causing the problem has been fixed http://www.arrl.org/news/update-to-logbook-of-the-world-bug-fix (http://www.arrl.org/news/update-to-logbook-of-the-world-bug-fix).  But there is still a large queue of logs steadily being processed.  Thanks for your patience.

no explanation as to why -- if the bug has been fixed -- the processing is getting further and further behind...  ???
would be nice to see a 'future plans for LoTW' paragraph ...


Title: RE: LOTW
Post by: N3QE on December 10, 2012, 09:00:10 AM
no explanation as to why -- if the bug has been fixed -- the processing is getting further and further behind...  ???
would be nice to see a 'future plans for LoTW' paragraph ...

The explanation is, "CQ WW CW". What would normally be nearly a month's or more of QSO's, were submitted the evening of Nov 25 and the rest on Nov 26 and a few days afterwards. While those were being processed, it seemed like it was "getting further and further behind" when in reality the system was processing more QSO's per hour than ever before.

While some trumpeted CQ WW Phone as the biggest contest, in terms of LOTW usage the CW guys are far more likely to upload promptly (and the conditions for CQ WW CW were very very good!) and result in LOTW slowdowns. Similar thing has happened every November for past several years.


Title: RE: LOTW
Post by: K4JK on December 10, 2012, 09:10:15 AM
Honestly I think LOTW is caught in a negative feedback loop due to several hundred impatient hams generating queries by refreshing their QSL list every 5 minutes, uploading duplicate QSOs multiple times, and uploading every single QSO as an individual batch file.


WHY IS THE SYSTEM SLOWING DOWWWNNNNNNN??????? <hits F5 80 times in one minute>


Title: RE: LOTW
Post by: N9KX on December 10, 2012, 10:41:00 AM
no explanation as to why -- if the bug has been fixed -- the processing is getting further and further behind...  ???
would be nice to see a 'future plans for LoTW' paragraph ...

The explanation is, "CQ WW CW". What would normally be nearly a month's or more of QSO's, were submitted the evening of Nov 25 and the rest on Nov 26 and a few days afterwards. While those were being processed, it seemed like it was "getting further and further behind" when in reality the system was processing more QSO's per hour than ever before.

While some trumpeted CQ WW Phone as the biggest contest, in terms of LOTW usage the CW guys are far more likely to upload promptly (and the conditions for CQ WW CW were very very good!) and result in LOTW slowdowns. Similar thing has happened every November for past several years.

well if it happened a year ago and not much was changed i guess the ARRL and amateur radio just cannot be expected to keep up in such a hi-tech world as we have today...

[that was sarcasm, btw]


Title: RE: LOTW
Post by: AD9DX on December 10, 2012, 01:46:35 PM
no explanation as to why -- if the bug has been fixed -- the processing is getting further and further behind...  ???
would be nice to see a 'future plans for LoTW' paragraph ...

The explanation is, "CQ WW CW". What would normally be nearly a month's or more of QSO's, were submitted the evening of Nov 25 and the rest on Nov 26 and a few days afterwards. While those were being processed, it seemed like it was "getting further and further behind" when in reality the system was processing more QSO's per hour than ever before.

While some trumpeted CQ WW Phone as the biggest contest, in terms of LOTW usage the CW guys are far more likely to upload promptly (and the conditions for CQ WW CW were very very good!) and result in LOTW slowdowns. Similar thing has happened every November for past several years.

While the slow down still frusterates me to no end... It sure beats waiting on cards from some of the easy ones.


Title: RE: LOTW
Post by: WA0CRI on December 10, 2012, 01:54:54 PM
WA0CRI: You should upload your log(s) again. They were clearly lost.

Same happened to me with my CQ WW CW log (1100+ QSOs). I uploaded it again and it was finally processed.

73,
Gary, ES1WST

Gary,
I have uploaded daily logs each day since Nov. 24 and not a single QSO has shown up - date of last QSO is still 11/24/12.


Title: RE: LOTW
Post by: NU1O on December 10, 2012, 02:40:50 PM

[that was sarcasm, btw]

Really?  I would never have known.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AF3Y on December 10, 2012, 04:43:12 PM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y


Title: RE: LOTW
Post by: AB8MA on December 10, 2012, 05:26:36 PM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

Interesting data there. Every two hours they fall 1 hour 40 minutes farther behind.


Title: RE: LOTW
Post by: NU1O on December 10, 2012, 05:35:34 PM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

Interesting data there. Every two hours they fall 1 hour 40 minutes farther behind.

I just noticed that ( don't know if I missed it before or it is new) and I see them falling further and further behind. They are no doubt getting many logs from the 10 meter contest now. I uploaded mine about an hour ago.  Another 600 QSOs in the que. My December 1st QSOs have now been processed by LoTW but no new QSLs since 5X1NH.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AD9DX on December 10, 2012, 05:39:15 PM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

Interesting data there. Every two hours they fall 1 hour 40 minutes farther behind.

I just noticed that ( don't know if I missed it before or it is new) and I see them falling further and further behind. They are no doubt getting many logs from the 10 meter contest now. I uploaded mine about an hour ago.  Another 600 QSOs in the que. My December 1st QSOs have now been processed by LoTW but no new QSLs since 5X1NH.

73,

Chris/NU1O

I wonder if the 10m contest will cause that much of an issue.  The band sucked.


Title: RE: LOTW
Post by: N9KX on December 10, 2012, 07:29:30 PM

[that was sarcasm, btw]

Really?  I would never have known.

73,

Chris/NU1O

that's alright, if I had been more patient and let you wait 11 days, LoTW would have confirmed it  ;)


Title: RE: LOTW
Post by: WW3QB on December 10, 2012, 08:40:27 PM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

Interesting data there. Every two hours they fall 1 hour 40 minutes farther behind.


I never would have guessed there are so many logs in the queue. I assume today's flood is from the 10m contest. But there are two big storm clouds nearby. There is going to be a rush for year end DXCC endorsements for the yearbook. That's a hard deadline to get applications in (unless extended). And on Jan. 5 there is the RTTY roundup.


Title: RE: LOTW
Post by: WA0CRI on December 11, 2012, 06:35:11 AM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

According to the queue, the system is processing logs submitted 12/2/2012.  What happened to my daily logs submitted prior to 12/2/2012?  This is why I wish they would be more forthcoming about what's going on.  It is obvious that not only have logs been lost but some accounts, such as mine, are in effect "locked out".

Help,  Doug  WA0CRI


Title: RE: LOTW
Post by: AB8MA on December 11, 2012, 07:17:33 AM
2012-12-11 04:01:43 Currently Processing Log Uploaded 2012-12-02 17:58:30
(8d 10h 03m 13s ago)

11 Hours later:

2012-12-11 15:02:34 Currently Processing Log Uploaded 2012-12-02 20:47:52
(8d 18h 14m 42s ago)

Without bringing out the slide rule, I would assume there was some downtime last night.


Title: RE: LOTW
Post by: WB3BEL on December 11, 2012, 07:45:03 AM

According to the queue, the system is processing logs submitted 12/2/2012.  What happened to my daily logs submitted prior to 12/2/2012?  This is why I wish they would be more forthcoming about what's going on.  It is obvious that not only have logs been lost but some accounts, such as mine, are in effect "locked out".

Help,  Doug  WA0CRI

I'd recommend that you log in and select the tab Your Account and then select Your Activity.

Look to see when the last log you submitted is indicated as File Processed.

If you see that a log was processed and you have no increase in your QSO records then there is some other problem.

If you see that a log was processed and you have an increase in your QSO record count but no matches.  Then you might have a logging problem like your time is off due to the time change or maybe some other error on your part.  Could be wrong band or wrong mode or wrong date or ??  Or maybe you just talked to ops who don't use LoTW.


Title: RE: LOTW
Post by: AB8MA on December 11, 2012, 07:46:06 AM
No sooner than I said all that, one of my SWAINS ISLAND confirmations showed up. :)


Title: RE: LOTW
Post by: W5DQ on December 11, 2012, 08:35:33 AM
For what its worth, LOTW now has a LIVE logbook queue status page!

http://www.arrl.org/logbook-queue-status

73, Gene AF3Y

Interesting data there. Every two hours they fall 1 hour 40 minutes farther behind.

The other 20 minutes is spent calculating how far behind they are :)


Title: RE: LOTW
Post by: WA8UEG on December 11, 2012, 09:04:11 AM
Thread typical of so many in todays society, I want it NOW, don't make me wait or I have to THINK to much to use it. If you are interested in ARRL awards or now WPX (WAZ is on it's way) then live with it. If not upload for those that do (IT'S FREE) if that's to much of a bother then don't, spend the $3.50 or so and send the card direct then wait a few months to get it back. If your not into awards or helping fellow hams then just forget it. life will go on.


Title: RE: LOTW
Post by: N9KX on December 11, 2012, 09:46:47 AM
Thread typical of so many in todays society, I want it NOW, don't make me wait or I have to THINK to much to use it. If you are interested in ARRL awards or now WPX (WAZ is on it's way) then live with it. If not upload for those that do (IT'S FREE) if that's to much of a bother then don't, spend the $3.50 or so and send the card direct then wait a few months to get it back. If your not into awards or helping fellow hams then just forget it. life will go on.

It is easy to dismiss complaints as mere complaining; it is another thing altogether to glean the merit from said complaints. LoTW is limping down the road, and anyone with an interest in competent technology or with compassion for technology in need of help has every right to chime in.  As I said, LoTW is a reflection on the ARRL and on amateur radio and could be a shining example of competence and state-of-the-art technology. 

So, if those with the power to correct the problems are disinclined to see the problem as important, it could very well be that the complainers are doing the ARRL a huge favor.  This is not a new problem.

If LoTW is *designed* to be 10 days behind -- just put that on the LoTW homepage and request patience for those who expect it to be faster.   ;)


Title: RE: LOTW
Post by: WA8UEG on December 11, 2012, 10:15:54 AM
Everyone at ARRL associated with LOTW is well aware and working on a fix to make the system faster. There is a news column on the LOTW sight that is updated and lets everyone know what to expect. Here is the one from today.

LOTW status in www.arrl.org/logbook-of-the-world will be updated daily to convey information of interest to the user community, including planned downtime and changes that will impact operations. In addition, the state of LotW's processing queue is now updated hourly in www.arrl.org/logbook-queue-status.

Updated hardware to improve LoTW's throughput is on order and expected in six to eight weeks. We appreciate the user community's patience while we while work to bring LoTW's performance to an acceptable level.   


Title: RE: LOTW
Post by: N9KX on December 11, 2012, 10:22:49 AM
Everyone at ARRL associated with LOTW is well aware and working on a fix to make the system faster. There is a news column on the LOTW sight that is updated and lets everyone know what to expect. Here is the one from today.

LOTW status in www.arrl.org/logbook-of-the-world will be updated daily to convey information of interest to the user community, including planned downtime and changes that will impact operations. In addition, the state of LotW's processing queue is now updated hourly in www.arrl.org/logbook-queue-status.

Updated hardware to improve LoTW's throughput is on order and expected in six to eight weeks. We appreciate the user community's patience while we while work to bring LoTW's performance to an acceptable level.   


Hopefully this means the software has been fully vetted and found 'up-to-speed' (or will be brought 'up-to-speed' in the interim)...


Title: RE: LOTW
Post by: WA0CRI on December 11, 2012, 10:23:59 AM
Quote from:  link=topic=86250.msg639403#msg639403 date=1355240703

According to the queue, the system is processing logs submitted 12/2/2012.  What happened to my daily logs submitted prior to 12/2/2012?  This is why I wish they would be more forthcoming about what's going on.  It is obvious that not only have logs been lost but some accounts, such as mine, are in effect "locked out".

Help,  Doug  WA0CRI

I'd recommend that you log in and select the tab Your Account and then select Your Activity.

Look to see when the last log you submitted is indicated as File Processed.

If you see that a log was processed and you have no increase in your QSO records then there is some other problem.

If you see that a log was processed and you have an increase in your QSO record count but no matches.  Then you might have a logging problem like your time is off due to the time change or maybe some other error on your part.  Could be wrong band or wrong mode or wrong date or ??  Or maybe you just talked to ops who don't use LoTW.

Just to clarify - I've submitted 13 logs since 11/24/2012 and the QSO count has remained unchanged.  Moreover, I have received QSL matches in the meantime, but none for QSO's after 11/24/2012.  When I look at the details of my LOTW activity it shows the message "processing aborted" on log submissions after 11/24/2012.

For those who say I may be impatient or expecting too much of the system - I've done exactly what was requested.  I waited until after Dec. 3 and resubmitted logs as directed by the LOTW notifications.  I just think there is something broken in the system.


Title: RE: LOTW
Post by: WA8UEG on December 11, 2012, 10:31:50 AM
Have you  called them? I have had nothing but great response when contacting them directly with question, concern or problem.


Title: RE: LOTW
Post by: WB3BEL on December 11, 2012, 10:40:48 AM
Just to clarify - I've submitted 13 logs since 11/24/2012 and the QSO count has remained unchanged.  Moreover, I have received QSL matches in the meantime, but none for QSO's after 11/24/2012.  When I look at the details of my LOTW activity it shows the message "processing aborted" on log submissions after 11/24/2012.

For those who say I may be impatient or expecting too much of the system - I've done exactly what was requested.  I waited until after Dec. 3 and resubmitted logs as directed by the LOTW notifications.  I just think there is something broken in the system.

Do you get the Processing Aborted - Invalid Certificate message?

If so see the hints under this section on:
 arrl.org/lotw-help.

I agree that you can get it sorted out if you call them.  If you don't want to talk to them you can open a help ticket by going to:
 arrl.org/lotw-help-ticket


Title: RE: LOTW
Post by: N9KX on December 11, 2012, 11:22:14 AM
No sooner than I said all that, one of my SWAINS ISLAND confirmations showed up. :)

I have had two NH8S confirmations today: 12 meters at 15:46:49z and 40 meters at 18:21:47z (which gives me 99 confirmations for 40 meters)

almost 3 hours between the 12m and 40m confirmations.  I wonder how big the NH8S upload is?
answer to own question: Total QSOs:  105,455  http://www.nh8s.org/



Title: RE: LOTW
Post by: AB8MA on December 11, 2012, 12:12:48 PM
NH8S obviously had more than one user file uploaded to LoTW. In between my first and second QSL there were a lot of other files processed.


Title: RE: LOTW
Post by: WA0CRI on December 11, 2012, 01:14:26 PM
Have you  called them? I have had nothing but great response when contacting them directly with question, concern or problem.
I did contact them and they got back to me within a couple hours with an e-mail reply referring me to their official notice about re-submitting any lost logs after 11/25/2012 and that logs will be processed in the order in which they arrive.


Title: RE: LOTW
Post by: AF3Y on December 11, 2012, 02:07:06 PM
Just checked LOTW and all five of my Swains QSLs are there as well as all six D3AA QSLs. :D

73, Gene AF3Y


Title: RE: LOTW
Post by: AB8MA on December 11, 2012, 08:30:35 PM
I remember when programmers such as myself could sit at a bar with hardware engineers and jokingly point beer bottles at each other while discussing performance issues at a particular site.

I was usually of the opinion that it was a hardware problem.

I was usually wrong.

[rant=on]
I now see that in the past 2 hours, an additional 104 log files has entered the system. Yet, in that 2 hours the backlog has increased by 1 hour and 39 minutes.

I assume that the updated hardware (due to arrive in 6-8 weeks?) includes more than solid-state drives. A small mom and pop shop could get that in an hour. What mainframe are they using, anyway?

I have never been a DBA, but it seems to me that some threshold has been exceeded and needs to be analyzed. Updated hardware is not a long term fix.

I hope I am wrong. See above.

[/rant]


Title: RE: LOTW
Post by: SV1XV on December 11, 2012, 11:38:24 PM
What mainframe are they using, anyway
CDC-7600, I believe...


Title: RE: LOTW
Post by: N9KX on December 12, 2012, 02:00:55 AM
What mainframe are they using, anyway
CDC-7600, I believe...

do you mean this? http://mysite.verizon.net/vzesz4a6/current/id74.html (http://mysite.verizon.net/vzesz4a6/current/id74.html)


Title: RE: LOTW
Post by: AB8MA on December 12, 2012, 07:27:40 AM
What mainframe are they using, anyway
CDC-7600, I believe...

I cut my teeth on that system while at Control Data Corporation. In 1984 I left CDC and worked for Cray Research on site at Ford until 1995. :)


Title: RE: LOTW
Post by: KY6R on December 12, 2012, 07:38:02 AM
I have never been a DBA, but it seems to me that some threshold has been exceeded and needs to be analyzed. Updated hardware is not a long term fix.

I sure hope they aren't using MySQL with the MyISAM engine - and table level locking . . . . . and single threaded ETL. Hardware won't fix that. I just fixed a big Data Warehouse that had this problem at Lithium Technologies . . . . OUCHIES . . .


Title: RE: LOTW
Post by: K4JK on December 12, 2012, 07:49:57 AM

I sure hope they aren't using MySQL with the MyISAM engine - and table level locking . . . . . and single threaded ETL. Hardware won't fix that. I just fixed a big Data Warehouse that had this problem at Lithium Technologies . . . . OUCHIES . . .
If they are using MyISAM, the good news is that when they get the new server they can compile InnoDB or HEAP (probably InnoDB) into the mySQL instance there, and then convert the table type when they copy the data over... That should be pretty seamless. If the ETL is a bottleneck too then that will have to be redesigned I guess...


Title: RE: LOTW
Post by: KY6R on December 12, 2012, 08:07:54 AM
If the ETL is a bottleneck too then that will have to be redesigned I guess...

If ETL is the bottleneck and they have row level locking - then they can download Pentaho for Free (the Community edition), and it would be a total piece of cake to automatically multi-thread ETL and get much faster throughput.

Of course - I am being a big armchair quarterback  ;D. They might already have Row Level Locking and Multi-threaded ETL. If they do - then adding hardware and tuning the parallelism would do the trick. Inserting LOTW files should be pretty easy, actually.


Title: RE: LOTW
Post by: N7SMI on December 12, 2012, 08:46:17 AM
Inserting LOTW files should be pretty easy, actually.

Yep. As I've monitored the numbers over the last few weeks and have done the math, they are processing at most 2-3 QSOs per second. Even though they're dealing with a fair amount of data to query, this is extremely slow. I'd expect no more than a few milliseconds per transaction. My experience has shown that unless you are very seriously underpowered, hardware upgrades are simply a temporary bandaid on a more serious database/query design problem.

Not to compare LoTW to Google, but I'm always amazed how Google can query about 100 billion web pages (they index or re-index about 20 billion per day) and give me useful results in less time than it takes me to search for a filename in a folder on my own hard drive.


Title: RE: LOTW
Post by: WW3QB on December 12, 2012, 09:11:47 AM
Inserting LOTW files should be pretty easy, actually.

Yep. As I've monitored the numbers over the last few weeks and have done the math, they are processing at most 2-3 QSOs per second. Even though they're dealing with a fair amount of data to query, this is extremely slow. I'd expect no more than a few milliseconds per transaction. My experience has shown that unless you are very seriously underpowered, hardware upgrades are simply a temporary bandaid on a more serious database/query design problem.

Not to compare LoTW to Google, but I'm always amazed how Google can query about 100 billion web pages (they index or re-index about 20 billion per day) and give me useful results in less time than it takes me to search for a filename in a folder on my own hard drive.

Check my math, but at 3 QSOs per second, it will take 35 days to get to a log submitted today processed. So it is too late to submit a log for DXCC year end processing. It also means that it will not catch up before the RTTY roundup logs hit.


Title: RE: LOTW
Post by: EI2GLB on December 12, 2012, 10:13:26 AM
Some of you may have seen Clublog now do a Log matching option to see if your QSO matches other users.

Seemly Clublog can upload 4000 per second,

and 2 or 3 hours after they switched on the Log matching it had allready matched 5 million QSO's

and that was only running when they Log Queue was empty.

Clublog is ran as a hobby LOTW is a money making racket

If I was still a member of ARRL I would be giving them a very strongly worded email saying to get there house in order.

I hope you that are members have done so.

Trevor
EI2GLB


Title: RE: LOTW
Post by: AB8MA on December 12, 2012, 10:23:05 AM
Almost 2% of the total number of QSO's processed by LoTW since it started are currenty sitting in the queue waiting to be processed.


Title: RE: LOTW
Post by: N9KX on December 12, 2012, 12:39:00 PM
Almost 2% of the total number of QSO's processed by LoTW since it started are currenty sitting in the queue waiting to be processed.

are you sure?  that would be 1 out of every 50  :-\  do duplicates get stuck in the queue and never discarded?


Title: RE: LOTW
Post by: NI0C on December 12, 2012, 12:54:31 PM
Quote
are you sure?  that would be 1 out of every 50

Let's do the math:
The current backlog reported is approx. 8.94 million QSO records.  The database contains approx. 462 million QSO records.  Dividing 8.94 by 462 and multiplying by 100 yields 1.9 percent. 

This is significant, because it shows how usage of LoTW has accelerated recently.  LoTW has been in place for maybe 100 months (more or less).  Dividing the database size of 462 million records by 100 yields an average of 4.6 million records per month input to the system.  Thus the current backlog is nearly two average months worth of QSO's. 

73,
Chuck  NI0C


Title: RE: LOTW
Post by: K4JK on December 12, 2012, 01:01:14 PM
I would wager a significant portion of those 8.9 million QSOs are duplicates from people uploading their entire or partial logs multiple times. Sure there is a sizable amount of contest logs too but I highly doubt organic growth caused usage to spike that much in the past two months.


Title: RE: LOTW
Post by: N3QE on December 12, 2012, 01:27:30 PM
Thus the current backlog is nearly two average months worth of QSO's.  

Yes, it is true. CQ WW CW is nearly two average months worth of QSO's in 48 hours :-). Add on ARRL 160, the 10M contest, the sweepstakes the few weeks before that... and come to the conclusion that November is hardly an "average" month!!!!

By the way Chuck, your very nice paper QSL card arrived several days before my 160 log was processed by LOTW :-)

Tim.


Title: RE: LOTW
Post by: N7SMI on December 12, 2012, 02:22:33 PM
Quote
Check my math, but at 3 QSOs per second, it will take 35 days to get to a log submitted today processed.

Correct. For kicks, today I sampled several hours worth of status data from the LoTW site and the average has been 159.8 QSOs processed per minute (high of 280 and low of 44), or 2.66 per second. At that rate it will take 39 days to process the 9 million QSOs currently in the queue.

I uploaded logs about every other day through November and the most recent QSL I have is November 19th which was 22 days ago.

I'll repeat my data collection regularly and see how things change, but it seems to me that the entire system is teetering on the edge of going tits up (<- that may be a colloquialism, but I think you get the idea).


Title: RE: LOTW
Post by: NI0C on December 12, 2012, 02:41:28 PM
N3QE wrote:
Quote
By the way Chuck, your very nice paper QSL card arrived several days before my 160 log was processed by LOTW :-)

I'm glad you liked the card, Tim.  I enjoyed working with KB3IFH to design some cards using my own photos.   
73,
Chuck  NI0C


Title: RE: LOTW
Post by: N4CR on December 12, 2012, 06:03:31 PM
For kicks, today I sampled several hours worth of status data from the LoTW site and the average has been 159.8 QSOs processed per minute (high of 280 and low of 44), or 2.66 per second.

I work for a fortune 100 company as a software and database specialist.

If I heard a system was doing what is stated above, there is no way I'd start by looking for a problem in hardware. Without being able to do hands on analysis, I'd guess this system is doing table scans on reads or writes. Once the size of the database exceeds memory constraints, it becomes disk channel bound. Which drops the speed by a factor of about 1000. A primary indication of this is that the CPU load is low and the disk channel (read/write light) is balls to the wall.

If it is this, the temporary cure is to add ram to get all of the database in memory again, the real fix is to find out why the table scans are occurring and fix the design flaw.

But this is all wild speculation from hallway talk and that's a HUGE if.

From experience, I've found that throwing hardware at a problem should be the last step after every advantage can be wrung out of the software.


Title: RE: LOTW
Post by: N9KX on December 14, 2012, 01:53:19 AM

I work for a fortune 100 company as a software and database specialist.

If I heard a system was doing what is stated above, there is no way I'd start by looking for a problem in hardware. Without being able to do hands on analysis, I'd guess this system is doing table scans on reads or writes. Once the size of the database exceeds memory constraints, it becomes disk channel bound. Which drops the speed by a factor of about 1000. A primary indication of this is that the CPU load is low and the disk channel (read/write light) is balls to the wall.

If it is this, the temporary cure is to add ram to get all of the database in memory again, the real fix is to find out why the table scans are occurring and fix the design flaw.

But this is all wild speculation from hallway talk and that's a HUGE if.

From experience, I've found that throwing hardware at a problem should be the last step after every advantage can be wrung out of the software.

any chance you will be in CT anytime soon and are feeling charitable?    :D

seriously though, how much time would you expect a good analysis of the software to take, and how expensive is that?


Title: RE: LOTW
Post by: N2RJ on December 14, 2012, 06:12:05 AM

Not to compare LoTW to Google, but I'm always amazed how Google can query about 100 billion web pages (they index or re-index about 20 billion per day) and give me useful results in less time than it takes me to search for a filename in a folder on my own hard drive.

I've been inside a Google datacenter and they have a metric ton of hardware, literally.

Of course that's not the whole story. They are also very innovative with software, making extensive use of things like memcache (in YouTube).

We switched over a lot of our stuff here to use memcache and the performance increase was pretty staggering.


Title: RE: LOTW
Post by: N2RJ on December 14, 2012, 06:14:24 AM
I have never been a DBA, but it seems to me that some threshold has been exceeded and needs to be analyzed. Updated hardware is not a long term fix.

I sure hope they aren't using MySQL with the MyISAM engine - and table level locking . . . . . and single threaded ETL. Hardware won't fix that. I just fixed a big Data Warehouse that had this problem at Lithium Technologies . . . . OUCHIES . . .

That was a pain in the neck for us, especially since we were using that god awful MySQL replication. We've now switched to InnoDB. Much better.


Title: RE: LOTW
Post by: N4KC on December 14, 2012, 08:18:24 AM
OK, you DB AND C++ guys, you ready to put your skills where your mouths are?  From the ARRL web site:

New hardware that will improve LoTW’s throughput is on order and is expected to be running in six to eight weeks.

Rick Murphy, K1MU, and Dave Bernstein, AA6YQ, have been charged with rebooting the Trusted QSL open source project. If you have demonstrably strong C++ development skills that you’re interested in applying toward improving LoTW’s usability and efficiency, please contact Bernstein via e-mail aa6yq(at)ambersoft(dot)com.

We appreciate the user community’s patience while we while work to bring LoTW’s performance to an acceptable level.


73,

Don N4KC
www.n4kc.com
www.donkeith.com
(Author of the new book RIDING THE SHORTWAVES:
EXPLORING THE MAGIC OF AMATEUR RADIO)
 
 


Title: RE: LOTW
Post by: AC4RD on December 14, 2012, 08:36:58 AM

Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.


Title: RE: LOTW
Post by: N2RJ on December 14, 2012, 08:42:06 AM

Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.


he has a chip on his shoulder about LoTW because *gasp* you have to actually show your ID to be authenticated in the system and he doesn't like that.


Title: RE: LOTW
Post by: N4CR on December 15, 2012, 01:15:00 PM

I work for a fortune 100 company as a software and database specialist.

If I heard a system was doing what is stated above, there is no way I'd start by looking for a problem in hardware. Without being able to do hands on analysis, I'd guess this system is doing table scans on reads or writes. Once the size of the database exceeds memory constraints, it becomes disk channel bound. Which drops the speed by a factor of about 1000. A primary indication of this is that the CPU load is low and the disk channel (read/write light) is balls to the wall.

If it is this, the temporary cure is to add ram to get all of the database in memory again, the real fix is to find out why the table scans are occurring and fix the design flaw.

But this is all wild speculation from hallway talk and that's a HUGE if.

From experience, I've found that throwing hardware at a problem should be the last step after every advantage can be wrung out of the software.

any chance you will be in CT anytime soon and are feeling charitable?    :D

seriously though, how much time would you expect a good analysis of the software to take, and how expensive is that?

If it's MsSQL I could do the work remotely.

As far as database tuning goes, my first pass would be to turn on advanced logging, take a half hour of log and then run that log through the tuning wizard. It not only tells you what to fix but it then tells you how much time would be chopped by the change. Tuning on newer versions of MsSQL also revolves around data statistics and often the statistics never get analyzed or added.

Also, a nightly tuning run of optimizing indexes and updating statistics should be put in place.

The second pass would be to inspect the code for things that can be optimized in how the code is doing it's work. This is harder, slower work and some of the time it's unnecessary after the tuning wizard does it's thing because the bottleneck turns out to be internal to SQL's operations.

I don't mind volunteering my time. But I'm only expert at optimizing Microsoft SQL. I have no idea what the database architecture of LoTW is. If they want my help, they can contact me at my QRZ email address.

Timewise, a index tuning wizard run for MsSQL could take as little as a few hours and can provide huge benefits. Especially if the incoming data always look the same, which I expect on this system, it's pretty much all the same in data structure and process. Probably not a lot of variety.

Analyzing a database for changes that involve moving from first normal form to third normal form (which I call the minimum for data design) could take a total rewrite. Or it might be that a few simple changes could to a lot.

Creating stored procedures of queries that are currently in outboard code can make huge differences because when you use external software to perform queries, it compiles the query each time. If you use a stored procedure, it compiles the stored proc one time and from then on it runs the compled code. Depending on the complexity of the code, it could take a few days or a few months. I would expect some immediate gains in the first day, though.


Title: RE: LOTW
Post by: N9KX on December 15, 2012, 01:49:05 PM
Thanks Phil for your input -- and for volunteering to help if the ARRL wishes to contact you.

Have you by chance had time to review these other threads on the issue?

http://www.eham.net/ehamforum/smf/index.php/topic,86920.0.html (http://www.eham.net/ehamforum/smf/index.php/topic,86920.0.html)

http://www.eham.net/ehamforum/smf/index.php/topic,86898.0.html (http://www.eham.net/ehamforum/smf/index.php/topic,86898.0.html)



Title: RE: LOTW
Post by: N4CR on December 15, 2012, 08:04:43 PM
Yes, I saw them. And it would not change a thing about how I went about solving the problems.

This is all beating a phantom horse. We don't even have a clue what operating system and database is in place. This is all conjecture and that seldom solves problems.


Title: RE: LOTW
Post by: W6GX on December 15, 2012, 10:31:56 PM
As you know LoTW had a bug where some of the uploaded logs got overwritten.  I decided to re-upload all of my QSOs that weren't confirmed in LoTW.  I already knew that the bug affected me as I found several QSOs were not found in LoTW despite the fact they were uploaded to LoTW.  I didn't do a detailed log comparison between LoTW and the HRD Logbook as that exercise would be tedious.  Anyhow I received 15 new QSLs after I re-uploaded about 2,200 old QSOs to LoTW.  According to LoTW there were 16 QSOs from the upload that were neither replacements nor duplicates.  So I assume these 16 QSOs were the missing ones due to the bug.  Below is what it reported on the re-upload.  I recommend anyone to re-upload all of their unconfirmed QSOs to LoTW.  This way you will be sure that all of your QSOs are properly added in LoTW.

2012-12-16 02:05:07 LOTW_QSO: Successfully processed 2177 QSO records in 154.827350 seconds
2012-12-16 02:05:07 LOTW_QSO: 1483 records were replacements
2012-12-16 02:05:07 LOTW_QSO: 678 records were duplicates
2012-12-16 02:05:07 LOTW_QSO: 15 QSL records entered
2012-12-16 02:05:07 LOTW_QSO: No errors encountered

73,
Jonathan W6GX


Title: RE: LOTW
Post by: AB8MA on December 16, 2012, 05:56:37 AM
What's the difference between a replacement and a duplicate?


Title: RE: LOTW
Post by: N4CR on December 16, 2012, 09:50:37 AM
What's the difference between a replacement and a duplicate?

A duplicate, all fields are identical.

A replacement is where some field got updated, but it's still the same call at the same time. Like you update the state or mode or band.


Title: RE: LOTW
Post by: W6GX on December 16, 2012, 09:54:18 AM
What's the difference between a replacement and a duplicate?

A duplicate, all fields are identical.

A replacement is where some field got updated, but it's still the same call at the same time. Like you update the state or mode or band.

That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

73,
Jonathan W6GX


Title: RE: LOTW
Post by: N3QE on December 16, 2012, 10:15:24 AM
That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

You didn't think you made any changes, but something made a change. For example a change in the frequency field (e.g. 14 is different than 14.000 is different than 14.036 is different than 14.03600 is different than 14.03642), or a different grid square (e.g. FM19 and FM19ka are different), or just rounding off the time (e.g. 1:34:12 is different than 1:34:00) all seem to cause replacements. Some of these happen when importing log files between differnent log programs (e.g. rounding off frequency) and others can happen at TrustedQSL signing time (e.g. grid square). Some log file conversions (e.g. ADI to Cabrillo then back to ADI) are pretty much guaranteed to round off times and frequencies. Some log programs will needlessly round off times and frequencies or change the precision of the recorded numbers even when just importing and then re-exporting ADI.

Re-uploading thousands of QSO's needlessly is one reason why LOTW is so far behind. At first it was just the onslaught of November contest logs. But right now I think we're seeing a snowball effect, somebody sees the log they submitted a few days ago isn't processed yet, so they "fix it good" by re-uploading every QSO they've ever had, then when that's not processed yet the export and upload yet again, bringing LOTW even further behind. Most logging programs let you create the ADIF based on date range. If you think a file from the past hadn't been processed, go to "Your Account", then "Your Activity", and if something from longer than 2 weeks wasn't processed, then just upload the same (signed) .tq8 file again. No need to re-export. If it was uploaded in the past week and a half... it's probably in the queue. Check http://www.arrl.org/logbook-queue-status


Title: RE: LOTW
Post by: W6GX on December 16, 2012, 11:05:34 AM
That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

You didn't think you made any changes, but something made a change. For example a change in the frequency field (e.g. 14 is different than 14.000 is different than 14.036 is different than 14.03600 is different than 14.03642), or a different grid square (e.g. FM19 and FM19ka are different), or just rounding off the time (e.g. 1:34:12 is different than 1:34:00) all seem to cause replacements. Some of these happen when importing log files between differnent log programs (e.g. rounding off frequency) and others can happen at TrustedQSL signing time (e.g. grid square). Some log file conversions (e.g. ADI to Cabrillo then back to ADI) are pretty much guaranteed to round off times and frequencies. Some log programs will needlessly round off times and frequencies or change the precision of the recorded numbers even when just importing and then re-exporting ADI.

Re-uploading thousands of QSO's needlessly is one reason why LOTW is so far behind. At first it was just the onslaught of November contest logs. But right now I think we're seeing a snowball effect, somebody sees the log they submitted a few days ago isn't processed yet, so they "fix it good" by re-uploading every QSO they've ever had, then when that's not processed yet the export and upload yet again, bringing LOTW even further behind. Most logging programs let you create the ADIF based on date range. If you think a file from the past hadn't been processed, go to "Your Account", then "Your Activity", and if something from longer than 2 weeks wasn't processed, then just upload the same (signed) .tq8 file again. No need to re-export. If it was uploaded in the past week and a half... it's probably in the queue. Check http://www.arrl.org/logbook-queue-status

Thanks for the explanation.

I use HRD and it tells me which QSOs have been uploaded to LoTW.  There's no ADIF file that I need to create manually.  I never re-upload QSOs unless I have to, as in the case I mentioned earlier in my post.  As for the increasing backlog of unprocessed logs, that could be mostly due to more QSOs made per person and also new comers signing up.  For sure some of it is due to the LoTW bug.  However trying to pinpoint the exact cause is no more than just speculation.

73,
Jonathan W6GX


Title: RE: LOTW
Post by: EI2GLB on December 16, 2012, 02:40:35 PM
Are you saying I have a chip then you are very much mistaken, I am a Hugh supporter of LOTW and encourage people to use it daily,


I might have said its not easy for us DX to get signed up but I have no problem with the level of id the require,

What I have a problem with it there lack of intrest in getting more users and spending money on improving the system,

LOTW has saved me thousands of $$ in direct QSL's but I have spent hundreds on awards,

Trevor
EI2GLB



Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.


he has a chip on his shoulder about LoTW because *gasp* you have to actually show your ID to be authenticated in the system and he doesn't like that.


Title: RE: LOTW
Post by: WA0CRI on December 16, 2012, 02:58:25 PM
I believe there are more serious problems with LOTW than are being discussed on the LOTW site.  The last update to my QSO count was 23 days ago in spite of daily uploads to the system.  My daily activity shows the uploads entered into the queue yet no change in the QSO count since 11/24/2012.  I have contacted ARRL and they told me to wait and the system will catch up.  This isn't a case of "catching up".  Their system is currently processing updates submitted 12/7/2012.  What happened to all my logs submitted prior to 12/7 ?????  It looks to me like some accounts have been permanently "locked out".  If somebody at ARRL reads this post, can you PLEASE do something to help me.  At least give me a straight answer so I know what to do.


Title: RE: LOTW
Post by: W6GX on December 16, 2012, 03:30:50 PM
I believe there are more serious problems with LOTW than are being discussed on the LOTW site.  The last update to my QSO count was 23 days ago in spite of daily uploads to the system.  My daily activity shows the uploads entered into the queue yet no change in the QSO count since 11/24/2012.  I have contacted ARRL and they told me to wait and the system will catch up.  This isn't a case of "catching up".  Their system is currently processing updates submitted 12/7/2012.  What happened to all my logs submitted prior to 12/7 ?????  It looks to me like some accounts have been permanently "locked out".  If somebody at ARRL reads this post, can you PLEASE do something to help me.  At least give me a straight answer so I know what to do.

This is what I see as your last upload.  When you do a most recent QSO query what is the last date of QSO that you see?

Last upload for WA0CRI: 2012-11-28 08:00:25Z

Could you post your account activities?  Below is an example of my account activities.

Date/Time   Via   File Processed   
2012-12-16 02:02:32    Web    LOTWUpload 2012-12-05 213817.TQ8    Result
2012-12-16 02:02:25    Web    LOTWUpload 2012-12-05 213612.TQ8    Result
2012-12-14 06:37:53    Web    LOTWUpload 2012-12-04 083228.TQ8    Result
2012-12-10 05:32:05    Web    LOTWUpload 2012-12-02 004343.TQ8    Result
2012-12-01 12:34:14    Web    LOTWUpload 2012-11-26 142352.TQ8    Result

73,
Jonathan W6GX


Title: RE: LOTW
Post by: N9KX on December 16, 2012, 04:12:32 PM

This is what I see as your last upload.  When you do a most recent QSO query what is the last date of QSO that you see?

Last upload for WA0CRI: 2012-11-28 08:00:25Z

Could you post your account activities?  Below is an example of my account activities.

Date/Time   Via   File Processed   
2012-12-16 02:02:32    Web    LOTWUpload 2012-12-05 213817.TQ8    Result
2012-12-16 02:02:25    Web    LOTWUpload 2012-12-05 213612.TQ8    Result
2012-12-14 06:37:53    Web    LOTWUpload 2012-12-04 083228.TQ8    Result
2012-12-10 05:32:05    Web    LOTWUpload 2012-12-02 004343.TQ8    Result
2012-12-01 12:34:14    Web    LOTWUpload 2012-11-26 142352.TQ8    Result

73,
Jonathan W6GX

I believe the "Find Call" link on the LoTW website shows you the date the last upload was *processed* and no the date it was uploaded.  Look up your own call using it and I think you will see...


Title: RE: LOTW
Post by: AF3Y on December 16, 2012, 04:45:45 PM
Now that my ProIII is in the rig hospital, I am bored, so read a bit of the thread here.

I just wonder if ARRL assumed that some users were going to upload daily, and if that has some bearing
on the overload/backlog problem.  I usually upload every couple weeks at the most and usually its a
bit longer, closer to monthly. I have never seen the need to upload daily, I guess.

Just thinking out loud......

73, Gene AF3Y


Title: RE: LOTW
Post by: N7SMI on December 16, 2012, 06:02:15 PM
I believe the "Find Call" link on the LoTW website shows you the date the last upload was *processed* and no the date it was uploaded.  Look up your own call using it and I think you will see...

I'm not sure what it means. Mine clearly shows the "Last upload for N7SMI" as being Dec. 15, but I know for a fact that my last upload was Dec. 12 or 13. It is either the last date a log was processed, or perhaps the date that an upload actually shows up in the queue, which seems to be a few days after the actual upload. Either way, it's a bit confusing and if it's not accurate would certainly lead to unnecessary re-uploads of log files, which is precisely the thing that is slowing LoTW down so much.


Title: RE: LOTW
Post by: WA0CRI on December 16, 2012, 07:46:34 PM
I believe there are more serious problems with LOTW than are being discussed on the LOTW site.  The last update to my QSO count was 23 days ago in spite of daily uploads to the system.  My daily activity shows the uploads entered into the queue yet no change in the QSO count since 11/24/2012.  I have contacted ARRL and they told me to wait and the system will catch up.  This isn't a case of "catching up".  Their system is currently processing updates submitted 12/7/2012.  What happened to all my logs submitted prior to 12/7 ?????  It looks to me like some accounts have been permanently "locked out".  If somebody at ARRL reads this post, can you PLEASE do something to help me.  At least give me a straight answer so I know what to do.

This is what I see as your last upload.  When you do a most recent QSO query what is the last date of QSO that you see?

Last upload for WA0CRI: 2012-11-28 08:00:25Z

Could you post your account activities?  Below is an example of my account activities.


73,
Jonathan W6GX

Jonathan,
I did another QSO query and the most recent was 2012-11-24  22:37:31.  Below is my account activity.

Date/Time Via File Processed   
2012-12-14 04:10:03 Web wa0cri.tq8 Result
2012-12-10 03:55:27 Web wa0cri.tq8 Result
2012-12-08 14:23:25 Web wa0cri.tq8 Result
2012-12-06 21:42:16 Web wa0cri.tq8 Result
2012-12-05 09:06:23 Web wa0cri.tq8 Result
2012-12-03 21:59:11 Web wa0cri.tq8 Result
2012-12-01 21:44:54 Web wa0cri.tq8 Result
2012-11-28 10:11:08 Web wa0cri.tq8 Result
2012-11-28 08:00:18 Web wa0cri.tq8 Result
2012-11-27 19:50:32 Web wa0cri.tq8 Result
2012-11-26 22:26:20 Web wa0cri.tq8 Result
2012-11-26 02:52:54 Web wa0cri.tq8 Result
2012-11-25 12:06:45 Web wa0cri.tq8

LOTW states they are processing logs submitted 12/7/2012.  Can somebody explain what is happening to my uploads???


Title: RE: LOTW
Post by: AB8MA on December 16, 2012, 08:23:57 PM
Each of the "Result" strings is a link which will give you a Logbook Activity Record of that particular upload.

For instance, if I click "Result" in 2012-12-03 08:42:52    Web    LoTW_AB8MA_20121127.tq8    Result


I get

Quote
2012-12-03 08:42:52 LOTW_QSO: Processing file: msg-22980-1.tq8
2012-12-03 08:42:52 LOTW_QSO: User file: LoTW_AB8MA_20121127.tq8
2012-12-03 08:42:52 LOTW_QSO: Certificate found for AB8MA - UNITED STATES OF AMERICA (291)
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA PT0S 2012-11-22 18:53:00Z 17M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA C6AUM 2012-11-23 02:28:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA 7P8D 2012-11-23 17:49:00Z 10M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA IO5O 2012-11-24 00:41:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA ED1R 2012-11-24 00:51:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA TO7A 2012-11-24 01:07:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA PJ4A 2012-11-24 01:51:00Z 40M DATA
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA EA5RS 2012-11-24 02:11:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA HK1NA 2012-11-24 02:18:00Z 40M CW
2012-12-03 08:42:53 LOTW_QSO: WARNING: QSO duplicates existing record
2012-12-03 08:42:53 LOTW_QSO:    QSO: AB8MA D4C 2012-11-24 01:01:00Z 40M CW
2012-12-03 08:42:54 LOTW_QSO: Successfully processed 22 QSO records in 1.237644 seconds
2012-12-03 08:42:54 LOTW_QSO: 10 records were duplicates
2012-12-03 08:42:54 LOTW_QSO: No errors encountered


Title: RE: LOTW
Post by: N9KX on December 16, 2012, 09:29:13 PM
I believe the "Find Call" link on the LoTW website shows you the date the last upload was *processed* and no the date it was uploaded.  Look up your own call using it and I think you will see...

I'm not sure what it means. Mine clearly shows the "Last upload for N7SMI" as being Dec. 15, but I know for a fact that my last upload was Dec. 12 or 13. It is either the last date a log was processed, or perhaps the date that an upload actually shows up in the queue, which seems to be a few days after the actual upload. Either way, it's a bit confusing and if it's not accurate would certainly lead to unnecessary re-uploads of log files, which is precisely the thing that is slowing LoTW down so much.

I checked the "Find Call" Lotw link for my own call and it yielded a date of my most recent QSL confirmation and not the most recent date of an upload.
   
Quote from: Lotw Find Call lookup link
Here you can query Logbook to find out the last time log data was uploaded for a particular call sign.

Last upload for K9AIM: 2012-12-10 08:34:03Z

(Note: The date shown is when the log was sent to Logbook, not necessarily the date the QSO took place.) 


For the wording to say 'last upload' *is* misleading and should be rewritten in the interest of clarity (I have uploaded anything to LoTW in over a week).


Title: RE: LOTW
Post by: WA0CRI on December 17, 2012, 06:05:05 AM
Each of the "Result" strings is a link which will give you a Logbook Activity Record of that particular upload.


When I check my "Results" for all recent uploads I get the "Processing Aborted" message as shown below:


Date/Time:    2012-12-08 14:23:25
User:    wa0cri
File:    wa0cri.tq8
Messages:   

2012-12-08 14:23:25 LOTW_QSO: Processing file: 20121201035346.14147
2012-12-08 14:23:25 LOTW_QSO: User file: wa0cri.tq8
2012-12-08 14:23:25 LOTW_QSO: Certificate found for WA0CRI - UNITED STATES OF AMERICA (291)
2012-12-08 14:23:47 LOTW_QSO: Processing ABORTED: Log file was previously processed with this result:


Title: RE: LOTW
Post by: W6GX on December 17, 2012, 10:18:42 AM
Each of the "Result" strings is a link which will give you a Logbook Activity Record of that particular upload.


When I check my "Results" for all recent uploads I get the "Processing Aborted" message as shown below:


Date/Time:    2012-12-08 14:23:25
User:    wa0cri
File:    wa0cri.tq8
Messages:   

2012-12-08 14:23:25 LOTW_QSO: Processing file: 20121201035346.14147
2012-12-08 14:23:25 LOTW_QSO: User file: wa0cri.tq8
2012-12-08 14:23:25 LOTW_QSO: Certificate found for WA0CRI - UNITED STATES OF AMERICA (291)
2012-12-08 14:23:47 LOTW_QSO: Processing ABORTED: Log file was previously processed with this result:


Why was the upload process aborted?

73,
Jonathan W6GX


Title: RE: LOTW
Post by: AB8MA on December 17, 2012, 12:55:56 PM
I would check the contents of your wa0cri.adi (or wa0cri.adif) file. Might not be a bad idea to include the date as part of the filenames.


Title: RE: LOTW
Post by: KH6DC on December 17, 2012, 06:10:25 PM
The massive size of the logs on LOTW being stored.  When will it explode because something's gotta give.


Title: RE: LOTW
Post by: N3QE on December 17, 2012, 06:23:34 PM
Each of the "Result" strings is a link which will give you a Logbook Activity Record of that particular upload.


When I check my "Results" for all recent uploads I get the "Processing Aborted" message as shown below:


Date/Time:    2012-12-08 14:23:25
User:    wa0cri
File:    wa0cri.tq8
Messages:   

2012-12-08 14:23:25 LOTW_QSO: Processing file: 20121201035346.14147
2012-12-08 14:23:25 LOTW_QSO: User file: wa0cri.tq8
2012-12-08 14:23:25 LOTW_QSO: Certificate found for WA0CRI - UNITED STATES OF AMERICA (291)
2012-12-08 14:23:47 LOTW_QSO: Processing ABORTED: Log file was previously processed with this result:


Why was the upload process aborted?


The upload process was not aborted.

You misunderstand the difference between "upload" and "process".

Any .tq8 file can be uploaded.

Uploaded files are queued for later processing.

When they are processed, they are compared with previous file uploads.

If the uploaded .tq8 is identical to one already processed, it tells you so, and doesn't process it again. It was already processed. It uses the word "aborted". That word I think was chosen to sound severe, because it doesn't make sense to upload the same file (or same QSO in different file) twice, and they want to discourage everyone from doing that. At the same time, when it happens and it can detect it and reject the file early on and save processing time, all the much the better.

Tim.


Title: RE: LOTW
Post by: KD6KVL on December 17, 2012, 07:25:52 PM
I'm a little concerned.  I've been watching for my upload on 12-7-12 and today the page says its working on uploads after my date, yet my qso's didn't make it in.
I suppose my log may have been lost, no big deal I can re-upload.
But what about the dx operator whos log is lost?
This systems only gift is that it counts as confirmation to ARRL awards.
The powers that be hold hands in the air about how secure it is for confirmations.
I'm giving money to club log, way better interface of information than lotw has been with way less money and trouble.
I don't see how verifing the id of both stations can allow fraud to not occur, dis-honest people will always find a way.
Club log is really showing the way here and the ARRL needs to take the hint.
I'm close to saying to heck with the paper awards, and QST isn't so great that I'm strongly considerring not renewing anymore.

Frank KG6N


Title: RE: LOTW
Post by: NU1O on December 17, 2012, 07:55:21 PM
Club log is really showing the way here and the ARRL needs to take the hint.
I'm close to saying to heck with the paper awards, and QST isn't so great that I'm strongly considerring not renewing anymore.

Frank KG6N

I agree Club Log is running circles around the system used at LoTW but ARRL runs the awards many hams seek such as DXCC, Honor Roll, WAS. etc.  

I am not really a member of ARRL for the purpose of reading QST.  There are only a few articles per year I find worth reading. I'm a member for QSLing, Awards, and the legislative work they do.

My point is there is more to the ARRL than QST but if you feel like quitting, you should quit and state your reason for quitting if you do. It would probably be a good thing if a large number quit over the LoTW issue. The ARRL is a big organization and money - more like a substantial loss of it -- is about the only thing which will grab management's attention to insure LoTW is fixed properly.

73,

Chris/NU1O


Title: RE: LOTW
Post by: AF3Y on December 18, 2012, 07:12:44 AM
GEE........... We REALLY do need some rare DX active....... ;D ;D

73, Gene AF3Y


Title: RE: LOTW
Post by: AA6YQ on December 18, 2012, 07:56:05 AM
Club log is really showing the way here and the ARRL needs to take the hint.
I'm close to saying to heck with the paper awards, and QST isn't so great that I'm strongly considerring not renewing anymore.

Frank KG6N

I agree Club Log is running circles around the system used at LoTW but ARRL runs the awards many hams seek such as DXCC, Honor Roll, WAS. etc.  


Club Log and LotW are "apples and oranges". Club Log is an excellent system, but does not provide (and did not set out to provide) the level of authentication and security the ARRL requires for DXCC awards.

   73,

        Dave, AA6YQ

   



Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 08:48:12 AM
Clublog will match qso's just like LOTW.
The time window is the same as well.
Just because I have to use some crummy security software on my log file does not mean that the qso occurred!  What's to keep someone from coercing with a dx station to make up a qso?

I agree from a performance standpoint, they are "apples and oranges"
I'd love it if once the ARRL has repaired the bottlenecks, to take note of the Clublog log features for keeping track of qso's and countries. 
For instance, why the heck are we limited to 25 qso's at a time to querrie?
It seems like if the LOTW had more friendly features like Clublog, maybe there would be more support from hams and even sponsors if needed.
I fully understand LOTW is the tool of an organization to more easily administer award credit, but what I'm saying is the logic is flawed to believe that because you've confirmed a file came from s specific party matches another's, that it must be "secure and authentic".


Title: RE: LOTW
Post by: AC4RD on December 18, 2012, 09:12:28 AM
What's to keep someone from coercing with a dx station to make up a qso?   ...  I'm saying is the logic is flawed to believe that because you've confirmed a file came from s specific party matches another's, that it must be "secure and authentic".

I'm sure if you have a better idea, one that is easier to implement, better assures authenticity, runs faster, etc., the ARRL would be GLAD to hear your suggestions.


Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 09:24:04 AM
You can never keep out all the fools from being fools.
The reason I said that is to enforce the point that security is not keeping the validity of the qso.
I know of a dx'er with qso's made by phone call to a friend to make the qso.
So if we can't be more secure than the honesty of some crummy folks, why go so far?
Clublog dances circles around LOTW.  They did what the ARRL should have when building lotw.
I know that the Clublog isn't qso credit for an award, but if the ARRL cares what users think, I have several respected DX'ers in my community who feel the same way.
It's a little nauseating to see the attitudes of the arrl reps when addressing these concerns.
Why not admit the other folks made a nice system and work to make lotw more like what the members want.
Frank KG6N


Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 10:02:08 AM
"I'm sure if you have a better idea, one that is easier to implement, better assures authenticity, runs faster, etc., the ARRL would be GLAD to hear your suggestions."

I very respected Dx'er in my area has communicated these opinions to the league and the response has been just short of "too bad".
Frank KG6N


Title: RE: LOTW
Post by: AA6YQ on December 18, 2012, 10:31:21 AM
Clublog will match qso's just like LOTW.

Matching QSOs has nothing to do with authentication or security.

The time window is the same as well.

Implementing a time window has nothing to do with authentication or security.

Just because I have to use some crummy security software on my log file does not mean that the qso occurred!  What's to keep someone from coercing with a dx station to make up a qso?

LotW does not prevent all dishonest means of forging QSOs, but does make it expensive for a user acting alone to forge a QSO.

I'd love it if once the ARRL has repaired the bottlenecks, to take note of the Clublog log features for keeping track of qso's and countries.  

ClubLog's award tracking features are very basic. For example, ClubLog can't highlight spots of DX stations you need for CQ DX Marathon. Logging applications are much better at this; isn't yours?

For instance, why the heck are we limited to 25 qso's at a time to querie?

Why are you doing that manually? Doesn't your logging application automate this for you?

It seems like if the LOTW had more friendly features like Clublog, maybe there would be more support from hams and even sponsors if needed.
I fully understand LOTW is the tool of an organization to more easily administer award credit, but what I'm saying is the logic is flawed to believe that because you've confirmed a file came from s specific party matches another's, that it must be "secure and authentic".

You seem to expect LotW to serve as your logging and award tracking application; that is not its purpose.

LotW is the only means of electronic confirmation that is accepted by the ARRL for its award programs.


Title: RE: LOTW
Post by: AA6YQ on December 18, 2012, 10:40:46 AM
You can never keep out all the fools from being fools.
The reason I said that is to enforce the point that security is not keeping the validity of the qso.
I know of a dx'er with qso's made by phone call to a friend to make the qso.
So if we can't be more secure than the honesty of some crummy folks, why go so far?

ARRL award programs like DXCC, Honor Roll, WAS, and VUCC are popular because they are meaningful. Yes, some ops forge 1.8 Mhz QSLs from 18 Mhz QSLs, and a few get away with it. Yes, some ops bribe DX stations into sending them QSL cards. But neither of these techniques will get you anywhere close to the top of the Honor Roll. Your "We can't provide 100% security, so let's not bother providing any security at all" approach would reduce ARRL award programs to a farce.

Clublog dances circles around LOTW. 

F16s can dance circles around submarines, but we don't abandon submarines. You continue to compare apples and orange, apparently because you don't understand the difference.



Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 10:51:52 AM
David,
Do you represent the ARRL in these statements?
Are you a paid ARRL staff, or a volunteer for this situation?
Why doesn't the ARRL give any note to feed back?
All I see or hear is defensive arguments and excuses.
In your comments it even borders insulting and a "holier than thou" attitude.
Like the critical thinking comment.  
I understand the purpose of lotw, no need to explain.
I am saying that perhaps if the league wants to develop a support amongst hams, maybe look into ways to make it nicer.
How many articles did QST have to convince us how easy and great it is?
How is the feed back handled?
To me, it seems the league treats members opinions on this the way the DXCC program has always been, a thorn in its side.  If it weren't for the revenue, another entity may have taken over awards long ago.
Cq's awards sure are gaining popularity, huh.
WPX award credits on lotw?
Why not make the user interface more user friendly.
Seriously, what's your big objection to that??
How could it hurt to make it a little nicer?
The information and ability is all there, I wouldn't think searching through the qsl/qso query form should have to be so archaic as to limit 25 at a time.  


Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 11:01:44 AM
As far as the the fidelity of the award, I have seen hams make qso's with another hams call and also use way above legal limit power.  The award only has as much value as the operator who did it.  Honor roll is only a matter time with today's technology.  Plenty of ops out there just waiting for expedition to expedition, and if they can't work them, phone a friend.


Title: RE: LOTW
Post by: EI2GLB on December 18, 2012, 11:03:22 AM
I am sure I seen a news release saying that there was 2 hams going to help with LOTW. Has anyone else seen this and can you provide a link to it

Thanks
Trevor
EI2GLB


Title: RE: LOTW
Post by: NK7Z on December 18, 2012, 11:13:24 AM
I am sure I seen a news release saying that there was 2 hams going to help with LOTW. Has anyone else seen this and can you provide a link to it

Thanks
Trevor
EI2GLB
I believe that was the Open tQSL, not LoTW.
73's


Title: RE: LOTW
Post by: KD6KVL on December 18, 2012, 11:14:32 AM
"You seem to expect LotW to serve as your logging and award tracking application; that is not its purpose."
Your kinda reaching here.
I have logging programs with plenty of award tracking.
Lotw has an awards section with a half decent interface to view all the options.
Why not use that format at the query page?

Telling me I don't understand the difference is offensive.

So if it's nothing but a confirmation system, what's your point with the dx marathon spotting??
Club log shows the information in many user configurable and easy to read ways.
This is what I think lotw should adopt.


Title: RE: LOTW
Post by: AA6YQ on December 18, 2012, 11:46:59 AM
Do you represent the ARRL in these statements?
Are you a paid ARRL staff, or a volunteer for this situation?

What I post are exclusively my positions and opinions.

Why doesn't the ARRL give any note to feed back?

That's a question for ARRL management to answer.

All I see or hear is defensive arguments and excuses.

The only thing I've defended in this thread are facts and logic.

In your comments it even borders insulting and a "holier than thou" attitude.
Like the critical thinking comment.  

Re-read the interaction leading up to that comment. It was a perfectly reasonable response to the implication that only machines can avoid jumping to conclusions.

I understand the purpose of lotw, no need to explain.
I am saying that perhaps if the league wants to develop a support amongst hams, maybe look into ways to make it nicer.
How many articles did QST have to convince us how easy and great it is?
How is the feed back handled?
To me, it seems the league treats members opinions on this the way the DXCC program has always been, a thorn in its side.  If it weren't for the revenue, another entity may have taken over awards long ago.
Cq's awards sure are gaining popularity, huh.
WPX award credits on lotw?
Why not make the user interface more user friendly.
Seriously, what's your big objection to that??
How could it hurt to make it a little nicer?

I have criticized LotW's usability since its inception, added functionality to the free DXLab Suite (http://www.dxlabsuite.com) to shield users from LotW's usability problems, provided documentation (http://www.dxlabsuite.com/dxlabwiki/QSLingLotW) to make the use of LotW more clear, and volunteered to reboot the Trusted QSL project from which TQSL and TQSLCert are constructed with improved usability as the primary driver.  Where in any of that do you see an objection to making LotW easier to use?


Title: RE: LOTW
Post by: AA6YQ on December 18, 2012, 12:05:09 PM
"You seem to expect LotW to serve as your logging and award tracking application; that is not its purpose."
Your kinda reaching here.
I have logging programs with plenty of award tracking.

Then why should LotW assign scarce development resources to replicate award tracking functionality that most ops already have in their logging applications?

Telling me I don't understand the difference is offensive.

You continue to conflate LotW with Club Log.

So if it's nothing but a confirmation system, what's your point with the dx marathon spotting?

I was illustrating the point that Club Log provides only basic award tracking.

Club log shows the information in many user configurable and easy to read ways.
This is what I think lotw should adopt.

Great. Go raise the required funding, and convince ARRL management to extend LotW as you suggest.


Title: RE: LOTW
Post by: KY6R on December 18, 2012, 12:17:02 PM
AA6YQ:

Thanks very much for jumping in and doing something to make LOTW a better system!

The ARRL DXCC program is the Gold Standard (for me at least), so LOTW is an important part of that.


Title: RE: LOTW
Post by: AB8MA on December 18, 2012, 03:05:31 PM
I am sure I seen a news release saying that there was 2 hams going to help with LOTW. Has anyone else seen this and can you provide a link to it

Thanks
Trevor
EI2GLB

I this this is what you were referring to:

http://www.arrl.org/news/logbook-of-the-world-web-page-now-features-daily-and-hourly-status-updates

Quote
Rick Murphy, K1MU, and Dave Bernstein, AA6YQ, have been charged with rebooting the Trusted QSL open source project. If you have demonstrably strong C++ development skills that you’re interested in applying toward improving LoTW’s usability and efficiency, please contact Bernstein via e-mail aa6yq(at)ambersoft(dot)com.


Title: RE: LOTW
Post by: N9KX on December 19, 2012, 03:16:11 AM

Re-read the interaction leading up to that comment. It was a perfectly reasonable response to the implication that only machines can avoid jumping to conclusions.


the 'welcome to machine' reply was not in response to a comment about jumping to false conclusions; it was in response to your comment that refraining from false assumptions works better to quell false assumptions than does disclosure.  'welcome to machine' was the name of a song by Pink Floyd released in 1975.

... refraining from false assumptions does sound great on paper ...  :P




Title: RE: LOTW
Post by: AA6YQ on December 19, 2012, 07:58:11 AM

... refraining from false assumptions does sound great on paper ...  :P


and works extremely well in practice.


Title: RE: LOTW
Post by: AC4RD on December 19, 2012, 09:18:32 AM
"I'm sure if you have a better idea, one that is easier to implement, better assures authenticity, runs faster, etc., the ARRL would be GLAD to hear your suggestions."

I very respected Dx'er in my area has communicated these opinions to the league and the response has been just short of "too bad".


What I said was, "If you can design and implement a BETTER system for the ARRL's LOTW, tell them about it."  I didn't say to complain to them because YOU don't like it.  Opinions are like ... well, you know what.  If you like clublog so much, use it.  And if you don't like LOTW, don't use it.  Simple as it gets, isn't it?


Title: RE: LOTW
Post by: N9KX on December 19, 2012, 09:57:14 AM

... refraining from false assumptions does sound great on paper ...  :P


and works extremely well in practice.

but perfection is elusive and creativity demands a willingness to play with false assumptions
yesterday's science contained false assumptions; and tomorrow's science will find the same problem with today's.
i suppose debunkers like yourself do play a useful role...



Title: RE: LOTW
Post by: AA6YQ on December 19, 2012, 01:47:30 PM

... refraining from false assumptions does sound great on paper ...  :P


and works extremely well in practice.

but perfection is elusive and creativity demands a willingness to play with false assumptions yesterday's science contained false assumptions; and tomorrow's science will find the same problem with today's. i suppose debunkers like yourself do play a useful role...

We weren't discussing protein folding or string theory or any other topic where thinking outside the box would be beneficial. Your misinterpretation of a statement from the ARRL's CEO led you to the incorrect conclusion that there's a team of developers dedicated to LotW, when in fact only a fraction of one person is available for LotW development and maintenance. There's no constructive justification for this particular false assumption.

The first rule of escaping from a deep hole is to stop digging.


Title: RE: LOTW
Post by: AB8MA on December 19, 2012, 02:33:31 PM
Sure hope the magic works.


Title: RE: LOTW
Post by: AF3Y on December 19, 2012, 02:53:48 PM
Scotty....... "Beam me up"..... Please?

73, Gene AF3Y


Title: RE: LOTW
Post by: HS0ZIB on December 19, 2012, 04:54:10 PM
Quote
... when in fact only a fraction of one person is available for LoTW development and maintenance

Well, if that is true, then the reason for all these problems is a lot clearer, at least to me....

Simon


Title: RE: LOTW
Post by: N9KX on December 19, 2012, 05:29:13 PM
Your misinterpretation of a statement from the ARRL's CEO led you to the incorrect conclusion that there's a team of developers dedicated to LotW, when in fact only a fraction of one person is available for LotW development and maintenance.

Actually I never assumed a team was dedicated to LoTW, just that there was an IT team available to at least weigh in on the LoTW issues if and when they arise -- so pot meet kettle. 

Quote from: AA6YQ
There's no constructive justification for this particular false assumption.

your opinion is noted


Title: RE: LOTW
Post by: AA6YQ on December 19, 2012, 06:33:05 PM
Your misinterpretation of a statement from the ARRL's CEO led you to the incorrect conclusion that there's a team of developers dedicated to LotW, when in fact only a fraction of one person is available for LotW development and maintenance.

Actually I never assumed a team was dedicated to LoTW, just that there was an IT team available to at least weigh in on the LoTW issues if and when they arise -- so pot meet kettle. 

Your exact words were no, I had taken K1ZZ's mention of "the present IT staff" below to mean there was a team available to spend some of their time on that..." .  I should have characterized your incorrect conclusion as "there's a team of developers available to LotW, when in fact only a fraction of one person is available for LotW development and maintenance."


Quote from: AA6YQ
There's no constructive justification for this particular false assumption.

your opinion is noted
[/quote]


Title: RE: LOTW
Post by: N6ORB on December 19, 2012, 08:15:25 PM
OK, guys. How about taking this to e-mail. Since there's no new information here, the two-person dialogue has become boring.

Dave, N6ORB


Title: RE: LOTW
Post by: N9KX on December 19, 2012, 08:52:09 PM
sorry you had to endure all that.  73