Call Search
     

New to Ham Radio?
My Profile

Community
Articles
Forums
News
Reviews
Friends Remembered
Strays
Survey Question

Operating
Contesting
DX Cluster Spots
Propagation

Resources
Calendar
Classifieds
Ham Exams
Ham Links
List Archives
News Articles
Product Reviews
QSL Managers

Site Info
eHam Help (FAQ)
Support the site
The eHam Team
Advertising Info
Vision Statement
About eHam.net

   Home   Help Search  
Pages: Prev 1 ... 16 17 18 19 20 [21] 22 23 24 Next   Go Down
  Print  
Author Topic: LOTW  (Read 47004 times)
N4KC
Member

Posts: 282


WWW

Ignore
« Reply #300 on: December 14, 2012, 08:18:24 AM »

OK, you DB AND C++ guys, you ready to put your skills where your mouths are?  From the ARRL web site:

New hardware that will improve LoTW’s throughput is on order and is expected to be running in six to eight weeks.

Rick Murphy, K1MU, and Dave Bernstein, AA6YQ, have been charged with rebooting the Trusted QSL open source project. If you have demonstrably strong C++ development skills that you’re interested in applying toward improving LoTW’s usability and efficiency, please contact Bernstein via e-mail aa6yq(at)ambersoft(dot)com.

We appreciate the user community’s patience while we while work to bring LoTW’s performance to an acceptable level.


73,

Don N4KC
www.n4kc.com
www.donkeith.com
(Author of the new book RIDING THE SHORTWAVES:
EXPLORING THE MAGIC OF AMATEUR RADIO)
 
 
Logged
AC4RD
Member

Posts: 1236




Ignore
« Reply #301 on: December 14, 2012, 08:36:58 AM »


Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.
Logged
N2RJ
Member

Posts: 1133




Ignore
« Reply #302 on: December 14, 2012, 08:42:06 AM »


Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.


he has a chip on his shoulder about LoTW because *gasp* you have to actually show your ID to be authenticated in the system and he doesn't like that.
Logged
N4CR
Member

Posts: 1650




Ignore
« Reply #303 on: December 15, 2012, 01:15:00 PM »


I work for a fortune 100 company as a software and database specialist.

If I heard a system was doing what is stated above, there is no way I'd start by looking for a problem in hardware. Without being able to do hands on analysis, I'd guess this system is doing table scans on reads or writes. Once the size of the database exceeds memory constraints, it becomes disk channel bound. Which drops the speed by a factor of about 1000. A primary indication of this is that the CPU load is low and the disk channel (read/write light) is balls to the wall.

If it is this, the temporary cure is to add ram to get all of the database in memory again, the real fix is to find out why the table scans are occurring and fix the design flaw.

But this is all wild speculation from hallway talk and that's a HUGE if.

From experience, I've found that throwing hardware at a problem should be the last step after every advantage can be wrung out of the software.

any chance you will be in CT anytime soon and are feeling charitable?    Cheesy

seriously though, how much time would you expect a good analysis of the software to take, and how expensive is that?

If it's MsSQL I could do the work remotely.

As far as database tuning goes, my first pass would be to turn on advanced logging, take a half hour of log and then run that log through the tuning wizard. It not only tells you what to fix but it then tells you how much time would be chopped by the change. Tuning on newer versions of MsSQL also revolves around data statistics and often the statistics never get analyzed or added.

Also, a nightly tuning run of optimizing indexes and updating statistics should be put in place.

The second pass would be to inspect the code for things that can be optimized in how the code is doing it's work. This is harder, slower work and some of the time it's unnecessary after the tuning wizard does it's thing because the bottleneck turns out to be internal to SQL's operations.

I don't mind volunteering my time. But I'm only expert at optimizing Microsoft SQL. I have no idea what the database architecture of LoTW is. If they want my help, they can contact me at my QRZ email address.

Timewise, a index tuning wizard run for MsSQL could take as little as a few hours and can provide huge benefits. Especially if the incoming data always look the same, which I expect on this system, it's pretty much all the same in data structure and process. Probably not a lot of variety.

Analyzing a database for changes that involve moving from first normal form to third normal form (which I call the minimum for data design) could take a total rewrite. Or it might be that a few simple changes could to a lot.

Creating stored procedures of queries that are currently in outboard code can make huge differences because when you use external software to perform queries, it compiles the query each time. If you use a stored procedure, it compiles the stored proc one time and from then on it runs the compled code. Depending on the complexity of the code, it could take a few days or a few months. I would expect some immediate gains in the first day, though.
Logged

73 de N4CR, Phil

We are Coulomb of Borg. Resistance is futile. Voltage, on the other hand, has potential.
K9AIM
Member

Posts: 914




Ignore
« Reply #304 on: December 15, 2012, 01:49:05 PM »

Thanks Phil for your input -- and for volunteering to help if the ARRL wishes to contact you.

Have you by chance had time to review these other threads on the issue?

http://www.eham.net/ehamforum/smf/index.php/topic,86920.0.html

http://www.eham.net/ehamforum/smf/index.php/topic,86898.0.html

Logged
N4CR
Member

Posts: 1650




Ignore
« Reply #305 on: December 15, 2012, 08:04:43 PM »

Yes, I saw them. And it would not change a thing about how I went about solving the problems.

This is all beating a phantom horse. We don't even have a clue what operating system and database is in place. This is all conjecture and that seldom solves problems.
Logged

73 de N4CR, Phil

We are Coulomb of Borg. Resistance is futile. Voltage, on the other hand, has potential.
W6GX
Member

Posts: 2286




Ignore
« Reply #306 on: December 15, 2012, 10:31:56 PM »

As you know LoTW had a bug where some of the uploaded logs got overwritten.  I decided to re-upload all of my QSOs that weren't confirmed in LoTW.  I already knew that the bug affected me as I found several QSOs were not found in LoTW despite the fact they were uploaded to LoTW.  I didn't do a detailed log comparison between LoTW and the HRD Logbook as that exercise would be tedious.  Anyhow I received 15 new QSLs after I re-uploaded about 2,200 old QSOs to LoTW.  According to LoTW there were 16 QSOs from the upload that were neither replacements nor duplicates.  So I assume these 16 QSOs were the missing ones due to the bug.  Below is what it reported on the re-upload.  I recommend anyone to re-upload all of their unconfirmed QSOs to LoTW.  This way you will be sure that all of your QSOs are properly added in LoTW.

2012-12-16 02:05:07 LOTW_QSO: Successfully processed 2177 QSO records in 154.827350 seconds
2012-12-16 02:05:07 LOTW_QSO: 1483 records were replacements
2012-12-16 02:05:07 LOTW_QSO: 678 records were duplicates
2012-12-16 02:05:07 LOTW_QSO: 15 QSL records entered
2012-12-16 02:05:07 LOTW_QSO: No errors encountered

73,
Jonathan W6GX
Logged
AB8MA
Member

Posts: 732




Ignore
« Reply #307 on: December 16, 2012, 05:56:37 AM »

What's the difference between a replacement and a duplicate?
Logged
N4CR
Member

Posts: 1650




Ignore
« Reply #308 on: December 16, 2012, 09:50:37 AM »

What's the difference between a replacement and a duplicate?

A duplicate, all fields are identical.

A replacement is where some field got updated, but it's still the same call at the same time. Like you update the state or mode or band.
Logged

73 de N4CR, Phil

We are Coulomb of Borg. Resistance is futile. Voltage, on the other hand, has potential.
W6GX
Member

Posts: 2286




Ignore
« Reply #309 on: December 16, 2012, 09:54:18 AM »

What's the difference between a replacement and a duplicate?

A duplicate, all fields are identical.

A replacement is where some field got updated, but it's still the same call at the same time. Like you update the state or mode or band.

That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

73,
Jonathan W6GX
Logged
N3QE
Member

Posts: 2024




Ignore
« Reply #310 on: December 16, 2012, 10:15:24 AM »

That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

You didn't think you made any changes, but something made a change. For example a change in the frequency field (e.g. 14 is different than 14.000 is different than 14.036 is different than 14.03600 is different than 14.03642), or a different grid square (e.g. FM19 and FM19ka are different), or just rounding off the time (e.g. 1:34:12 is different than 1:34:00) all seem to cause replacements. Some of these happen when importing log files between differnent log programs (e.g. rounding off frequency) and others can happen at TrustedQSL signing time (e.g. grid square). Some log file conversions (e.g. ADI to Cabrillo then back to ADI) are pretty much guaranteed to round off times and frequencies. Some log programs will needlessly round off times and frequencies or change the precision of the recorded numbers even when just importing and then re-exporting ADI.

Re-uploading thousands of QSO's needlessly is one reason why LOTW is so far behind. At first it was just the onslaught of November contest logs. But right now I think we're seeing a snowball effect, somebody sees the log they submitted a few days ago isn't processed yet, so they "fix it good" by re-uploading every QSO they've ever had, then when that's not processed yet the export and upload yet again, bringing LOTW even further behind. Most logging programs let you create the ADIF based on date range. If you think a file from the past hadn't been processed, go to "Your Account", then "Your Activity", and if something from longer than 2 weeks wasn't processed, then just upload the same (signed) .tq8 file again. No need to re-export. If it was uploaded in the past week and a half... it's probably in the queue. Check http://www.arrl.org/logbook-queue-status
« Last Edit: December 16, 2012, 10:21:20 AM by N3QE » Logged
W6GX
Member

Posts: 2286




Ignore
« Reply #311 on: December 16, 2012, 11:05:34 AM »

That's weird.  I didn't make any changes to the 2,200 or so records I re-uploaded to LoTW.  I was expecting 2,200 duplicates and none of them should be replacements.

You didn't think you made any changes, but something made a change. For example a change in the frequency field (e.g. 14 is different than 14.000 is different than 14.036 is different than 14.03600 is different than 14.03642), or a different grid square (e.g. FM19 and FM19ka are different), or just rounding off the time (e.g. 1:34:12 is different than 1:34:00) all seem to cause replacements. Some of these happen when importing log files between differnent log programs (e.g. rounding off frequency) and others can happen at TrustedQSL signing time (e.g. grid square). Some log file conversions (e.g. ADI to Cabrillo then back to ADI) are pretty much guaranteed to round off times and frequencies. Some log programs will needlessly round off times and frequencies or change the precision of the recorded numbers even when just importing and then re-exporting ADI.

Re-uploading thousands of QSO's needlessly is one reason why LOTW is so far behind. At first it was just the onslaught of November contest logs. But right now I think we're seeing a snowball effect, somebody sees the log they submitted a few days ago isn't processed yet, so they "fix it good" by re-uploading every QSO they've ever had, then when that's not processed yet the export and upload yet again, bringing LOTW even further behind. Most logging programs let you create the ADIF based on date range. If you think a file from the past hadn't been processed, go to "Your Account", then "Your Activity", and if something from longer than 2 weeks wasn't processed, then just upload the same (signed) .tq8 file again. No need to re-export. If it was uploaded in the past week and a half... it's probably in the queue. Check http://www.arrl.org/logbook-queue-status

Thanks for the explanation.

I use HRD and it tells me which QSOs have been uploaded to LoTW.  There's no ADIF file that I need to create manually.  I never re-upload QSOs unless I have to, as in the case I mentioned earlier in my post.  As for the increasing backlog of unprocessed logs, that could be mostly due to more QSOs made per person and also new comers signing up.  For sure some of it is due to the LoTW bug.  However trying to pinpoint the exact cause is no more than just speculation.

73,
Jonathan W6GX
Logged
EI2GLB
Member

Posts: 484




Ignore
« Reply #312 on: December 16, 2012, 02:40:35 PM »

Are you saying I have a chip then you are very much mistaken, I am a Hugh supporter of LOTW and encourage people to use it daily,


I might have said its not easy for us DX to get signed up but I have no problem with the level of id the require,

What I have a problem with it there lack of intrest in getting more users and spending money on improving the system,

LOTW has saved me thousands of $$ in direct QSL's but I have spent hundreds on awards,

Trevor
EI2GLB



Clublog is ran as a hobby LOTW is a money making racket


Wrong, Trevor.  There is NO fee required to use LOTW.  There are fees for AWARDS if you use LOTW for that--but there have always been fees for awards, there's no change in that.  And there is NO fee required just to use LOTW.  Though perhaps there should be.


he has a chip on his shoulder about LoTW because *gasp* you have to actually show your ID to be authenticated in the system and he doesn't like that.
Logged
WA0CRI
Member

Posts: 34




Ignore
« Reply #313 on: December 16, 2012, 02:58:25 PM »

I believe there are more serious problems with LOTW than are being discussed on the LOTW site.  The last update to my QSO count was 23 days ago in spite of daily uploads to the system.  My daily activity shows the uploads entered into the queue yet no change in the QSO count since 11/24/2012.  I have contacted ARRL and they told me to wait and the system will catch up.  This isn't a case of "catching up".  Their system is currently processing updates submitted 12/7/2012.  What happened to all my logs submitted prior to 12/7 Huh??  It looks to me like some accounts have been permanently "locked out".  If somebody at ARRL reads this post, can you PLEASE do something to help me.  At least give me a straight answer so I know what to do.
Logged
W6GX
Member

Posts: 2286




Ignore
« Reply #314 on: December 16, 2012, 03:30:50 PM »

I believe there are more serious problems with LOTW than are being discussed on the LOTW site.  The last update to my QSO count was 23 days ago in spite of daily uploads to the system.  My daily activity shows the uploads entered into the queue yet no change in the QSO count since 11/24/2012.  I have contacted ARRL and they told me to wait and the system will catch up.  This isn't a case of "catching up".  Their system is currently processing updates submitted 12/7/2012.  What happened to all my logs submitted prior to 12/7 Huh??  It looks to me like some accounts have been permanently "locked out".  If somebody at ARRL reads this post, can you PLEASE do something to help me.  At least give me a straight answer so I know what to do.

This is what I see as your last upload.  When you do a most recent QSO query what is the last date of QSO that you see?

Last upload for WA0CRI: 2012-11-28 08:00:25Z

Could you post your account activities?  Below is an example of my account activities.

Date/Time   Via   File Processed   
2012-12-16 02:02:32    Web    LOTWUpload 2012-12-05 213817.TQ8    Result
2012-12-16 02:02:25    Web    LOTWUpload 2012-12-05 213612.TQ8    Result
2012-12-14 06:37:53    Web    LOTWUpload 2012-12-04 083228.TQ8    Result
2012-12-10 05:32:05    Web    LOTWUpload 2012-12-02 004343.TQ8    Result
2012-12-01 12:34:14    Web    LOTWUpload 2012-11-26 142352.TQ8    Result

73,
Jonathan W6GX
Logged
Pages: Prev 1 ... 16 17 18 19 20 [21] 22 23 24 Next   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!