Plan A

Digital HF

In many ways, the most awe-inspiring resource that amateurs interested in digital communications have at our disposal is access to the HF bands. The long distances and wide areas covered by even relatively low powered HF signals allow amateurs to build, organize and operate an independent digital communications network on a global scale, something no other group of private individuals can reasonably expect to do. We are a uniquely privileged group of people, in this respect.

At this time, early in the new century, amateurs are only beginning to realize some of the great potential inherent in digital HF communications. Hams have enthusiastically taken to the new soundcard digital modes such as psk31 that allow great DX ragchewing at low power with modest antenna systems, all within a fraction of the bandwidth previously thought to be necessary for such communications. Alternately, the new wide modes such as PACTOR III and Q15x25 mode allow data throughput that was unheard of, only a few years ago. Despite these advances, we are only just beginning to take advantage of the great potential of the HF bands. Digital HF is still very new, and we have a long path of discovery and innovation ahead of us.

Point to Point

Amateurs have developed point-to-point networking technique on HF, taking advantage of the great distances it is possible to regularly and reliably communicate there. A network of Packet, PACTOR, and CLOVER stations have been providing global-scale communications services to amateurs for close to two decades now, with RTTY and AMTOR mail systems going even further back than that. It wasn't until Packet radio and the packet BBS came on the scene though, that an organized amateur radio digital network emerged as a global entity. Perhaps it is for this reason that this anarchic network incorporating several modes besides packet is still generally referred to as "the packet network".

Amateur Radio BBS stations use a simple but effective hierarchical addressing system that allows them to route messages to other Amateur BBS's anywhere on the planet. This system has given us a reliable and useful global messaging system, with limited bulletin capability. The bulletins are a problem in this point-to-point network, since they are addressed to "All" within a large, possibly even global distribution. The HF BBS forwarding system is great for sending a targeted message to a particular destination, often doing so faster than the Internet. For wide-area distribution of bulletin traffic though, the point-to-point networking we employ on HF is less than ideal. It doesn't take too many bulletins at all to slow the entire network down to a near-halt, eventually affecting its ability to do its primary tasks; Moving targeted emergency communications, NTS and personal messages.

Many Packet BBS stations today have responded to the bulletin glut by reducing the number of bulletin distributions that they will accept or forward. Almost all HF packet BBS stations in the U.S. forward ARRL and AMSAT bulletins, if nothing else. Most forward the @USA and @ALLUS distributions as well, plus statewide bulletins. The global distribution @WW is most often passed up, simply because it is too much data to squeeze through the pipe, and trying to do so tends to gum up the works.

This situation does not hold true with the wider modes like CLOVER, PACTOR II and PACTOR III. Through a combination of grabbing more bandwidth with a wider signal and utilizing clever data squeezing technique, these systems generally have enough through-put to handle all the available traffic today, including the @WW distribution. The problem with wide-bandwidth systems of course is that you cannot put very many of them on the air at one time, and this limits the number of locations that can be served by wide-bandwidth systems. They have to be used judiciously, with moderation.

The HF digital network has greater capability and capacity than in past years, but at this writing has not fully recovered from the slump it experienced in the late 1980's and so does not pass nearly as much traffic as it once did. Much of the original integrity of the network on HF has been regained, but the VHF/UHF networks that many of these HF BBS stations serviced are not as active as in years past.

The HF Digital Networking Conundrum

No coin has only one side, and the case of HF radio communication is no exception to this rule. The great distance and coverage area of HF transmissions is balanced by the reduced "recycle-ability" and lower data transmission rates on HF.

If you transmit 50 watts on 440 MHZ, chances are that hams 100 miles away will never hear you. They can "re-cycle" the same frequency without either of you being bothered. If you transmit with 50 watts on 20 meters though, you can wind up being heard almost anywhere in the world. Because of this reduced "recycle-ability" of HF signals, only so many stations can be on the air at one time, especially on a wide-coverage band like 20 meters, where you can literally end up being heard anywhere. So we have the capability to cover wide areas and long distances on HF, but the number of stations doing so and the amount of bandwidth they take up must by necessity be limited.

From a digital networking standpoint, there's much more to it... Consider the following:

---------------------------------
Distance     / Frequency 

Long haul       20 / 30 meters, sporadically 15 / 10. 
Medium          40 / 80 meters.
Short           VHF/UHF and higher. 
---------------------------------


---------------------------------
Baud Rate    / Frequency 

300 baud        12 - 160 meters 
1200 baud       10 meters 
9600 baud       VHF
1 MB            UHF and higher
--------------------------------- 

Anybody who is familiar with point-to-point networking practice will tell you that the long-haul "backbone" links within the network should be much faster and of higher capacity than the short-range, local "user access" links. If user access is at 1.2kb for example, and you expect to host up to eight users on the long-haul "backbone" link at one time, then 9.6kb will be a minimum "backbone" speed if you wish to maintain what is known as "transparency", where the 1.2kb users never detect a network-related slowdown.

Transparency is what you shoot for, as a point-to-point networker. If users don't know that you are there, then viola! You get no complaints. The greater the ratio between backbone and user-access speed, the greater your level of transparency, and the greater peak number of users and applications your network can support at one time.

Perhaps you've guessed the big conundrum by now... Our long-haul, "backbone" frequencies on HF are much, much slower than the VHF/UHF short-range "user access" frequencies! That is the direct opposite of the desired situation for point-to-point networking, putting transparency far beyond our grasp. This represents the core challenge of digital amateur radio networking. - The stone-wall that will not go away.

This is a challenge that terrestrial (wire/fiber-optic) networkers do not know how to overcome because in their experience, they can simply hook up a higher capacity cable for the backbone. We can't do that for any price, and so must consider network structure and organization that would be completely nonsensical from the point of view of a wired system networker. These fundamental differences between digital ham radio and terrestrial wired systems are why we end up being so dissatisfied with the results we get, when we model our efforts after conventional digital networks such as FidoNet or the Internet.

What this means is that we must develop a networking system of our own, and not put ourselves further into the trap of trying to emulate the performance, capability or methods of terrestrial networks that run over a wire or cable. We can reasonably expect the digital ham radio net to perform better than wired networks - but not at the same tasks, and not by doing things in the same way.

Instead of dissing ourselves because we cannot "compete with the Internet", we must look for and develop the strong points to be found in working with radio. It is a wonderful challenge and opportunity for amateurs; a lot more fun and far more productive than to continue trying to fit our RF peg into the Wire hole.

Suggestion: Quit modeling our RF network exclusively after existing wired networks.

What we as amateurs are doing is singular and new. We have the privilege of doing something unique. Hams are the pioneers here, the innovators, because no other group of private individuals has access to so many slices of prime spectrum as we do. The ability to interface with establishment (wired) networks should of course be retained to facilitate emergency communications, but internally our network should be designed by hams to reflect our network's singular makeup and unique operating environment. To do otherwise can only lead us to more disappointment, and more poor performance.

Imagine trying to operate a boat on the water the same way you would a car on the road system... Stoplights, turn lanes, slam on the brakes, and don't bump over the curb! It would be pretty ridiculous, and is directly comparable to modeling our global RF network on any of the wired networks. They have the roads to follow, but we must direct ourselves over the waters, where there are no curbs, no stoplights and most significantly, no roads! ( High-capacity, long-haul backbone links. )

It should be obvious to all that what we are doing is fundamentally different from any other network in existence because our resources, regulation, our reason for being, and our organizational structure is totally different from that of any other case.

It was natural for amateurs to look to professional networkers from the telecommunications and I.T. industries to advise us on building a digital network of our own. There was no way for us to understand at that time that our needs and resources would be so different from those related to the existing networks. We began to find out though, when professionals associated with wired networks only had so much to contribute to our cause before they ran across a set of parameters they had no idea how to handle.

At least part of the frustration these experts experience stems from the basic unfairness of the situation that they have been put into. Put on the spot as professional networkers to help design a Ham Radio digital network, they are embarrassed and frustrated when they discover that they cannot "deliver" because Ham Radio's long-haul "backbone" links will always be of lower capacity and slower speed than user access, the direct opposite of the parameters they have been trained to expect and to work with.

We have been unfair to these hams, and we really ought to let them off the hook. They could not do everything we asked, but still we have learned a lot from experts in wired systems, and we owe them our most sincere thanks, along with an apology for putting them on the spot like we did. Many of these individuals became so frustrated that they soured on the hobby, and understandably so. It was an honest mistake that we made though, and it's easy to see how we ended up making it... Sorry, guys!

The Ham Radio Perspective

Once we get past the expectations and perceived needs of wired networking and start to look at amateur radio digital networking as a new thing, unique in the world, we are suddenly freed of several unreasonable burdens. Simultaneously we discover opportunities that were not apparent to us before. Instead of attempting to stretch and squeeze our resources (assets and expectations) to fit the procrustean bed of wired networking's architecture and protocols, we can now take a look at what we have, determine what we want, and develop a digital communications network that perfectly fits our unique needs and circumstances.

As has always been the case, amateurs serve themselves and the public best when we innovate, not being content to merely emulate. This is especially so when you consider amateur radio's unique resources and structure, which sets us apart from any other communications entity. Amateurs enjoy freedoms, abilities, resources and responsibilities unique in the world today. This is our privilege as hams, and to approach the unique task of building a global amateur radio digital network is one of many unique honors the amateur service offers to its members.

Amateur radio is justly famous for its ability to draw together individuals from around the world into a truly global fraternity of service, advancement and amity. The amateur service adopted this fine brace of attitudes intact from the experimenters whose pioneering work with radio preceded the organization, and this fact is reflected by the many great activities that amateurs engage in today, alternately in cooperation and/or in competition so as to encourage advancement of the art.

Amateurs who participate in the ham radio digital network have special reason to be proud, as their work represents a new watermark in cooperation and fellowship among amateurs on an international scale. Working with little guidance while going where none had gone before, very often upon "shoestring" resources, amateurs around the world responded to the Packet revolution in the 1980's by building an independent amateur radio digital network with a global footprint. Still active and advancing today, this network has enormous potential that we have yet to even partially develop, employ and enjoy.

What We Have

The digital amateur radio network as a global entity today is given that definition almost solely through the service provided by BBS SYSOPs, forwarding messages on HF frequencies as part of a global point-to-point message forwarding network. The HF component of that network uses several modes and protocols including Packet, CLOVER, PACTOR, and others. These hams are pioneers in an aspect of amateur radio that has a limitless future of service and advancement, that is more than likely to embody the core of the worldwide amateur service in the near future and for many years to come.

Incremental advances in throughput have been accomplished for our HF operations, but always at the price of increased bandwidth. In practical terms, there is a speed limit on the low bands, and assigning ever-wider chunks of our spectrum for digital use is not a viable answer to the challenge we enjoy as ham radio digital networkers. Instead of intruding upon other hams in order to put an ineffective band-aid upon our lack of a high-capacity backbone on HF, we should make more effective, efficient use of what we have. We have a lot, in fact, that we simply are not using at all, and this applies every bit as much to HF as it does to the VHF and higher frequencies.

What We Have Overlooked

To see what we have with HF that we have not been using, we only need to take a look at what is special about radio, and what is special about HF when compared to the higher frequencies. The special things I remember hearing about HF were the long range and wide coverage area possible with HF signals.

What we have today is a point-to-point network modeled after wired network architecture, which takes good advantage of the long range that HF signals can offer. This network has given us a lot and will always be vital to our overall integrity, but it has also fallen short of our needs in some areas and needs more help than VHF/UHF networks can reasonably be expected to provide.

What we have ignored and not used is the other positive attribute inherent in HF communications; the large area that HF signals can effectively cover. This is much like having a floodlight and a laser pointer in each hand, and trying to navigate in the dark with just the pointer.

Happily enough, we already use the protocol that will unleash the true power of HF digital communications, but only AMSAT routinely utilizes this communications protocol today, where one transmitting station serves many receiving amateur stations simultaneously. This communications protocol is called MULTICAST, and its use by amateurs was pioneered by AMSAT.

Many hams are familiar with the National Weather service's EMWIN transmissions, which use a similar multicast protocol for terrestrial digital communications over a local area.

A terrestrial multicast package for hams, RadioMirror, was developed by John Hansen W2FS but as with the other multicast systems mentioned above, RadioMirror was designed for use on VHF and UHF bands, not for the conditions to be encountered on HF.

Development of multicast server and client software specifically designed for use on HF would be a good example of how we can fill the functional gaps left by point to point networking. By happy coincidence, point-to-point networks return the favor, filling gaps in what multicast can accomplish. Working together, the two can provide us with capability on HF that is undreamed-of today. It has been undreamed-of because we have been too distracted by the wired-network model, spending our time trying to navigate in the dark with a laser pointer instead of just turning on the floodlight that was there for us all along.

To demonstrate the natural advantage of multicast to a group of people, write the words "Point to point networking" on a piece of paper, and have the folks pass it around hand to hand until everybody has gotten the message.

Then use a projector (or hold up a large sign) to place the word "Multicast" above the podium, where everybody can "receive" it simultaneously. The larger the number of message recipients, the more obvious the speed advantage of multicast over point-to-point, just as it is in the real world, on the air.

This enormous speed advantage of distributing information simultaneously to an unlimited number of recipients located within a wide coverage area has of course been available to us as hams all along, but we have never thought to develop it because it did not fit in with the wired-network concepts we have been modeling our digital amateur radio networks after. It is time now that we started looking "beyond the wire", utilizing the built-in advantages inherent in radio as compared to wired communications.

A single HF multicast transmitter can serve an unlimited number of client (receiving) stations, and those receiving stations only need a shortwave receiver and a computer with a soundcard to receive that data. Even a simple kit receiver would do the job, and SWL enthusiasts as well as hams would enjoy the educational material and information about ham radio that our multicasts will contain. Any type of file may be sent via multicast, but HTML and text will long be the mainstays because they pack so much information within relatively small files, and because most messaging is text or HTML based.

* Multicast directly addresses one of HF's greatest drawbacks (low recycle-ability) by allowing one transmitter to distribute data to an unlimited number of receiving stations.

* Multicast directly addresses one of HF's greatest assets (wide area coverage) that has previously been overlooked and so has not been effectively utilized by amateurs.

Wired networking technique could only take us so far with radios... It is time for us to move on now, and develop amateur radio networking technique.

How Multicast Works

Multicast stations act as a file server/client system. RadioMirror, the terrestrial multicast server/client package can send individual files, or entire directories complete with sub-directories. It can send files/directories once, or include them in its continuous transmission of data. A mirror image of this directory structure and its files is reproduced on many client station's hard-drives simultaneously.

On the RadioMirror server's end, the files/directories are sent after being cut up into one kilobyte "data blocks". Each data block has a checksum so the client software will be able to check and see if it has received that data block with no errors. The data is cut up into blocks like this to facilitate error correction. They are cut up into one kilobyte blocks because RadioMirror is designed to work with a KISS TNC, and 1 kilobyte was about as big a packet as could be managed.

Once the server has sent all of the data it has queued, it simply starts back at the beginning again and keeps right on going continuously. The receiving (client) stations that have incomplete files due to corrupted or incompletely received data blocks then get a chance to fill in those gaps every time the data is re-sent. Once a file has been received in its entirety, RadioMirror puts the file on the client's hard drive and that file will not change after that unless the server sends a newer version.

You can have files like "Today's News" that would change and be updated every day, and you would have other files that contained standardized information that only updated every once in a while; FAQ's, technical data, web addresses and so on.

One multicast transmitter can serve an unlimited number of client stations simultaneously. No other digital system for HF is so efficient with spectrum, as the others are all one-on-one communications systems, point-to-point. To get anything done with them over a significant area, dozens of transmitting stations must be on the air at once. With multicast, dozens of receiving stations are utilized instead to accomplish this, a much more sensible and responsible approach. Multicast has an enormous speed advantage over traditional point-to-point systems because of this, even over those that operate at much higher baud rates. - And it does all this while taking up a lot less spectrum.

How to Use Multicast on HF

After attempting to utilize RadioMirror on HF, I found that it was best to use RadioMirror as intended, on the VHF and UHF bands. On HF, several drawbacks immediately became apparent that kept it from being useful there. This is in no way a criticism of RadioMirror, a fine system that was never intended to be used on HF.

* One problem was that I had to power down my 100w transceiver to 15w while transmitting multicast, which is effectively continuous-duty. Otherwise I could be sure that my rig would smoke itself after a short while. Since weak, long packet signals are the very hardest ones to reliably copy on HF, I could see a real problem there. - It had me thinking about buying a 1kw amplifier, then tuning it down to 70w, something I could not expect a significant number of other hams to do. If the multicast server couldn't put out a strong, clear signal, then its efficiency goes rapidly down as its copyable RF "footprint" shrinks.

* Another problem was the long time between updates (data transmission cycles) at 300 baud, the fastest data rate for packet on HF. The lower data rate itself was not a big issue, but the greatly expanded delay in sending updates was. The expanded delay between updates, coupled with the much greater number of updates likely to be needed on HF did not look like a good combination.

* The third major drawback I saw was that RadioMirror required those wanting to receive the data to have a KISS TNC. To get a truly large number of receiving stations, a soundcard solution would be necessary.

For years I mulled this around while the solution stared me in the face. - Then one day it dawned on me!

If we used multiple psk streams instead of Packet, all three of the drawbacks listed above would simultaneously be addressed:

* PSK works just fine at low power levels. My first contact with a Japanese station, I remembered, was on a minimal antenna utilizing 15 watts of power while operating PSK31. I also remembered working QRP psk31 stations on 30 meters. A psk "usable footprint" would be much larger at 15 watts than one for RadioMirror's one kilobyte-long packets, driving efficiency (copy-able coverage area) back up to awesome levels again without frying my 100w rig or requiring an expensive amplifier.

* Many psk programs are capable of encoding/decoding several psk streams at once. Q15x25 mode utilizes 15 psk streams, for example. By using multiple psk streams, and time-staggering the data each one sends, then rapid, multiple opportunities for fills and updates are suddenly a reality, greatly enhancing speed, copy-able coverage area and reliability. While RadioMirror sends its updates and fills in serial fashion, this psk system would do so in parallel (time-staggered), in addition to the original serial update system. (X number of data blocks are being sent at once, "X" being the number of psk streams you decide to use.)

* PSK is a soundcard mode, which would make the transmitted data available to anybody with a shortwave receiver and a computer.

Configuration Options

There are many ways that multiple psk-stream multicast software could be configured. To stay legal in the area of bandwidth, no more than 15 psk streams can be utilized. There are also practical and legal limitations to the baud rate of the individual data streams. Working within these limitations still offers us many potential types of multicast transmissions, tailored to fit specific jobs.

Using multiple streams to effect rapid fills and updates has been mentioned. Another option is to assign psk streams to multiple data feeds. For example: Two psk streams are dedicated to a flash news and announcement system while six other psk streams serve to move the bulk of the everyday data for a total of eight psk streams. The smaller data source's update cycle would be much shorter, so that data would tend update itself much faster and propagate through the network much quicker.

You can double the rate at which the total data is transferred to the recipients by tapping the data at the beginning with several psk streams, simultaneously tapping into the data at the middle with several more. This way, all the data blocks are sent in the time it would normally take to send half of them, while retaining the ability of multiple streams to effect rapid fills and updates on the fly.

The optimum number and configuration of psk streams to use would be a question that can best be answered by trial and error, under varying conditions. Up to fifteen streams can be used of course, but if we could get by with four or five psk streams, a multicast transmission would use no more bandwidth than a standard HF packet signal. It would be slower than the 15-stream version perhaps, but that would be offset by its reduced bandwidth. There is a happy medium there that can best be discovered by direct experimentation. To experiment is so much better than to guess and opine.

Because there is so much to learn, I am hoping that the first "multipsk" or "pskast" (or whatever it ends up being called) software will be sysop-configurable as to the number of data streams utilized. The client software should be able to handle anything from one to fifteen multicast streams without user intervention, so that SYSOPs can experiment with different numbers of data streams under different conditions, to do different jobs.

Networking Multicast

Run both client and server software in the same computer, receiving a distant multicast on one band/freq while passing on updated files via multicast transmission on another. The Server only transmits, and never receives. The client only receives, never transmitting.

Conclusion

Our present dissatisfaction with HF digital communications is solely due to an artificial, self-imposed limitation. We have stunted our own progress by depending far too heavily upon the lessons and expectations of wired networking, which amateur radio is not. There were and are many valuable lessons to be learned from wired networking, but to rely solely upon a non-ham technology to guide the development of our own has proven to be nonproductive, leading to less than satisfactory results.

Part of the reason we have fallen into such a trap was through being content to emulate rather than innovate. More than most, hams were in a good position to appreciate the innovation of the Internet, and our natural awe there led us to believe that we could do no better than to emulate the Internet in our own turn - a supposition that has since proven to be in error, as our ham radio network has different needs and capabilities than any of the wired networks we have since attempted to model our efforts upon. The dissatisfaction and discord some have suffered because of the ensuing frustration has stunted our growth, and tarnished the bright future for digital ham radio that was so widely lauded in the early1980's.

This proposal for psk-driven multicast as the backbone of a global information network is one of many advances that are possible for us when we finally look beyond the wire, when we are no longer content to emulate.

In the mean-time: This proposal has the potential to revolutionize HF digital communications, giving amateurs an information network that can reasonably be expected to cover the entire planet from pole to pole and out to near-earth orbit; Something we can only dream of wistfully while limiting ourselves exclusively to point-to-point networking as we have to date.

Multicast will distribute more data faster to an enormously wider area than our present system, and it will do it while utilizing a fraction of the bandwidth currently required for such a task. At the same time it will remove a crippling burden from our present digital network, allowing it to do its primary job with impressive speed and reliability once again.

Now is a good time for us to remember the bright promise of a global digital network built, operated and maintained by hams. - The attitude that energized us in the 1980's. By moving back to an attitude of innovation and by once again appreciating the unique opportunity we as amateurs are privileged to enjoy, we can step out from behind the protective, smothering coat-tails of the Internet and strike out to see what we can do on our own - as hams using radio.

It is my hope that with this paper I have reminded a few of us how great the amateur service really is, and how much we can accomplish together. - That the boundless pride and optimism about digital amateur radio prevalent in the early 1980's was right all along.

This is my proposal. Thank you for taking time to look it over!

Programmer wanted: Contact:

Charles Brabham, N5PVL
Director: USPacket
http://www.uspacket.org
n5pvl@uspacket.org
Weblog: http://www.hamblog.com/blog_n5pvl.php

Member Comments: