I have a 90 foot heavy freestanding triangle tower (about 12 foot wide base, 6 inches wide at top) which i plan to install a Optibeam OB4-40 mono yagi (50 foot boom) (80 kg antenna)
The weight of the antenna is virtually meaningless, while boom length and wind loading are extremely critical. How the wind load is distributed on the antenna are critical, and if that is like other Optibeams I have seen, it has a square boom. A square boom aggravates wind load issues compared to a round tube, so I would carefully look at wind load and torque.
Worry about weight is generally pretty silly until you get into many hundreds of pounds to several thousand pounds, depending on the tower.
At first i wasnt sure whether to guy the tower for extra safety but now im thinking i definitely should.
I would not do anything without consulting a mechanical engineer, or at least someone with considerable experience and a logical mind.
Most towers like you have are NOT designed for resistance to twisting. They are designed to support fairly low side-pressure loads at the top, with very minimal twist on the tower. The general use are small surface area vertical antennas, not long booms or sidemounts.
Most failures I have seen in towers like that have been top section failures from twisting fatigue, when FM antennas with radomes or dish antennas are installed in the narrow area.
You have very roughly a four inch radius torque tube at the top, and you have a 25 foot radius lever working on that four inch tube. This is around a 1:75 multiplication of whatever force is pushing sideways out at the boom end. If you have a 30 PSF side load at a boom end caused by a wind gust on a long element, the skinny top has to handle about a ton of twisting force.
There are very few 8 inch lattice construction towers that can take such a force, and my numbers could be off as much as 2:1 or more depending on how the top is built.
Of course this twist is applied where you mount the rotor.
At the very top, with a bearing or sleeve, it is all side force and the tower really just braces the mast.
I have many opinions of what guy wire to use and how many and where i should install them on it. Im hearing i should probably use either gal or stainless steel wire atleast 8-10mm thick (maybe thicker), and install a single guy off each side ( +/- 130 foot long) mounted about 2/3 the height up on the tower (not right at top).
I'm not a mechanical engineer, but I have installed many large towers and seen many installations that have failed. If I wanted to use a tower like that, I would build a star bracket that mounts where the rotor mounts and re-enforce the tower around the rotor out to the bracket. I actually would pull the eight inch area off and throw it away, unless it was super heavy construction.
I would never use a Rohn 25G to support a 50-foot boom antenna with long elements, and it is 12 inch face triangular. I'm assuming yours is 8 inch square, but it might be triangular.
No matter what, a 50-foot-boom, long-element antenna is terribly unsafe on a support only a foot or less diameter. It has nothing at all to do with weight, because it could have thousands of pounds of twisting torque.
But then another ham thinks guying it like this may make it even less safe than not guying it (however thats possible).
He might be correct. You would localize stress at the guy attachment point on a tower not designed to tolerate a large load in a tiny area. I would never guy directly to a tower not rated to be guyed. With load spreading at the attachment point it would be a different story, but that would require custom guy brackets.
The commercial antenna companies ive asked for advice have been no help to me, and its hard to find someone here with experience in this.
I wouldn't touch it myself, unless it was in a secure area and I could build some good star brackets with multiple attachment points to spread the load. I would guy at the rotor, and re-enforce that area.
My Rohn 55G triangular uniform cross section tower has an 18 inch zig-zag braced face with 1-1/2 inch legs that are fairly thick, and it twists quite noticeably in the wind below a 50 foot boom 40 meter Yagi with a round 3" boom.
I can't fathom using something nearly 1/2 the effective radius. I wouldn't even remotely consider a 12" face tower with a 50 foot boom antenna. Good 12 inch towers handle something like a 30-foot boom 20-meter antenna when properly guyed, but a long boom just kills them.
My 25G tower has star brackets at the rotor and I still limit booms to about 30 feet, with nothing larger than 20 meter elements. I can watch my 55G and tell it is twist stressed from the 50 foot boom antennas.
The nice thing about guying is if done correctly, if it fails, it all collapses in a nice pile around the base.