Martin, the only thing that immediately stuck out was your use of a traditional drills instead of a Unibit.
Indeed, I use high-quality Bosch bits specifically indicated for aluminum and so forth. I do own a couple of Unibits but have never used them.
Keep your drill press speed down and use a drilling lubricant.
I use the indicated speed for the material and size of hole, as provided by the drill press manufacturer. No problem so far. I always start with a small pilot hole and then progress to a larger bit if that's called for.
When drilling metal, I always lubricate the bit with "3-in-1" oil.
Also, because this aluminum is so soft, it will produce a sizeable burr on the inside of the cabinet.
I use a couple of old drill bits, larger than the drilled hole, and deburr by rotating the bits by hand. Seems to work fine and it's the technique suggested by ARRL in its manuals from 50 years ago.
However my key "anti-burr" technique is that the underneath of the aluminum is always clamped firmly against a flat piece of wood, with the clamp only an inch or two away from the area that's being drilled. This reduces the amount of burr, as well as preventing deformation of the metal. This is easy if you are drilling through a single piece of flat metal. For a prebuilt chassis such as the Bud in this case, it requires some kind of jig.
For large holes, I have a set of eight Greenlee punches.
Thank you for your post -- sometime I'll try out the Unibit!