Setting aside software-defined radios, pretty much the only transceivers with any significant number of undocumented commands are from the Yaesu 817/857/897 series. KA7OEI reverse engineered much of the 817's EEPROM memory as described inhttp://www.ka7oei.com/ft817_meow.html
The 857 and 897 CAT systems are based on the 817 design.
While it may be based on the 817, I am currently documenting the 897 eeprom. The read command (0xBB) is the same and just as undocumented, however the placement of where certain bits are is different.
For my perl script that dumps the eeprom, and the memory map that I am compiling you can go to http://www.0xdecafbad.com/?p=154
any suggestions, additions, etc would be gratefully accepted.
The process I am using, which is presumably what HRD, k7oei, and others have done is quite simple, although tedious. You dump the prom and store it to a file, then make a change on the radio and dump it again. Look for the sections that changed, and try to identify what those changes mean. For the menu option changes it will likely be 1 byte (maybe 2) that changes. If you see a bunch of changes that could be because it appears to write to the eeprom when you switch to certain menus (so it can remember where you were when you boot up again), or change the frequency or such. This means that you have to filter those changes out. Additionally many of the items appear to be bitmasks, so a good calculator that will switch between hex and binary is a good thing to have to make it easier to see which bits are changing so you can identify which applies to a given setting.