First things first.
The first thing you need is a radio. I would start with something like a FT-1000MP, and install an Inrad roofing filter. This will serve you well for many years. Spend some bucks on a good, used, contest grade radio.
Next, we need a computer. An older PC will be fine. N1MM is a great program, and is free. Don't let the price fool you - its great. A CRT will be fine (cost effective).
Next, we need a tower and antennas of some sort. Many great suggestions have already been made.
I would NOT buy an amp at this point, for several reasons. Spend your money on antennas - that way you can transmit better AND hear better. Low power contesting will help fine-tune your operating skills. When running HP, this puts into into the HIGH POWER category in most contests. The folks with big antennas, SO2R and every operating aid they can affort run in this category (SOHP). By running low power, you are competing against similar stations - tribanders, one radio, etc. Low power also helps minimize RFI type issues, both in your home and your neighbourbood.
Later on (as in quite a few years later), should you have the funds and still feel the need to run high power, THEN get an amp. Once you know how to effectively use low power, and learn most of the tricks, you will be surprised how effective an amp can be. Buying one now will not get you those critical operating skills.
When looking for gear, buy USED stuff. At this point, you are just learning the ropes. Will you make mistakes? Hell YES!! This will help stretch your budget, and when you do make a mistake, minimize the effect. The one exception I would make is your operating chair - go to an office supply place and buy a comfortable chair. You WILL be spending a lot of time in it, may as well be comfortable!
Last, but not least, look around on the web. There are lots of station pictures, writeups, etc. Most of what you find - the folks have a lot more money to spend, and have been at it for a lot longer. Once you get over the WOW!! factor of that, look beyond that to find good ideas you can incorporate into your own station.