You will want to get a kaku obi (belt) as soon as possible. A couple of loaners are available, but this should be the priority. You can get one for about $40 from e-Bogu and other sources. For this, I recommend natural fibers over polyester.
The next priority is a keikogi (jacket) and hakama (pleated divided skirt). e-Bogu has a decent set for around $100. Black is preferred, but blue/indigo is ok too.
You will need a bokken (wooden sword) with a tsuba (hand guard). Kingfisher Woodworks makes very high quality bokken that can take a beating. Their L5 Appalachian hickory might seem expensive, but I recommend them above all others.
Tozando generally makes the lowest-price iaito (unsharpened practice sword) of decent quality. Getting the appropriate length is really important, so please consult with me before you order one.
Nihonzashi is a good source for shinken (sharp swords), but you won't need one of those for at least the first six months, if ever. They offer lots of other quality stuff and they are definitely our go-to for sharpening.
(For better or for worse, these links are unsolicited and uncompensated. Consider these recommendations for good deals on beginner-level stuff that is of decent quality.)