Don't go out and buy a full set of cycling clothing from the get-go - buy according to need. It's easiest to start out with clothes you may well have to hand: football shirt / running shirt, fleece, jogging bottoms, basic waterproof cagoule, lightweight walking shoes etc and build from there.
A lot of (cheaper) non-cycling sportswear can be incorporated into your everyday kit - running and football stuff for instance makes for good (wicking) base layers, while skiing thermals make nice mid layers when it's cold.