My reco - I wouldn't spend a lot of money until you try out some different things and get a feel for what you like. At the very least you're going to need a (half decent) computer to work with (unless for some strange reason you want to spend tens of thousands of dollars on a purely hardware based setup).
From there I'd recommend downloading the demo version of Ableton Live and then go through the built-in tutorials to develop a basic understanding of how it operates. If you like it (and this is something you want to stick with) you can buy a full copy when the demo expires.
For sounds, you'll need plug-in synths. There are a number of half decent ones available with the Abelton Suite (not sure if they're available in the demo or not). Also, you can get demo versions of some of the major ones out there - check out Native Instruments Massive. There are also a who bunch of free ones available, e.g. The 27 best free VST plug-ins in the world today | MusicRadar.com
Note that there are lots of effects that are included in that list - focus on the synth sounds for starters and then expand into the effects later.
I'd recommend going through as many video tutorials as possible from youtube. Many are crap, but you'll figure out which ones pretty quickly. It's helpful to have a dual screen setup and walk through exactly what they're doing in the tutorial with your own setup (i.e. only do tutorials related to the instruments you have, at least at first).
Assuming you already have the computer, you can do all of that for free. If you like it and want to delve further down the rabbit hole, then you can start looking at interfaces, outboard gear, hardware synths, etc.