Other software wasn't really meeting my needs.. and I had done some FX processing software before (Jesusonic), as well as started writing realtime encoding/decoding/mix engines (NINJAM).. so actually I started playing with making a multittrack Jesusonic, and then realized how much more it'd be..
It started out named "jmde" (justin's music development environment or something), but then I went through some ideas for things you could pronounce, including CRAPR, TRAMP, CRAMPS, RAP, JEAH, etc (all standing for things), but ultimately settled on REAPER (Rapid Environment for Audio Prototyping and Efficient Recording).
But that's not a technical answer, I'll save that for #2.
2. program it technically (talking about audioprocessing, VST implementation, mixbus implementation)
I had already written a bunch of the I/O code for Jesusonic and NINJAM, i.e. ASIO drivers, WAV reading/writing, etc etc.. So really it was a matter of writing a win32 application, which I guess I have a lot of experience doing at this point.
Adding VST was a matter of getting the SDK from Steinberg, and (since the SDK seems very focused on writing plug-ins not hosts) figuring out what was required to host. DX plug-in support is an even bigger mess, so I got Christophe to help on this as he had a lot more experience with DX.
I've described programming style a bit in my blog, you can read part of it here:
Haha well function always came first.. And as it became obvious that it was ugly as hell, we evolved it. The evolution you can view here: http://reaper.fm/timewarp.php
4. test reaper with those 1000 of systems out there
Fortunately as a Windows application, most stuff is pretty uniform.. but you have a few different OSes to deal with, different audio drivers, MIDI drivers, etc. We get our users to test, and occasionally when there's a big problem we either go buy hardware or get it loaned to us by our (gracious) users...
5. get in contact with your customers? (forum, by phone, service hotline)
We try to be very responsive on our forums, and tend to reply to email when possible as well. A few of our commercial customers get phone support (if I'm around) when they're out recording in the field.
feel free to answer very technical (I would like a very technical answer on 1 and 2).
I apologize if I wasnt technical enough.. if you have any specific questions I'd be happy to answer!