The reality is that there is no clear answer because you guys never actually came up with an agreement. Lesson learned obviously.
First, your argument that he has no stake in the master recordings pretty much has no grounds whatsoever so you can give up on that one – it would be a massive uphill battle you are nearly certain to lose. Your best case scenario is that you both co-own the sound recordings and you can make a legal argument to this point. However, he probably has a stronger argument that he owns 100% of the sound recordings because he actually paid you. Although being paid for studio time, or to produce or to engineer does not automatically waive your rights to the sound recording, it is generally held that in the absence of any agreement, it is typically considered a work for hire. In other words, if it walks like a duck and talks like a duck, it’s usually a duck. Not always a duck, but usually a duck. Typically, if someone pays another person to engineer and produce and mix that work will be understrood in the agreement (which you don’t have) to be work for hire or an equivalent whereby that person gives up their interests in ownership to the sound recording.
So to summarize, with regard to the sound recording rights:
Least likely scenario that could be argued: you own 100% of the sound recordings
Somewhat likely scenario that could be argued: you both co-own the sound recordings
Most likely scenario that could be argued: he owns 100% of the sound recordings
Which argument would win in court depends on how good your arguments are, third party witnesses to discussions that could actually be an agreement, and a lot of detail that isn’t obviously in this thread. But based on what’s been supplied, those are the three scenarios. If you both co-own the recordings then either party can use them or license them on a non-exclusive bases so long as they make sure the other party gets their pro-rata share. But neither party can do anything exclusive, nor destroy the recordings. Note that because we are talking about sound recording rights, any use would also have to clear with the publisher for the mechanical, performance or sync rights inherent in the composition. If you successfully argue that you own 100% of the sound recording rights, then you basically give him a slam-dunk to require that he get all profits until he is reimbursed what he paid you – so think long and hard before you make that argument.
None of this, of course, has anything to do with the rights to the songs.
My personal belief is that if I were in your shoes, I would never expect to have any ownership in the sound recordings unless there was a specific discussion about my retaining some ownership, and of course that would then be put in writing. If I were working at a seriously reduced rate as you’ve suggested, then I might argue for co-ownership, but never 100% ownership. That’s a little nuts in my book.
Oh, and by the way, never get legal advise on this forum because 99% of people on here have no freakin' clue what they are talking about.