Mix decomposition into separate instrument tracks

[rad9k]

Many amateurs asks about vocal separation from mixed track. Most people that know a little bit about digital audio processing thinks that “mix decomposition into separate instrument/vocal tracks” is not possible.

Some statisticians / machine learning experts (including me ;p) know that such things are possible (always based on some king of guessing, so 100% accurate results are often not possible).

Now we have working proof. It is called hit’n’mix. Additionally it allows for note transpose for separated tracks (melodyne can do it too, so it is nothing new).


Hit'n'Mix DJ Mashup Software


BBC News - New utility that breaks music down into its components

[rad9k]

*kind not king

[narcoman]

When we say it's not possible we mean it's not "possible 100% accurately". YOu would need an expert system {basically a priori knowledge of the original - in other words have the multis and then.... well you get the picture} to do it. In terms of what we as music recordist want or judge, anything but near 100% success is a fail. Statistical and inference techniques get nowhere near approaching 100%..... you'd be lucky to entertain 10% success. It'd be interesting to compare a "before and after" scenario to at least see if there is a correlation between a full Fourier analysis {since there is absolutely no chance of gaining a point by point correlation}.

Secondly you can't entertain splitting co-incident frequencies. There is no longer any separate information present.

So - this mathematician only "amusingly" agrees with the idea of separating stems out. You can't do it in reality - much information is lost upon mix. SOME position and frequency information remains and it is that and that alone that can assist you in separating out. Certainly the goal of separating out parts in a mix is no more doable now than at any time in the past 50 years OTHER THAN there are now implementations of some of the esoteric wave filtering algorithms, either harmonic, pictorial or stats based. DNA and other such methods have no awarded PhDs around them as they contain no new methodologies...... we have merely journeyed down an implementation path and not generated any news maths for deconstructing complex wave forms.

We can calculate PI to more places than ever before - yet the techniques for doing so have been around for a very long time. Again - in the same way - an increased empowerment of implementation and NOT new techniques.

The problem is - without direct comparison you can never tell what you have extracted..... it is entirely likely {and as a statistician you will know this} that the complex harmonic integration will be separated out into previously none existent separate waves. ALso - it can never work for uncorrelated material {either pictorially, harmonically or statistically} - so reverb will always **** you up!!
Hit n mix sounds like crap by the way and in the end that's all that matters but I concede that it WILL end up in some mashup or electronic music forms where it may work very well..

[rad9k]

I would love to be able to continue this thread properly but as I do not have time to play with hit’n’mix I can not (within 2 weeks from now).


But I would like to add some important statement :O)

Secondly you can't entertain splitting co-incident frequencies. There is no longer any separate information present.

It is only partly true. There is no longer such information in the mix. But there ARE (in some universal knowledge space – I’m using metaphor of course and probably hit’n’mix is not using this knowledge) some instrument models that describes sound of particular instrument. By proper analysis of not co-incident frequencies we can parameterize such a model and get proper output. Of course “proper“ does not mean 100% as in separate track but probably better than hit’n’mix.

There could be some problems with access to this universal knowledge space. Models needed for process I described above might not be publically or even anywhere a viable. But I’m sure that those can be constructed so from my perspective – they already exist. Sounds like some marketing bullshit, so…. probably to prove my statements I would have to point to some working example. I hoped that hit’n’mix is such a thing – but as You wrote that it sounds “like a shit” – It might be not :O)

[narcoman]

All instruments, unfortunately, produced uncorrelated harmonics {if such a thing is name-able!!}. You don't just get fundamental based harmonics - you get uncorrelated noise elements, room combined uncorrelated reverbs, and tonal elements that are not part of the fundamental or any related harmonic.

[thenoodle]

Hey I'll try it since the demo is free. Before that though, their little video demo, as with a lot of these types of programs, is a bit of smoke and mirrors. The mixed track itself is a predominant bass drum, hh ,solo synth, mono bass, etc with only about one polyphonic instrument in the mix, being a string synth. Mostly a lot of mono stuff not sharing common frequency areas. This kind of stuff is relatively easy to un-mix with a few cheap tools around.

But yeah, I'll try the demo. I'll try it on a few mixes that have duplicate rhythm guitars, lots of polyphonic parts, overlapping frequencies of non common instruments, ride cymblas etc .. and no easilly pulled center items. That'll show the demo's capability in more of a real-world context. I do realize that the company isn't exactly promoting the software for that, but that's the real world challenge for really un-mixing stuff.

So far to date, I have two programs that are very good with the above type torture tests. Not perfect, but getting closer each year. If you have to ask what they are, you can't afford them. If you already know what they are, you probably aren't gonna share the info since its a great income generator for services.

[narcoman]

Well I'm afraid I'll have to ask you what they are cus I've no idea which yer on about! Unless you mean ADRress, in which case I'll pass .... I've worked with other stuff from the DIT.

When I was doing my PhD it was certainly something we discussed. Had loads of ideas of what to do, but the issues were always the same.... Uncorrelated material was the issue. So none harmonic distortion, reverb, overtones and any excessive time domain information. Looked at statistical information, Bayesian models too . Problem is you need a highly modelked expert system. The conclusion of that expert system us, problematicslky, a perfectceescriptor if the multitracks . Kind of a catch 22.

Further - consider the source. A mixed stereo (or more) music "file" or sample. This file/sample is NOt a waveform additive version of the multitracks. No - it is a processed....nay distorted, compressed, phase altered, harmonically and in-harmonically uncorrelated (reverberated) unique mulch!! It has no relationship to original multi track so the BEST you can hope for is some kind of separated - and possibly useful - new creation! It will never ever be the multitracks - so it contains little educational information. It will obviously have use artistically.... but it is not "de-mixing" - but rather creating a new constituted soup that has no relation to the original multi's but DOES bear a statistically piecewise relationship with the finished, mastered track.

Research-wise - I'm unsure as to why Dublin carries on looking at this because it carries no new math and is nothing more than a recursive Fourier system coupled with some inferences (correct or otherwise) based on statistical guessing! Sure is fun - but it's not mathematically or statistically that interesting! IT also ins't interesting to the professional recordist... so..... Karaoke?
Basically - the results of the computationally brute force techniques are endorsing the same results I saw nearly 20 years ago! A nicer version of gre same issues. All very interesting, and indeed useful... But the problem has a recursive confirmation problem....

[narcoman]

Originally Posted by thenoodle

If you already know what they are, you probably aren't gonna share the info since its a great income generator for services.

I know of two companies using these techniques (as outlined above) to claim they are making "new versions" for music that needs to re-create a classic sample. When the labels get wind of what they're doing (and I'm not a bean spiller) I'm expecting court cases!!!

[rad9k]

Narcoman (nice nick bdw :O), I know about issues You are writing and generally it seems that we can have a nice chat that even might not finish as a flame :O), but as I said – only proof of my statements would be working decomposition soft…
And as I have some other ideas (some novel composition computer aid based among other things on pitch set class theory, call it object based, modular super midi sequencer) in my spare time I’m doing those….

Thenoodle, do You have any results yet ? I’m very much interested in those
And those other soft You do not want to tell us – does they work for You ?

[narcoman]

Absolutely ! We have a saying " the proof of the pudding is in the eating"....! In other words - go for it and let's see what it does!