Play & Record 10 times - AD/DA loop back test

[didier.brest]

Updated DiffMaker performance list

Echo AudioFire 12
parameters: -14,39msec, -0,743dB (L), -0,774dB (R)..Corr Depth: 11,8 dB (L), 11,3 dB (R) Diff. RMS -23,2 dB (L) -22,5 dB (R)

Steinberg MR816X
parameters: 80,99usec, -0,122dB (L), -0,122dB (R)..Corr Depth: 2,7 dB (L), 2,7 dB (R) Difference: -23,6 dB (L), -23,0 dB (R)

Rolland MMP2 with frequency matching
-15,6msec, -0,929dB (L), -0,929dB (R)..Corr Depth: 4,7 dB (L), 4,7 dB (R) Diff. RMS -23,3 dB (L) -23,8 dB (R)

Echo layla 24
-44,82msec, -0,929dB (L), -0,840dB (R)..Corr Depth: 6,8 dB (L), 6,8 dB (R) Diff. RMS -25,5 dB (L) -24,9 dB (R)

Focusrite Safire
-10,25msec, -0,982dB (L), -1,291dB (R)..Corr Depth: 6,2 dB (L), 6,3 dB (R) Diff. RMS -25,8 dB (L) -25,1 dB (R)

Fireface 400
-356usec, -1,467dB (L), -1,062dB (R)..Corr Depth: 7,7 dB (L), 7,6 dB (R) Diff. RMS -26,5 dB (L) -25,9 dB (R)

Apogee AD&DA 16-X
-12,92msec, 0,066dB (L), 0,210dB (R)..Corr Depth: 21,4 dB (L), 21,9 dB (R) Diff. RMS -31,8 dB (L) -30,8 dB (R)

Orpheus
13,04msec, 0,171dB (L), 0,058dB (R)..Corr Depth: 12,8 dB (L), 12,6 dB (R) Diff. RMS -32,3 dB (L) -31,7 dB (R)

Universal Audio 2192
parameters: 13,13msec, -0,183dB (L), 0,316dB (R)..Corr Depth: 17,7 dB (L), 17,4 dB (R) Diff. RMS -35,2 dB (L) -34,6 dB (R)

Digidesign 192 I/O
-1,701usec, -0,021dB (L), 0,159dB (R)..Corr Depth: 17,9 dB (L), 17,7 dB (R) Diff. RMS -35,5 dB (L) -34,8 dB (R)

Motu 2408 MK3
-1,284msec, -0,782dB (L), -0,283dB (R)..Corr Depth: 18,1 dB (L), 17,9 dB (R) Diff. RMS -35,5 dB (L) -34,9 dB (R)

Mytek 8x192
-147msec, -0,194dB (L), -0,327dB (R)..Corr Depth: 16,9 dB (L), 16,6 dB (R) Diff RMS -35,5 dB (L), -35,0 dB (R)

Metric Halo
-70,81msec, -0,229dB (L), -0,157dB (R)..Corr Depth: 18,1 dB (L), 17,9 dB (R) Diff. RMS -35,7 dB (L) -35,0 dB (R)

Note that the performance order of the MR816X, the Fireface 400, the Orpheus and the Mytek are the same like in my single loop test with a very different clip.

[nms]

I'd say this does some mythbusting to the comparison that was done between MR816x and Orpheus. The MR is primarily suffering from low end loss. @didier - Do you still have your MR or did you get rid of it? What are you using now?

I'm shocked no one has mentioned how according to this test the Motu 2408 is the 3rd cleanest? I have a hard time believing a 9 yr old Motu you can pick up for $350 is more transparent than a $4800 orpheus as well as the rest of that list. The low ranking of the MR816 certainly goes against all the GS chatter.

[nms]

Curious to see how this one fares.. MR816 DA to Virus Ti2 Polar as an AD.. which is output back into the MR816x via S/PDIF while being clocked by the Virus :P

This definitely wins the most complicated award at least lol

Download


Where the MR816's previous loop stats are concerned, being forced to go through the preamps colors the sound enough to make for low stats. I think it may be a similar case for the Babyface which is supposed to have better converters than FF400. I'm not big on the low end loss on the MR loop though that's for sure and it's surprising that I've never heard anyone mention that. Blatant oversight.

[kraku]

Added the Steinberg MR816.

Didn't add the hybrid setup with TI in it. The reason: if I add one hybrid AD/DA setup in the list, we're going to have hundreds of variations pouring on this thread in a matter of days.

[didier.brest]

Originally Posted by nms

Curious to see how this one fares.. MR816 DA to Virus Ti2 Polar as an AD.. which is output back into the MR816x via S/PDIF while being clocked by the Virus

-32,89usec, 0,331dB (L), 0,831dB (R)..Corr Depth: 9,4 dB (L), 9,2 dB (R) Difference: -28,7 dB (L),-28,5 dB (R)

That is between the Fireface 400 and the 0rpheus.

[karumba]

today i've had some thoughts which metric could be nice to show the analytical difference between all those converters. i found the distribution (PDF/histrogram) of the sample differences (original-converter) quite interesting. here the results:

[mattiasnyc]

Originally Posted by nms

The key to tests like these is accuracy & standardizing the setup and test process. You want to calibrate your i/o gains to unity using a 1k sine wave test tone testing & adjusting one ch at a time and then leave it for all 10 passes ideally. I used a -1db 998hz sine which goes out at that level and comes back into ableton registering -1.00db on the meter. If not working with detented or software controlled i/o levels it takes some precise knob wiggling but anyone should be able to get within .05db I think at least. In 10 passes I didn't adjust a single thing and my final file was within 1db

The test would be more accurate with a short 1khz test tone at the start of the file for level matching after where L/R are concerned for interfaces that don't have detented gain adjustment or are out of wack a bit.

Kind'a late to the "standardization party" though....?

Where does one even begin to figure out where the optimal headroom is for each converter? Surely we can't assume that they all perform their best when pushing 0 (or even -1 for that matter), right?

[johnnyc]

I think this test was a good idea, anyone up for expanding on it and refining? Some suggestions:

1) We develop some sort of calibration standard so everyone tests the same thing
2) Reduce the number of loops to perhaps 4, would save time and better represent real applications
3) Add more sample clips, say perhaps 4-5 different genres each 1 min long
4) Test at different levels, perhaps -1, -6, -15, etc

Might be a bit ambitious but think it would be a good resource for people if we could get it done.

[nms]

Originally Posted by mattiasnyc

Kind'a late to the "standardization party" though....? Where does one even begin to figure out where the optimal headroom is for each converter? Surely we can't assume that they all perform their best when pushing 0 (or even -1 for that matter), right?

Seems like a good time to me. Already some great headway has been made here with a great cross section of converters being volunteered. Just because you calibrate to unity gain levels in that manner doesn't mean you run test signals that hot. Although.. ideally you would want to run a few hot tones to measure if any of the converters choke. If destruction happens at that level, even a bit, you want to know!

@Johnnyc - it's not ambitious at all, it's a great idea and progresses deeper into the value of the results!

[mattiasnyc]

Originally Posted by nms

Just because you calibrate to unity gain levels in that manner doesn't mean you run test signals that hot.

I know. I was saying that without a standard plenty of people might run the material out/in hotter than one would in any normal or "proper" circumstance. And I'd be willing to bet that it's "intuitive" for many because 0dBFS = a common standard (ceiling) that is absolute, so when none other has been discussed...well... you get the point...

At any rate, I made this point like weeks ago, and so did others I think, and it would have been more interesting if what you guys are now hinting at had been done from the start. But alas, people don't listen.....

[johnnyc]

Originally Posted by mattiasnyc

At any rate, I made this point like weeks ago, and so did others I think, and it would have been more interesting if what you guys are now hinting at had been done from the start. But alas, people don't listen.....

You came into the discussion after many converters were already tested. Plus you were trying to make multiple points, some were valid some weren't. So saying people don't listen is not very accurate

[mattiasnyc]

Originally Posted by johnnyc

You came into the discussion after many converters were already tested.

Ok, fine. So many had already done some work. Seems to me however it'd have been a good idea to think through how people do the test sooner rather than later.

Originally Posted by johnnyc

Plus you were trying to make multiple points, some were valid some weren't.

What wasn't valid?

Originally Posted by johnnyc

So saying people don't listen is not very accurate

Ok, fair enough.

[nms]

Mattias.. You were right in thinking the test process needed to be laid out more specifically.. all it takes is one wonky element to throw the results as was seen with the patchbay earlier. The more you can preserve the integrity of the test process the better. Just as the more content tested the better! So let's get cracking with round 2 here using the file I compiled. I think it's relevant to post passes for 1, 4, &10 passes and propose that as the standard, all converters set to unity i/o using the test tone so the test file is running thru at the same levels for all. I think it'd be easily the most valuable resource for online converter comparisons on the forum yet and provide plenty of areas to analyze. A big area is having a broader range of tones and music to test so that was what I wanted to address in the new test file. As much as I like hearing mr Neville sing about rain I want to hear the results with a drum loop from a tightly produced thumping club track. The cymbals in that will do some talking I'm sure.

I'm selling my Slate Dragon compressor now to fund a converter upgrade, instigated by my MR816 results, so this is all relevant for me. I'm going ahead with the new test file and hope to see the rest of you join. We're making some great progress here boys.

[johnnyc]

[QUOTE=mattiasnyc;7149142]
What wasn't valid?
[QUOTE]

There were 2 points, the first being the absolute level, important in the sense of whether we are in the sweet spot or perhaps running the converters too hard, very valid point.

The second issue was calibration in the sense of what if there are losses with each pass, suppose you lose .2 db in level each time. This was shown not to matter that much in the calculations as you can just gain match them after the fact. (obvioulsy wouldn't work if the losses would really large).

[johnnyc]

Originally Posted by nms

So let's get cracking with round 2 here using the file I compiled. I think it's relevant to post passes for 1, 4, &10 passes and propose that as the standard, all converters set to unity i/o using the test tone so the test file is running thru at the same levels for all. I think it'd be easily the most valuable resource for online converter comparisons on the forum yet and provide plenty of areas to analyze. A big area is having a broader range of tones and music to test so that was what I wanted to address in the new test file. As much as I like hearing mr Neville sing about rain I want to hear the results with a drum loop from a tightly produced thumping club track. The cymbals in that will do some talking I'm sure.

I think we should form some consenus on the file first. I like the idea of sine waves and saw tooths, but would rather have them in a separate file, good for analysis but not so much for listening. Music wise should get a rock tune in there and a jazz/classical/acoustic instrument clip as well.

How do you do unity i/o in a practical sense? Many converters do not have a trim available. Also I think using files that peak at 0dbfs is a bad idea, you are pushing the converters plus have the problems of intersample peaks. Should normalize to like -6dbfs or something or even -10dbfs. Finally you have the problem of no standard analog level existing so do we wanna calibrate analog or digital? So even if we cal for unity i/o one converter may be at +4dbu while the other is at 0dbu

[nms]

The whole idea of all in one file is because it is SO much less work to get a wack of samples recorded all in one go. As nice as it would be to have 10 mins of sample content, I think we need to keep it to around a minute length and cover the basics. This is a lot easier for people to manage and thereby will increase the likelihood of people doing it which is essential. People are busy so that's got to be a consideration.

All in one file is best for sure. After people post the files we can easily chop that up. For the music parts I prefer to chop the resulting files in sample accurate loop sections which can be accurately RMS matched in wavelab or sound forge to a hundredth of a db. With short seamlessly looped sections it is so easy to spot differences when you get 2 of them repeating back and forth. I'll extract the music loops into their own files, RMS matched, and people can download and compare any converter to any converter seamlessly. It'd be a lot of work to extract every sound from every converter to its own file so I'm thinking just do this with the music loops and for the other tones people can download the full file for whichever converter and extract and analyze any portion of it.

I think I have our bases reasonably covered with the diff tones I included.. and the house track drumloop is a perfect example of top notch modern dance music production chart hit business. Kraku's clip does cover a fair amount of sonic ground and allows us to measure the results of any new converters with the previous tests results. We can update each of those converters with the new test figures as they come in. If there's something someone else really wants included then do suggest.. but I'm fine with one dance music clip and one clip of music with well produced live instruments & vox.

[didier.brest]

Originally Posted by nms

@didier - Do you still have your MR or did you get rid of it? What are you using now?

I still have it! I see here than other converters are still better than excellent, which may be useful when many AD-DA conversions are needed on the same stuff. I do not need that. Despite the huge difference of the correlated depths mesasured by Audio DiffMaker, I do not hear any difference between the MR and the Mytek, or the original files in my single loop test.

[nms]

here's the Mytek vs the MR816.. RMS level matched and seamlessly looped together. I can't tell the difference here but I bet I could easily with the drumloop I put in my test file. MR suffers low end rolloff which is easily spotted in dance music and clear sparkling cymbals with less sounds can make it easier to focus on differences.

Zippyshare.com - MR816-vs-Mytek-8x192.wav (Download, don't use player)
I'd be interested to know if anyone can blindly pick out between those 2 with that clip

Also.. we should try a different analysis range in Audio Diffmaker. 100hz to 12khz is missing too much relevent info. 40hz-18khz is my suggestion

[kraku]

I'm all for a round 2, but first we need to think through really carefully what we want to test, why and how.

Obviously people want both listenable results and also signals which are optimal for automated metering. Also transient preservation and/or distortions should be tested.

I'd say its still a good idea to run the signal through just 10 times, not 1, 4 and 10 times. In the end the only thing that matters are the relative differences between the converters so people know what they'll get. If people wanted to hear good sounding music, they wouldn't run the signal through this test at all, but of course that would defeat the purpose of this test in the first place.

Also by listening to the current results its already obvious that even the cheapest modern converters are nowadays of amazing quality. You definitely don't have any worries for processing your signal multiple times using outboard gear (ITB-->OTB-->ITB-->OTB-->ITB). If you're hard pressed to hear the difference by ear after 10 loopback rounds, then the metering result differences don't matter much, since you can't hear them in practice.

So let's design what we should do to get the most clear results for the round two. What's the process? What's the source material? What do we test?

[Audiop]

Interesting tests but what are we looking at?

The biggest errors are typically frequency deviations and noise. The frequency deviations is easily measured, noise as well and it's typically inaudible.

If I'm not missing anything I think these tests may give false impressions of "correctness" since no weighing against human hearing is baked in.

Some electronics/converters has relatively much low frequency noise which easily can account for most of the measured differences in these tests while still not audible. That very converter may still shine in a listening test and in real use.


/Peter

[kraku]

Maybe one of the signals should be digital silence fed through DA & AD?

[didier.brest]

The 'correlated depth' measured by Audio DiffMaker leaves the noise apart. Audio Diff Maker correlated depth from comparison of a recording A and a recording B = A + noise is very high.

Originally Posted by nms

Also.. we should DEFINITELY be using a different analysis range in Audio Diffmaker. 100hz to 12khz is missing too much relevent info. 40hz-18khz is my suggestion

This frequency band is selectable and I think that it impacts only the delay and gain correction (which is much better than determining the gain correction only at 1 kHz). The correlated depth is a value for the whole frequency band (from 0 to half the samplig frequency, to be confirmed).

Anyway the difference file generated by Audio DiffMaker from two files A and B being compared is the difference between A on one hand and B corrected in delay and gain on the other hand.

[johnnyc]

kraku, my suggestion of 4 loops was in the interest of time, would rather have a longer clip with a few musical styles, than a shorter one with less variety. 10 becomes a little cumbersome, but willing to do whatever people agree to. Overall it just can't get too complicated or it will discourage people from participating. A snippet of digital silence might be interesting too.

audiop, for me am looking at sound first and then mesaurements second, it's nice to be able to see both, sometimes they correlate sometimes not.

[Audiop]

Originally Posted by johnnyc

audiop, for me am looking at sound first and then mesaurements second, it's nice to be able to see both, sometimes they correlate sometimes not.

They always correlate, in contrast to our understanding of the matter.. :-)


/Peter

[Audiop]

Originally Posted by didier.brest

The 'correlated depth' measured by Audio DiffMaker leaves the noise apart. Audio Diff Maker correlated depth from comparison of a recording A and a recording B = A + noise is very high.

Hi Didier! Can you help me read more about this?

I could only see that gain, timing and frequency response is accounted for.


/Peter

[didier.brest]

Hi Peter,

Time alignment and gain adjustment are determined so that the correlated depth is maximised. Determining the optimal gain value is an iterative processing for which the initial step is 0.1 dB (default setting). Then the step is decreased so that to achieve 0.0001 dB accuracy (default setting). In the default setting, which I used, there is no frequency response compensation. A frequency response of some device can be measured and compensated as an option, which I did not use.

The correlated depth is the ratio of the RMS power of both recordings (after gain compensation) to the component of their difference (after time alignment and gain compensation) : so any frequency response amplitude and phase difference contributes to reduce the correlated depth.

Originally Posted by didier.brest

This frequency band is selectable and I think that it impacts only the delay and gain correction (which is much better than determining the gain correction only at 1 kHz). The correlated depth is a value for the whole frequency band (from 0 to half the samplig frequency, to be confirmed).

tuttNot correct: the correlated depth measurement is also band-limited according to this setting.

[GoldMember]

Originally Posted by kraku

I'm all for a round 2, but first we need to think through really carefully what we want to test, why and how.

So let's design what we should do to get the most clear results for the round two. What's the process? What's the source material? What do we test?

Other converter tests i`ve seen have a file with 3 different music styles,
Piano Voice, Rap, Rock...
it gives a nice perspective of the converters.

some converters shine in one area but no the other...

we should take the original files from here:
Hi End Converter Shootout - Blind Test
& here:
Avid Analog A-D I/O VS .....
& from here:
Different Tubes In A Tube Tech CL1B
& here:
Rosetta 200 Vs Aurora 16 Shout Out The "test"
Rosetta 200 Vs Aurora 16 Shout Out The "test"
Rosetta 200 Vs Aurora 16 Shout Out The "test"
& here:
peeder's Completely Obnoxious Converter Shootout
peeder's Completely Obnoxious Converter Shootout

other interesting converter tests: but no usable files...
A/D Converter Shootout (with samples)
The Lacquer Channel Mastering Converter Shoot-Out
Real A/B TEST ? 2ch converter RME vs Apogee for to choose...

[johnnyc]

Originally Posted by didier.brest

tuttNot correct: the correlated depth measurement is also band-limited according to this setting.

S*** really? That certainly changes things. And would help account for the fact that correlated depth and rms null don't always agree.

[didier.brest]

I think that the main reason why the correlated depth and the difference RMS level do not agree is that the correlated depth takes into account only the signal-correlated component of the difference, not the uncorrelated component, which is noise-like. The energy below 100 Hz and above 12 kHz is usually much lower than the energy between both these bounds. If this property is also valid for the difference (which is likely but not ensured), taking both spectrum ends in the correlated depth computation should not change much the result.

[nms]

well, ghettojo3n appears to have kickstarted the process with the copy of the combo file he downloaded a couple days ago and did the test with proper setup.. so let's start from there and as far as any other test content goes we can add a second file with diff music clips according to whatever gets put together. I processed his results and have started a new thread for us to continue on here: http://www.gearslutz.com/board/gear-...ut-thread.html

@ didier.best - you're the man for official Diffmaker testing so if you're up for continuing as such then that'd be solid. We should test at a range of at least 40hz-18khz though I'm thinking.