Someone PMed me to post this test, and since I was curious myself, here it is!

This is a carefully put-together scientific aural A/B comparison test with the goal of removing as much bias as possible in a forum such as this. Some of you know I'm working on a proprietary comparison "application," which is much more scientifically relevant. But in the meantime, I'll post tests like this every so often if people continue to have interest. This is a "preference" test, so please state your preference per group.

There are reasons behind the structure of this test and its files, but every attempt has been made to allow the subject to "PASS," not "fail." If you do "fail" the test, it is very likely that you do not have a preference in this instance, even though you may think you do!. There's more to this test than meets the eye (and ears). Unfortunately, I don't have much time to defend science and the ensuing arguments toward tests such as this.

In order to provide any meaningful results, either on an individual or group level, you must compare all groups that are posted in the test. In this case there are ten [10] groups of short samples, twenty [20] in all.

In a nutshell, in order for you to actually have a preference with any real-world relevance, you must consistently choose the same device at least 9 out of 10 times. In other words, if you choose the same device 8 out of 10 times or less, that is below the acceptable confidence level of 95%. To put it another way, if you don't choose the same device at least 9 out of 10 times, it is likely that you are just guessing. However, nothing is absolutely certain. As I said before in another thread, this test does not prove anything.

Finally, if you truly are not hearing differences (even though you may think you are), it is likely that you will "flip flop" between the devices about 50% of the time.

You may record what you believe you are hearing between the samples. For example, "Group 1, Sample A sounds thicker than Sample B, so I think it's the hardware." Keep in mind that you are only comparing samples per group. In other words, you're not comparing one sample in Group 1 to another sample in Group 2. You must compare only two samples at a time per group.

Here's an example:

G1_Sample_A: Nebula, because it sounds strident.
G1_Sample_B: This is definitely the hardware because it's easy on the ears.

or...

G2_Sample_A: Nebula EQ
G2_Sample_B: Hardware EQ


Not valid (groups aren't the same):

G3_Sample_A: Hardware EQ
G4_Sample_B: Nebula EQ

I've found the best way to compare these samples and to improve the accuracy is to line the samples up one group at a time in the timeline of your DAW. So, the next group of samples will not fall below the first group... the next group will begin shortly AFTER the first group, and so on.

For each group, continually loop the samples so that they will continually play with no gaps, with ONE sample muted. Then, SOLO the muted sample in order to perform a seemless switch between the two. Keep going back and forth until you make a decision for that group. Record your findings.

This is a stacked test of multiple tracks, where each track has been treated with quite a bit of EQ, relatively speaking. Each version was mixed down, so that one sample in a group is a mix containing only the hardware EQ, and the other sample in the same group is a mix containing only the Nebula EQ.

I will post an encrypted key before I reveal the results. Have fun!

AlexB_1_of_3.zip
AlexB_2_of_3.zip
AlexB_3_of_3.zip


EDIT ------------------ New 96K Files Added ----------------------

This is the same type of test with ten groups, but they are randomized again. Please extract all files to one directory and see above for instructions.

AlexB_1_of_5_96K.zip
AlexB_2_of_5_96K.zip
AlexB_3_of_5_96K.zip
AlexB_4_of_5_96K.zip
AlexB_5_of_5_96K.zip

Nice, will take a listen. I have the Black eq as well, havent used it very much lately since ut got my hands on the German Mastering Console

Maybe it's time to pick it up again.

/Jon

very thorough protocol! I look forward to listening!

so are you saying the 3 groups have the same hardware software arrangement or is it random throughout the 10 pairings?

was this session @ 96k?

OK

First of all they are very close but I would say B better (?hardware) in group 1
and A better in the other 2 groups

There's 10 groups, 20 samples total. Unzip all three files to one directory. Each group has two samples: Sample A in group 1, for example, may be the hardware mix, or it may be the Nebula mix. Same with the other groups; Don't assume sample A or sample B is always the hardware; it's random.

No, it's 44.1, so in theory Nebula could be at a disadvantage here. At 96k, the mix, however so small, was bringing my laptop to its knees; I wanted to hear everything with all Nebula instances running at once.

However, now that I have all the settings in place, I could easily mix everything down to 96k offline, but let's see if people are really hearing a difference first.

Finally, I didn't do much, if any tweaks to Nebula, which in doing so can increase the quality even further, but again let's see...

Again, there are ten groups. Please do all of them. You can PM me or post your answers here if you'd like; something like...

The hardware is....

G1: Sample __
G2: Sample __
G3: Sample __
G4: Sample __
G5: Sample __

... and so on...

ok I have submitted my answers via PM and anxiously await my score...

I took the test - convinced that I got it 100% correct - and that the difference was clear.

Turns out things werent quite what I thought!

I challenge people to take this test - very interesting!

PM the OP for answers to keep from biasing the thread...

Here's the settings I used on the EQ:

Kick: +12 dB @ 30 Hz; +9 dB @ 7 kHz
Snare: +9 dB @ 5 kHz (shelving); +12 dB @ 100 Hz
Hi Hat: -9 dB @ 100 Hz (shelving)
Overheads: +6 dB @ 12.5 kHz (shelving)
Room Mics: -12 dB @ 100 Hz (shelving); +12 dB @ 240 Hz
Toms: Didn't mix and are only in overheads/room mics.

Now thats some crazy boosting/cutting right there

/Jon

Yeah I usually don't go that nuts but wanted to see if the emulation would capture it faithfully

I've added a 96K version. See first post!

I would say:

Hardware=

Zip1 = A, B, B
Zip2 = A, B, A
Zip3 = A, A, B

Since you're referencing only three zip files, I'm assuming you're doing the 44.1K version. You only gave nine answers; there's ten groups in all. There's three groups (six samples) in Zip1, four groups (8 samples) in Zip2, and three groups (six samples) in Zip3. Thanks.

cool test, thanks!

I did just the first zip files and not sure what is what but this samples sound alike (after listening each of the samples not longer than 5 sec so i could be terribly wrong):

G1 - A
G2 - B
G3 - B
G4 - A
G5 - A
G6 - A
G7 - A
G8 - B
G9 - B
G10 - A

I did not download or listened or compared anything in this thread. I simply prefer hardware EQ to software counterparts. I don't really need to participate in any tests. I don't doubt myself or my preferences. I don't feel need for validating anything. I just twist the knobs and listen.

Eversince I've fully migrated from ITB to OTB I've lost interest in any comparisons tests. I do compare different brands of red wine to each other however. Does not require downloading and staring at computer screen. To each his own. All the best.

Hahaha... Man, sometimes I don't understand why people even take the time/energy to post comments like this. Must be a Gearslutz thing, I guess...

@ SoSueMe: Thanks for sharing your test with the rest of us. Some of us actually do appreciate it.

PMed

You're welcome!

Finally took the test !

Using 96khz files @24bit ;

G1 - B
G2 - A
G3 - A
G4 - B
G5 - A
G6 - A
G7 - B
G8 - A
G9 - A
G10 - B

The selection was completely based on preferred sound and i think i managed to distinguish the same device in each group. Group 10 was a little more difficult to judge.

Although this was a test for eq, one of these devices had more space/depth and punch.

Edit ; lol and i've just noticed that i've almost picked the complete opposite to 'dotl' - so that must be a good sign of accuracy at least.

I literally can tell no difference between them. This is the first time in my life I can't tell a difference in a shootout. Thanks for doing this. I don't even want to know which is hardware cause it doesn't matter.

Thanks! I find it really interesting how very few people KNOW they cannot tell the difference versus people that THINK they can tell the difference. But perhaps many of those took the test and just didn't respond due to the *****slut factor'.

Knowing you cannot tell a difference takes a little humility when you are posting on this board in front of peers. I opened your reaper project and randomly muted different groups, and then played the whole thing. I would be surprised if anyone can tell a true difference. Usually plugins change things in the highs that are noticeable VS hardware. They lose width or just don't sound as sparkly. That is obviously not the case here. It just reaffirms my love for Nebula

Thanks for sharing. It's a good test procedure in my opinion.

They are close. They don't sound the same. But i've done the test 4 times in different order, and did'nt found the same 'favorites', exept for the group 6, where my favorite is always A.

Seriously, the performance by itself is WAY more different in sound from some section to others, than the EQ used in the same section. And the drummer is good and consistant.

Can you post the results? It would help the lowend slutz like me to make an opinion on our favorites.

You can't claim A & B do not sound the same with any reasonable certainty from a specific group unless you randomize A & B from that particular group into a minimum of ten or more groups, not knowing the new order, then perform another scientific test on those ten new groups.

Furthermore, you can't claim your favorite from group 6 is A with any reasonable confidence because there could very well be bias there. For example, you may have heard something you liked, whether placebo or not, from A one or more times, and then never was able to break that particular association from A, even though what you may have heard wasn't actually there.

This is why the test is broken into ten groups. This is a scientific test based on known probabilities and null hypothesis. The test needs to be looked at and performed as a whole. A preference in only one group is meaningless; you need to give your preference in all groups [10]. If you do not have a preference from a particular group, then that group counts as "no preference".

Yes, which is fine in a test like this. The sections you hear are from one "performance" that is very short. The EQ settings are the same in every section; they were not adjusted/changed per section.

That would render this test invalid as any further testing may have injection bias since the subjects may know what is what ahead of time, forming opinions either consciously or subconsciously.

You can PM your preferences to me for each group (please specify whether you tested the 44.1 or 96K version), and I'll be able to tell you whether or not you actually have a preference with very high certainty. In other words, you may think you're hearing differences when in reality, you may not be. That is the goal of a scientific test like this.

I did randomize A&B. I still claim they don't sound the same. And i do think it's different from saying " I can pick A in group2 each time", or "this one is better", or "this one is the software", or even, "i like this one the most".

It just mean i have certainly heard something, but quite marginal and i don't identify what.

Yes. I know.
In fact, i could have reply only this : "i failed the test".
I just tried to give some details and listening procedure...
..and put in perspective that the differences are less important than playing constistency, although the performer is good.

And, yes, i agree you just can't post the result, it would bias the test.

I will try tomorow, and if i can pick favorites with enough confidence, i will PM.