First of all, you have to ask yourself the question, "Do I want this test to be scientific or not"? And, what exactly is "scientific?" Is it worth putting the extra effort into it?

It is the general consensus that unless a test implements statistical significance utilizing known probabilities and null hypothesis with a minimum number of trials (10 - 25), and a minimum confidence level of 95%, the test would not be deemed as scientific. Of course, significance testing goes far beyond the field of audio.

A very brief description of ABX testing can be found here, which touches on what I'm referring to. There are very large charts available that list confidence levels achieved based on the number of trials performed and answers given.

IMO, if a test will produce an outcome different than its scientific equivalent, then it is flawed. If you disagree with this, then you're effectively arguing against science.

Is a test that implements less than ten trials, for example, flawed? Based on scientific reasoning, yes. What if, for example, a subject performed ten trials, but chose 7/10 as being their preference. In other words, three times they chose preamp 'B' as their preference, and seven times they chose preamp 'A' as their preference? A person not familiar in the area of statistical significance would likely say that preamp 'A' is definitely their preference. But according to science, it is not! In fact, they do not have a preference according to science (without a minimum degree of certainty). If someone says otherwise, then they should back up your reasoning with science.

In significance testing, we want to be relatively certain that the subject is not guessing, factoring a minimum confidence level of 95%. In other words, if a subject chose preamp 'A' 9/10 times in a randomized scientific test, then we can be highly confident that he/she is indeed not guessing (9/10 trials has a confidence level of over 95%).

Not following the above guidelines is just one of many ways a test can be flawed, but it's not that hard to come up with a test that is reasonably flaw-free.

As this thread goes on, you'll start to see how other flaws will be called out and/or questioned. But, how many tests even follow the above guidelines? 1, 2, 3 maybe on this entire board?

Yes, level matching (performed properly) is mostly relevant when talking about fine details between two units, but more preamps fall into that category than you might think when compared to each other. It's a judgment call while observing the files in post. If differences between two preamps, for example become great enough as in your description, then level matching becomes less relevant because those differences will overtake fine level matching.

One way I level match is to perform a sample-accurate "RMS inverse null match," while watching the meters over short intervals. I adjust the level of one inverted sample until the output is at its lowest on average over time. Of course, there are other methods I use... it depends.

An example of where level matching is extremely important is when comparing A/D converters. Often people will talk about differences in converters that relate to frequency response, for example, similar to what you say above. But in reality, that really doesn't happen. In many cases I can peak match on these and rarely will someone be able to pick out one converter over the other in a scientific test. All of the sudden they sound the same.

I actually did another thread comparing a high end standalone converter to a low end interface's converters running through its preamps at minimum level with pads, and over thirty people could not tell the difference.

Was that test flawed? At a scientific level, since nobody could reliably pick out one converter over the other, the only way it could be flawed is if the files were somehow tainted, or made 'similar' in some way. I was very thorough in making sure this did not happen. All connections were completely discrete, and I performed peak level matching for this test.

Now you've got me interested. Maybe the tests will save me though.

Asking subjects for their preference is social science.

If these data will indicate something other than personal preference, what criteria will the respondents be given for selection? What are the variables and the controls?

This is something some people would argue against, making claims like "the loudspeaker isn't a real instrument and isn't capable of reproducing what an instrument produces." and "degeneration loss due to sampling a sample." Etc., etc. But, IMO I think it's fine as long as you have wide bandwidth monitors with very wide dynamic capability. Mine have more bandwidth and dynamic capability than some instruments I own! So, what if I played a high quality vinyl direct to disk recording??

Preference is real world. When you choose a preamp in the real world, it's by preference. Plus, just because we ask for preference doesn't mean that's what we're really after. Controls are overrated... when we listen to a recording on a CD, for example, we don't have a point of reference (a control). How do I know what the band sounded like in the studio? Most people don't care. All they care about is that it sounds good (preference).

How will the population be defined? Any and everyone? Any "golden ears"?

Impossible to make such definitions in a forum such as this. I somehow have the feeling you're getting at something. By the way, "scientific" is being used somewhat loosely here. We are on a forum, after all.

If I'm getting at anything it's just curiosity about how this will unfold.

It's hard enough to design experiments for hard science, where things are physically measured and results are repeatable. Designing social science experiments is difficult and good statistical method is required.

It will be interesting to see how the preferences of your cohort weigh out and, especially, how they are characterized and interpreted.

A preference is a personal opinion. Without a selective cohort and selection criteria --replacing mere preference-- the findings can only identify a batch of personal opinions in some ratio. Such a finding may be interesting but it'll be open to a broad range of interpretation. That is not hard science; at best it will be a weak application of social data science.

FWIW, I love listening to well made clips designed to facilitate gear comparison. But it's a subjective affair with lots of variability that's best done when the setup is clean, well described, and the listeners are allowed to decide what they like and why.

I am going to state the obvious Rumi, so forgive me.

We must remember the number 1 pitfall:

The more analytical you get here, the less you are actually listening.

Bottomline is that you are doing this test for yourself, so you are allowed to do whatever it takes to get your head into the proper listening space.

Setup the test as planned, with enough caution to mental trickery, and then execute the tests ONLY when you are able to suspend all this mumbo jumbo and listen to your fullest capability.

We are not scientists here. We are not selling numbers. We are selling emotion. We choose gear based on emotional conveyance first and foremost. To say otherwise is to put music second.

Afterall, dont you want to choose the preamp that moves you best? That illicits the most obvious emotional response from (*cough*) the music?

If you are worrying too much about the validity of the experiment, you could miss out on which preamp truly delivered the goods!

Don't worry about us here. Just do whatever you have to do to get your mind out of the process (counterintuitive i know, but think about it lol) and then pick the pre that you enjoy the most!

And then, if you still happen to feel like sharing with us... lucky for us! BUT we should be a secondary response for you in this matter.

Listening is done intrinsically for yourself. Sharing your opinion is done extrinsically for others. We are wolves that are going to chew on your opinions with our minds... cuz we aren't there with you.

Please consider my words here a reminder for myself as much as it is for you.

Have fun! heh

If you REALLY want to give a proper comparison test, you might want to model it like this:

abx_test

Keep it ABX. Let your subjects hear A. Let them hear B, then let them hear X (which could be either A or B, but they don't know). Have them identify X. Give them at least 5 seconds between, so they can forget the last thing they heard. Make certain the "A" and "B" examples are as similar in output level as possible, and that they were created exactly the same way, aside from the change in preamp.

That will tell a much different story than most of the standard crap that leans towards confirmation bias. And that's one type of comparison examination.

Another type of comparison examination is to continually switch between A and B and listen to the potential differences. This method is not a reliable way of making certain that a listener is able to perceive a difference in the first place. But it's fine for when the perception of a difference isn't the point.

If I was attempting to be honest about my evaluation of a pair of preamps I'd start with the ABX, then I'd move on to the AB comparison. The ABX would tell me just how much of a difference I'm really dealing with, if any. The AB would allow me to examine further the differences I might have perceived. I think that without a proper ABX we are often hearing our mental representations of difference. ABX, done well, keeps confirmation bias in check. But it's a challenge to do well.

Yes. We got you thinking. Now try to have fun.

ABX & AB are far from the only game in town. But none of it means anything if what you're really listening to is not what was intended.
As an example, and this has happened to me a few times over thirty years of recording/testing...

I'll compare two high end preamps with two mics, for example. I'll place the mics as close as possible while trying not to disrupt their polar patterns. The mics are a matched set and they're omni. I'll run an ABX test and will choose one over the other 10/10+ times. Then I'll switch the preamps and again, make a solid choice 10/10+ times BUT it turns out that I chose the other "preamp" the second time. Then I'll move the mics around and do it all over. Same thing!

What does this mean? What it means is that the mics and their positioning are overshadowing the sound of the preamps. No it is not the brand/model of the mics... I've done this experiment with many mics.

So my point is that he may not truly hear what these preamps really sound like. Of course, he could just rely on "emotion" and subconciously pick the one that has a pleasing red color, has a highly regarded brand name, and is backed by much more hype as being regarded as the better unit (very typical in gearslut land).

There is not much to find on the Pueblo website about the design concept. Is it transformerless, is it opamp based on the inputs, or is it instrumentation opamp based, or fully discrete? Or maybe a combination.

But SoSueMe, if he relies on emotion, chances are it will have little to do with the bling factor or brand name of the mic! But I agree with you in that we must keep the 'mental trickery' out of the equation.

ABX works great. Just keep in mind that you will need to rotate the mics across the preamps and somehow work that into your A/B listening.

And as far as capsule 'overshadowing' is concerned. Do not use close micing technique for this test. The voice is good for mic shootouts but not for preamps when you are trying to listen for broad spectrum acoustic energy and transient performance (speed) through a micpre.

3 of the same capsules at 2-3 feet away from an acoustic guitar *should* result in inaudibly different pickup responses. Just be careful they arent too directional, cuz then the sound hole becomes a factor. Omnis or wide cardioid, like the mk21s are preferred IMO.


This is all we get for now:

DC-Coupled preamplifiers. Ideal for ribbon, dynamic and tube microphones. Also useful as DI preamp, Passive EQ/Compressor gain make-up, etc. Outstanding imaging and ambiance retrieval. Ultra-Low noise. Virtually overload proof. All unnecessary features and components omitted for signal path purity.

IMO emotion is often highly governed by subconcious bias injection. I've witnessed it many times well *after* the emotional response.

And what about my other comment concerning the mics overshadowing the sound of the preamps under test? What then? Should I just rely on gut feeling? The ABX test is telling me that I prefer both preamps equally? No.

By the way, I mostly use 4006As or CMC6 + MK2 not close to the source.

Would active ATC SCM 100 qualify for that test? That would simplify the tests quite a bit. Although I must say I have a completely un-scientific preference for real instruments...
You could send me a file that would be best for such a test, and I can record it through the preamps (once they're here). Or I take a Chesky record or something similar with good dynamics and sound.

Thank you, and everyone who has contributed!

I am open to the inputs stated here, and I've gotten some inspiring and helpful things to consider from this thread. However, in the end I will do the test in a way that is doable and appropriate, given the circumstances and intended purpose. (And yes, I am aware that I have asked SoSueMe to share more about scientific tests, and I am thankful for all contributions!)

Let's not forget that this is about choosing mic preamps, and the tests should help to be able to distinguish between them, and choose the best one for a given application. And yes, it's about music.

I know science has its place (I've studied quite a bit in my younger days, and have taught science and philosophy), but it can get ridiculous. For example, the social science department at the university I studied and taught a long time ago was so concerned about being scientific that they occupied themselves almost completely with statistical evaluations that were more or less irrelevant and ridiculous.

I've checked that ABX site, and I'm a lousy ABXer. I don't get that ABX is meaningful for our kind of tests. At least I for myself am not really interested in being able to identify a certain unit, I would like to use it and make good recordings with it. And for that I need to have an impression about how it sounds, and what its strenghts and weaknesses are, compared to the other options available.

The main point seems to me to be to make sure that we are not fooled by "louder is better", different mic placement, and different sounding mics. I will rotate the preamps in relation to the mics, and we might even have RMS as well as peak leveling, or whatever seems most reliable to someone.

And yes, the mind has a lot of tricks to fool us (and in my experience preferences as well as emotional attachments belong to the mind), so I am thankful for SoSueMe's offer to prepare the files for us.

BTW, one thing I often do to check if I'm fooled by "0,1 dB louder = better" is to make one of the files louder or softer in 0,1 dB increments, and listen carefully to the perceived changes. That can even be revealing with bit-identical files. It's good listening training. The perceived differences are often surprising, and not always predictable or explainable.

IMO it would be great and much more valid than the multiple mic method due to the possible issues I mentioned.

However, there's the arguments that will ensue like the ones I mentioned and the "cumulative effect" of the original sound sources altready having run through preamps and other studio equipment. FWIW I believe you most definitely can still realize all the benefits/attributes of the preamp used (or other equipment) when re-recording the source.

EDIT: Nice monitors, by the way!

Remember guys and gals, audio can and often should be fun! Just a thought..

Perfection!

btw Rumi, I also like the idea of reamping an instrument or a mix (especially thru those big beautiful ATCs!), but IMO this should be done in addition to a non 'transducer' sound source, that radiates freely in a room.

The reamp test will definitely help i think! Consider it another layer of data.

It reminds me of a studio in San Diego that had a player piano (a Yamaha that played Bach off a CD under the keyboard), and I used it to shootout pres and converters by myself! I used ORTF Schoeps -> Buzz MA2.2 (no trannie)/Martech/GML/Avalon 2022 (VERY transparent class A)/ULN2 (mk1) -> mytek/avid/apogee/lavry blue (this was before Burl existed). I had the place to myself so that player piano was essential. 6 hours of shooting out all that gear by myself while listening to Bach on a Yamaha C7....

I ended up keeping the Buzz and Mytek... and bought the ULN2 for field recording. The Myteks sound shockingly natural on piano - arguably the best jazz converters on the planet.

I agree. Another layer of confirmation.

You're just one state away from me. I'll pay shipping back and forth and have the Mytek back to you within a couple of days so I can see for myself and anyone else who's interested if there's really a difference between it and my cheap Steinberg unit.

Yes, it really should be.

Personally I don't find significance testing fun at all. I do it out of necessity. I'm a skeptic by nature and I'm not rich. Those two things, as well as the price of high end audio equipment, are the main reasons why it's necessary for me.

I'm not going to drop a couple of grand on a converter, for example, unless I'm absolutely certain that any perceived differences in it compared to the next converter are actually real. I've done enough testing over thirty years to know that it's extremely easy to be fooled. I've been shocked too many times after the fact when finding out later what I thought was a big difference was no difference at all. The only way to know for sure (assuming the differences between equipment are not obviously large) is properly implemented significance testing.

Some people may think I'm nuts for saying many converters sound the same (I'm not talking about specs; I'm talking about what we *actually* hear from them -- not what we *think* we hear from them), but in my mind I'm not completely wasting thousands of dollars by flushing it down the toilet.

That said, I'm not knocking others ideas on the matter. Like Rumi says, it can get ridiculous and there are many scenarios where I don't need to do it at all, and I'm glad not to in those cases so I can concentrate on the music and have fun. BTW, the only time I do testing like this is when I'm considering a purchase and I feel it could be wasted (more common than you might think).

SoSueMe, I just this last week sold 24 channels of Mytek 192 DSD to GearSlutz member: Avast (the owner of Avast studios in Portland). Or else I would have shipped you one of the units for a quick test!

I sold my multitrack converters to be replaced by mastering converters. The Myteks were so good that I almost kept them for mastering but I didnt need that many channels!

I now use Burl B2 ADC, Benchmark DAC2, Crookwood DA & AD (soon!) and Black Lion Avid HD. I just love having all those colors for mastering - neither one of them is 'better' than the other at this level.

I can send you the Benchmark DAC2 for a quick test (you pay for shipping ok?). Total transparency. It a good 1 or 2 steps above the old DAC1.

BTW, how come you are using high end mics (DPA 4006 and Schoeps) and high end pres, but with prosumer Steinberg converters? Or do you use another converter? I am not being rude, just curious. : / Maybe the Steinbger is a sleeper unit that is as good as the Forssell, Mytek and Benchmark?

OK, back on topic: pre amp shootout!

PMed!

...and there is nobody here who has actually had the unit in close proximity to test, and not used a screwdriver and just took a look inside?

Another THAT 1512 (INA 217-SSM 2019) based pre-amp? If it would be fully discrete I would have expected a bit more about it in the description. It has not made me curious so far.

Knowing Scott, I would be *very* surprised if it was any sort of an opamp/chip based design. I've known him for probably 15 years and when he's modded gear in the past (what he was doing before he started Pueblo), one of the first things he did was replace or bypass chips and opamps...

I do own a set of them, and I haven't opened them up because I feel there really is no need to. He is a bit secretive about the design because the things he's done in it are pretty unique. When you have pros at the level that have been buying this gear jumping in, there are two things that really matter. 1. What does it sound like. 2. How well is it built and how well will it work in real-world situations.

I can tell you from experience that it passes both of those tests with flying colors. Its brand of transparency may or may not be your cup of tea. That's a subjective judgment that can go either way based on personal taste.

I can tell you that I've put it in situations where I've heard a room I know very well (8 years of concert recording in the same space) like I had heard it for the first time. I've put it on the front end of microphone splitters and it has no issue driving hundreds of feet of mic cable that is on the other side of a low-impedance transformer. (I've had other major name gear fail that test). I've had them on with mics plugged in and had to wonder if signal was present because the noise floor is essentially non-existent.

Before you start slagging on a design, it might pay for you to actually know something about it.

--Ben

I am meeting Scott on monday to pick up 2 channels.

I won't open it up and take photos (because I have no interest), but I will hook up a pair of Schoeps to record acoustic guitar through it and then post the files here...

I'll shoot for wednesday.

Therefore I have asked here what the design idea is behind the box. There are very good op-amp based designs, no slagging at all, think about Millennia, one octal transistor stage, followed by op-amps,very good preamp. Nothing against INA217 or THAT 1512, think about DAV. But it surprises me if you use that box, you have not asked more in detail.

I use my ears, not my eyes, when deciding to spend money on gear for tracking, mixing and mastering.

However, when buying cooking appliances I may use my eyes to make sure it'll look good in my kitchen, but that says quite a bit about my skills as a chef.