The No.1 Website for Pro Audio
 All  This Thread  Reviews  Gear Database  Gear for sale     Latest  Trending
Why Shoot Outs are not fool proof.
Old 6th June 2018
  #1
Lives for gear
 

Why Shoot Outs are not fool proof.

Here's a good example of why casual shoot outs don't work in comparing the sonic value of two products:

Compare two of the exact same speakers on a switcher, but one is 1/4 dB louder- you hear no level difference but you DO hear more bass and treble. The more you switch it the more you hear the bass and treble is "more" than the other speaker. You know intellectually its the same speaker but now you're thinking these are not a pair- one is defective, its lacking bass and treble. You can really get yourself convinced.

Then swap them L/R without changing anything. Now its the other one with more bass and treble! So now you are thinking, in trying to explain this, they are different alright buts its the friggin' source channels, they sound different...... There's something wrong with the monitor controller or the mixer or the amp! Oh my God, how will I chase that down? So you meticulously check each item in line, get other products you have and check those, still unsure where this difference is coming from. Arrggghh!!

All the while its a mere 1/4 dB level difference between channels. The mind is easily fooled in audio!
Old 6th June 2018
  #2
Lives for gear
 

The 'Mind' is not fooled ... the calibration for the test was flawed.
Old 6th June 2018
  #3
Lives for gear
 

Quote:
Originally Posted by RJHollins View Post
The 'Mind' is not fooled ... the calibration for the test was flawed.
Ah, the mind is perception, perception is reality.....
Old 6th June 2018
  #4
Snap decisions don't always work out. George Massenberg said it best, you need to live with a piece of gear for some time to determine it's true value.
Old 6th June 2018
  #5
Lives for gear
 

Quote:
Originally Posted by Jim Williams View Post
Snap decisions don't always work out. George Massenberg said it best, you need to live with a piece of gear for some time to determine it's true value.
Exactly. In reading about some shoot outs here plus having organized a few, I know first hand about how they can work and how they can fail. We did one for Telefunken- very precisely calibrated comparison of their new 251, U47, C12 vs a "best of the best" vintage versions that took a bit of effort to obtain. The purpose of the shoot out was to say these new Telefunken mics are not far off the best of the best vintage mics and you can trust them. It was not to prove they were better or beat the vintage mic. Anyway, we have two different singers male and female sing through all them. I watched as people in the control room sitting way off axis to the monitors picked differently than the on axis folk. I watched as people chose differently based on music- then on the timing of the switch between mics, then on what was louder passage vs what was softer one...it was variation in results based on conditions other than the mics despite careful calibration. I thought it was an extremely close comparison but some listeners thought one mic clearly "blew away" another. Wow, didn't hear that! So that's why I say there are SO many factors in a shoot out that are hidden or at least not obvious, that living with something and giving it time is the only way.

Brad
Old 6th June 2018
  #6
Shoot outs are completely fine to understand what your initial preference is towards different equipment and how it compares. And how it sounds, "off the hip"...It is determining only one thing....how things "compare", and which "things" you want to check out later in the Real World. I find most Pimps and Pushers tend to get all wishy washy about "how it compares"...Errr...Which has maybe lead to this place becoming a non-stop spin cycle of "how things compare"...Personally, I don't give two turds how things compare because its completely irrelevant in my world. Its about what I like, out of the comparisons. It has Zero, Zilch, Nada relevance to where I think each tool would be good and when. So, I disagree that a pro cannot determine a preference from a shootout, regardless how bush-league it was setup. Whether or not this leads to the User preferring one thing or another is also totally irrelevant. Assuming all pro's know you have to work with the gear to actually know it and where your preference lands.

Last edited by Doc Mixwell; 7th June 2018 at 11:38 AM..
Old 6th June 2018
  #7
Lives for gear
 

Quote:
Originally Posted by Doc Mixwell View Post
<SNIP> So, I disagree that a pro cannot determine a preference from a shootout, regardless how bush-league it was setup. <SNIP>
I don't think anyone is saying you cannot determine something for yourself-what I'm saying is shoot outs run by others are suspect. You can be fooled. Plus results like "this $500 speaker beats this $10,000 one! Everyone agrees!" should be completely disregarded.

Brad
Old 6th June 2018
  #8
Quote:
Originally Posted by Brad Lunde View Post
I don't think anyone is saying you cannot determine something for yourself-what I'm saying is shoot outs run by others are suspect. You can be fooled. Plus results like "this $500 speaker beats this $10,000 one! Everyone agrees!" should be completely disregarded.

Brad
I have never been "fooled" by someone elses shootout. because from the very first moment I choose to check it out, I understand it is just their engineering.

No one has ever fooled me this way. Sorry, but no
Old 6th June 2018
  #9
Lives for gear
 

Quote:
Originally Posted by Doc Mixwell View Post
I have never been "fooled" by someone elses shootout. because from the very first moment I choose to check it out, I understand it is just their engineering.

No one has ever fooled me this way. Sorry, but no
You are the brilliant example then of one whom this thread does not apply to!

Brad
Old 7th June 2018
  #10
Lives for gear
 

Brad is spot on. Every shootout I've been invited to I always take with a grain of salt. You can never put 2 pairs of monitors in the same place and you can never guarantee that things are calibrated correctly. It's a broad take at best though you can get some insight out of the moment (for example the first time I head ATC 150s. ). Mics are tough because of placement and calibration next to impossible. I don't bother listening to posted shootouts.
Old 7th June 2018
  #11
I've found shoot outs useful for eliminating contenders. Bad stuff always sounds bad no matter how long you live with it. Shoot-outs are like the game of Clue.
Old 8th June 2018
  #12
Lives for gear
 
Will The Weirdo's Avatar
I start with the belief that every test outside of testing it yourself is a flawed test IMHO, lol.

I do appreciate online shoot outs, but I enjoy shootouts for what they are, a simple comparison that will only help me determine if one tool is worth the time of me testing it for myself. They only help weed out the contenders as Jim put it.

Microphones and monitors are the hardest to determine their value as a tools in shoot outs too, I need real time with each to determine if it fits my workflow and needs.
Old 8th June 2018
  #13
Lives for gear
 
Lance Lawson's Avatar
 

Shootouts? A few I've seen were spot on. Some gear is so close in performance as to be a matter of taste.
Post Reply

Welcome to the Gearslutz Pro Audio Community!

Registration benefits include:
  • The ability to reply to and create new discussions
  • Access to members-only giveaways & competitions
  • Interact with VIP industry experts in our guest Q&As
  • Access to members-only sub forum discussions
  • Access to members-only Chat Room
  • Get INSTANT ACCESS to the world's best private pro audio Classifieds for only USD $20/year
  • Promote your eBay auctions and Reverb.com listings for free
  • Remove this message!
You need an account to post a reply. Create a username and password below and an account will be created and your post entered.


 
 
Slide to join now Processing…
Thread Tools
Search this Thread
Search this Thread:

Advanced Search
Forum Jump
Forum Jump