The No.1 Website for Pro Audio
 Search This Thread  Search This Forum  Search Reviews  Search Gear Database  Search Gear for sale  Search Gearslutz Go Advanced
New ABX blind-testing software...
Old 7th April 2014
  #1
Gear Nut
 

Thread Starter
New ABX blind-testing software...

--------------------------------------------------
UPDATE: version 2 released... see below...
--------------------------------------------------
UPDATE: version 2.1 released... see below...
--------------------------------------------------
UPDATE: version 2.2 released... see below...
--------------------------------------------------
UPDATE: version 2.3 released... see below...
--------------------------------------------------

(Sorry if this is the inappropriate subforum for this -- mods please move if so -- it seemed appropriate to me...)

I just released a new ABX double-blind testing tool for all of your brutally-honest gear testing needs... I was tired of not being able to find one that worked the way i wanted it to, and worked cross-platform, and was still being developed, etc.

You can check it out at: ABX audio testing tool

Let me know if you have any trouble with it. And let me know what you'd like to see added. I plan to add the ability to play certain sections of files rather than always starting at the beginning. And depending on interest/support i might add the ability to have an arbitrary number of files to choose between (ABCDE...X)

It runs on linux, windows, and mac (64bit only for the mac, sorry.)

Just download it, extract from the archive, and run the executable, and it should be good to go.

It's in Java, but there's no need to install anything: the download files are a bit large because they include everything it needs to run with them. Tradeoffs, you know.

If you have a 64bit Mac -- please let me know if it works for you. I haven't been able to find one to test on.

Thanks, enjoy!

Old 20th April 2014
  #2
Gear Nut
 

Thread Starter
Hi -- proud to announce that version 2 has now been released: ABX audio testing tool

Adds:
- playing certain ranges in the files only, looping
- gain/offset adjustment
- blind shootouts (files are shuffled, you choose your favorite)
- ABCDE....X trials (ABX with more than two files)
- many UI improvements

Hope you like it!

Old 20th April 2014
  #3
Lives for gear
 
feck's Avatar
Cool idea!
Old 20th April 2014
  #4
Old 20th April 2014
  #5
Lives for gear
 

Looks good ... will check it out !
Old 20th April 2014
  #6
Gear Guru
 
Ethan Winer's Avatar
 

Lightbulb

Quote:
Originally Posted by chconnor View Post
Hi -- proud to announce that version 2 has now been released.
Excellent, thanks for the update, I'll check it out (and answer your email too) when I get a chance.

--Ethan
Old 20th April 2014
  #7
Nrt
Lives for gear
 

Thank you for making Mac version!
Old 20th April 2014
  #8
Gear Nut
 

Thread Starter
Quote:
Originally Posted by Nrt View Post
Thank you for making Mac version!
Sure! But you can thank Java, really. :-)

May I ask, as i've been having some issues with users not getting it to work on the Mac -- did it actually work for you? And what version of OS X are you using?
Old 20th April 2014
  #9
Lives for gear
 
TREMORS's Avatar
Quote:
Originally Posted by chconnor View Post
Sure! But you can thank Java, really. :-)

May I ask, as i've been having some issues with users not getting it to work on the Mac -- did it actually work for you? And what version of OS X are you using?
I have an irrational dislike for JRE on my computer but tempted to try this.
Old 20th April 2014
  #10
Gear Nut
 

Thread Starter
Quote:
Originally Posted by TREMORS View Post
I have an irrational dislike for JRE on my computer but tempted to try this.
In case you didn't see it on the webpage -- you don't have to actually install a JRE. The all-in-one package includes a JRE with it that doesn't need to be installed anywhere, so whatever your concerns I think that should assuage most of them?
Old 22nd April 2014
  #11
Gear Nut
 

Thread Starter
Version 2.1 just released -- now you can switch from file to file without starting over again at the start... the "Hold pos." button maintains the current seek location when changing.
Old 22nd April 2014
  #12
Gear Addict
 
david1103's Avatar
 

Thanks!
Old 22nd April 2014
  #13
Gear Head
 

Cheers!

I'm not certain that there isn't any, but I never found found this for Linux before.

Mind you, blind or not, AB[X] testing always drives me to distraction, spotting differences that... no, wait, that is there on the other sample too, I just didn't hear it... and so on until I just give up and go back to listening to music.

Still, this is going to be a useful tool, and if I can discipline myself to get on with it, I hope to start by [dis]proving some of my standpoints on sample rates.
Old 22nd April 2014
  #14
Gear Nut
 

Thread Starter
Quote:
Originally Posted by Thad E Ginathom View Post
I'm not certain that there isn't any, but I never found found this for Linux before.
There are a couple out there (e.g. this), but they're not especially easy to find or use or especially featured or supported, yeah. It frustrated me that people would say in forum posts "just do a blind ABX test" but when you actually went to do it it was such a pain to find a tool (which also worked, on your OS, supported the file types, had the right design and features, etc.) And is there any blind shooout software out there?

I was hoping this could be a "just use this" solution. The download file size is a minor bummer, but less so these days, i'm hoping.

Quote:
Mind you, blind or not, AB[X] testing always drives me to distraction, spotting differences that... no, wait, that is there on the other sample too, I just didn't hear it... and so on until I just give up and go back to listening to music.
I hear that. It's not always totally clear how/when one should best use such a tool, and it does take some discipline (and time and focused attention). I personally go around most of the time assuming that i can't hear a difference, and most often use ABX to prove that to myself, as opposed to the traditional opposite goal. :-) That way i spend more time listening to the more important aesthetic stuff.

Quote:
Still, this is going to be a useful tool, and if I can discipline myself to get on with it, I hope to start by [dis]proving some of my standpoints on sample rates.
Awesome. That's why i wrote it. :-)
Old 22nd April 2014
  #15
Gear Head
 

Quote:
I was hoping this could be a "just use this" solution.
Me too...

Attempts so far have not worked out. Apparently FLAC is not in the list of supported formats as per your website link? Conversion (using sox) to WAV, and trying that, I go the error message that compressed WAV was not supported. I didn't compress it. Honest! Well, not knowingly. I guess this is likely to be a problem with my knowledge of file formats, rather than your software.

Quote:
I personally go around most of the time assuming that i can't hear a difference, and most often use ABX to prove that to myself, as opposed to the traditional opposite goal. :-) That way i spend more time listening to the more important aesthetic stuff.
This is a pretty good approach. In fact I try to forget all about, for instance, what kind, or sample/bit-depth, etc, file I am listening --- unless something is obviously, audibly wrong, and then it becomes a trouble-shooting job.
Old 22nd April 2014
  #16
Lives for gear
 
JPeters86's Avatar
 

This is very nice, thanks! Now we can finally prove that all software EQ's sound the same heh
Old 22nd April 2014
  #17
Gear Nut
 

Thread Starter
Quote:
Originally Posted by Thad E Ginathom View Post
Attempts so far have not worked out. Apparently FLAC is not in the list of supported formats as per your website link?
That's correct. The underlying libraries in use here leave a lot to be desired, but they support mp3 and aac, and uncompresed wav and aiff, so they seemed like a good compromise, despite mkv or flac support (which, wtf? the underlying libraries themselves use GStreamer and this guy demonstrates how easy it is to add MKV support which is left out for ... no reason at all, apparently).

Quote:
Conversion (using sox) to WAV, and trying that, I go the error message that compressed WAV was not supported. I didn't compress it. Honest! Well, not knowingly. I guess this is likely to be a problem with my knowledge of file formats, rather than your software.
If you can get an example wav file to me that is failing i can check that out... It certainly shoul read any wav container containing uncompressed audio in it.

Thanks,
-c
Old 22nd April 2014
  #18
Gear Head
 

Thanks. I am no expert at this stuff, so I'll look over the sox options again first. I might have made a blunder, although the basic functions are usually easy with sox
Old 24th April 2014
  #19
Gear Nut
 

Thread Starter
Version 2.2 now out... besides the standard little UI tweaks to make it better, it adds drag-and-drop support: drag a file (or group of files) to the program for easy setup of your tests. If you drag a single file to a particular row, it will replace that file. Otherwise it will insert them.
Old 24th April 2014
  #20
Lives for gear
Quote:
Originally Posted by JPeters86 View Post
This is very nice, thanks! Now we can finally prove that all software EQ's sound the same heh
Actually, I think this software is going to support the claims a lot of "listeners" have been making about software comparisons.

But I predict that the "scientists" who always claim there can't be differences (the "digital is digital" argument) will just make new "scientific" arguments to discredit the results created with this new app.

All this software will do is take the insane arguments to a new level of insanity. Most "scientists" on gearslutz are unlikely to accept anything the "listeners" are claiming even if the listening results are scientifically verified with an app like this.

Their "scientific" dogma is that their technical explanation proves people can't be hearing anything, so any listening results must therefore by definition be flawed. It's a little bit like saying "we know that cholera is caused by miasma so even though you have statistical observations showing outbreak clusters near wells, it can't possibly be transmitted by water because you don't have a theory showing how that transmission would actually take place."
Old 24th April 2014
  #21
Gear Nut
 

Thread Starter
Quote:
Originally Posted by MarsBot View Post
Actually, I think this software is going to support the claims a lot of "listeners" have been making about software comparisons.

But I predict that the "scientists" who always claim there can't be differences (the "digital is digital" argument) will just make new "scientific" arguments to discredit the results created with this new app.

All this software will do is take the insane arguments to a new level of insanity. Most "scientists" on gearslutz are unlikely to accept anything the "listeners" are claiming even if the listening results are scientifically verified with an app like this.

Their "scientific" dogma is that their technical explanation proves people can't be hearing anything, so any listening results must therefore by definition be flawed. It's a little bit like saying "we know that cholera is caused by miasma so even though you have statistical observations showing outbreak clusters near wells, it can't possibly be transmitted by water because you don't have a theory showing how that transmission would actually take place."
Not to open a can of worms, here, but: I think it depends a lot on what exactly is being claimed by whom, and, in turn, how the tool is used.

This software can identify when you can and when you can't identify a particular file in comparison to another file. If you're making a claim that strongly implies that you can or requires you to be able to discriminate between two files, it can tell you if you're actually hearing something or if you're fooling yourself. That's not really subject to debate: if you say "I can tell A from B" and I say "Ok, which is playing now, A or B?" and you can't tell me with any statistical significance, that is meaningful, and I don't think it's fair to dismiss is as some kind of myopic science dogma.

But it's true that there are a lot of ways it can be misused. Results can easily be faked. Or someone might equalize the gain on various files but unknowingly make one a little louder and then pick that one repeatedly in a test. There are also a lot of situations in music production where it is not trivial to isolate the debated factor: if a complex production supposedly benefits from analog summing during intermediate tracking, it might not be practical to come up with an A and B where the only difference is the summing used in the middle of the production, even if you plan ahead.

And let's say that we do see some surprising results, like a lot of "non-scientists" post that they can hear the difference between oxygen-free copper and a clothes hanger. As explained on my site: "if you post two totally identical files and 300 people test them in a trial of 20 comparisons, 15 of those people will likely get 15 or more correct with a 95%+ confidence result, purely by chance." Those 15 are sure to post their results, and most of the others won't. In other words, you can't infer things from the results posted on the internet unless you account for all the tests that happened.

Or people will take tests over again and again until they randomly get a good result (perhaps not maliciously: not everyone understands the statistical fallacy that represents, and they might think that they got the good result because they finally learned what to listen for.)

In the end, I don't actually think a tool like this is automatically useful for online debates: there are too many ways to fake the results, too many ways to test poorly, and too many ways to misinterpret the results. I think it could be a useful tool for debates if used in a controlled environment, or if users are very clear (and verbose) about how they did their testing. (Perhaps we need a protocol: assured level-matching, a certain number of practice listenings, no more than X trials, guaranteed posting of negative results, etc.)

So I think if you see people posting things like "hey, I can tell the difference between 320mbps MP3 and 16/44.1 PCM", you are indeed likely to see a lot of rebuttal from "the scientists" on the grounds I laid out above.

I would hope that if negative results are posted ("nope, they sound identical to me") that "the scientists" would make the same contrary points about the credibility of the results, but, human nature being what it is, maybe not.

My main goal for the software is as an educational tool: so people can prove or disprove things for themselves. I do personally believe that 90% of the stuff audiophiles worry about is mythology, but my agenda isn't to make a tool to prove that point, just to give us some convenient way to check ourselves.

If you've ever had the experience of gaining up a track and saying "oh yeah, that's much better" and then realized you gained up an empty track by accident, then you are human like the rest of us, and you can benefit from this type of tool. :-)
Old 25th April 2014
  #22
Lives for gear
Don't get me wrong. I think the app is great. If nothing else, people can check themselves. I do believe that cognitive bias exists and can impact audio evaluations. I just don't think it is as pervasive or as immutable to compensation as a lot of the "scientists" do. And I believe the jury is still out when it comes to that. Maybe somebody will be able to do some organized testing using your app.
Old 25th April 2014
  #23
Thanks for this! Been using the ABX tester in FB2K for many years, but this has some nice extra features and is a little easier to use.
Old 27th April 2014
  #24
Gear Guru
 
Ethan Winer's Avatar
 

Lightbulb

Quote:
Originally Posted by chconnor View Post
I think it depends a lot on what exactly is being claimed by whom, and, in turn, how the tool is used. This software can identify when you can and when you can't identify a particular file in comparison to another file.
I agree. As a "scientist" I'm totally willing to change my opinion in light of compelling evidence. If someone proves using an ABX tool like your program that they can hear stuff I previously considered inaudible, I'll gladly take that as useful new information!

--Ethan
Old 27th April 2014
  #25
Gear Head
 
bandpass's Avatar
 

Quote:
Originally Posted by Thad E Ginathom View Post
Thanks. I am no expert at this stuff, so I'll look over the sox options again first. I might have made a blunder, although the basic functions are usually easy with sox
There are 2 variants of PCM wav. So try one of:
Code:
sox in.flac out.wav
sox in.flac -t wavpcm out.wav
Old 28th April 2014
  #26
Gear Head
 

Oh yea! Thank you bandpass. It is quite likely that, even with hours of RTFMing (and sox has loads of FM!) that I would never have understood that. wavpcm does the trick

The software is reading the files, and it is playing music, which I can hear. What I need now is to specify the ALSA device. I can play through JACK, using ALSA bridge (KXStudio, Cadence, etc) but that is no good, as the sample rate is fixed in JACK. I need to leave out JACK and play direct to the device.

Sorry if I am not making much sense.

EDIT... some progress. With google's help, I found the sound.properties file. Looks like this is where I need to be. Now I just need to specify my hw:DAC,0 playback device.

I know nothing about java, not even hello world. Yes, I am trying to find the answer for myself, but spoon-feeding, as with bandpass's vital hint, is still welcome

nb: I prefer not to touch the .asoundrc file, because it makes JACK and its associated stuff work.

nb2: my system is Ubuntu 12.04 based, but I do not use Pulse Audio.
Old 28th April 2014
  #27
Gear Nut
 

Thread Starter
Quote:
Originally Posted by Thad E Ginathom View Post
Oh yea! Thank you bandpass. It is quite likely that, even with hours of RTFMing (and sox has loads of FM!) that I would never have understood that. wavpcm does the trick
Yeah I briefly glanced at the sox man page but couldn't find any information on whether PCM would be the default in an MP3->WAV conversion (which one would expect, but apparently wasn't the case...)

Quote:
The software is reading the files, and it is playing music, which I can hear. What I need now is to specify the ALSA device.
One thing I can tell you is that, unfortunately, the program won't change the sample rate on playback (there is no such supporting code in the library I'm using... even if there was, it would have to be addressed with a lot of code, since devices take some time to switch rates and you'd have to ensure that that didn't tip off the listener), so if you want to compare different sample rates you'd have to set the rate of the device to the highest you want to hear and accept the fact that it will upsample the lower ones on playback... that might be good enough, but it's maybe not a scientifically-satisfying way to go.

But if you're just trying to get the rate in general, then... in Windows at least you have to set it through the control panel interface... I'm afraid i don't know how to do this in Linux, and it's likely different in different setups... So maybe you can set it in JACK and accept that it isn't going to change from file to file.

The sound.properties file may or may not apply; I think it may apply to the older java media framework which I'm not using, but if it has an effect, please let me know! I'm starting to wonder if I should have used the older framework... I used the new one because it gives access to a wide array of formats (mp3, aac, etc) but the older framework would have given me access to actual audio data and would have enabled some other things (auto-gain leveling, possibly setting sample rate on the device?, etc.)

Quote:
nb: I prefer not to touch the .asoundrc file, because it makes JACK and its associated stuff work.
You could always back it up first and then poke around in it. :-)
Old 29th April 2014
  #28
Gear Head
 

Quote:
One thing I can tell you is that, unfortunately, the program won't change the sample rate on playback
Nor should it! It may only be maths, but, like people, some seem to be better at it than others. I stopped using Aqualung for mixed-sample-rate listening, as I found its SRC (both up and down) to be very poor. Now I play direct to the DAC/interface and let it sort out what it is receiving. One knows pretty quickly if it gets it wrong!

However, I need to set that device for playback ---much as, I suppose, one does in Windows--- in the playback software.
Old 29th April 2014
  #29
Gear Nut
 

Thread Starter
Quote:
Originally Posted by Thad E Ginathom View Post
Nor should it! It may only be maths, but, like people, some seem to be better at it than others. I stopped using Aqualung for mixed-sample-rate listening, as I found its SRC (both up and down) to be very poor. Now I play direct to the DAC/interface and let it sort out what it is receiving. One knows pretty quickly if it gets it wrong!

However, I need to set that device for playback ---much as, I suppose, one does in Windows--- in the playback software.
What I mean is that Lacinato ABX (as with 99.999% of music players, maybe even 100%?) will play at whatever sample rate the device is set at... so if the interface is at 44.1 it doesn't matter what sample rate the file is at, because the operating system and the interface will change it to 44.1 on the way out. To have the music player change the playback sample rate of the interface you have to negotiate with the audio device driver (or the OS API?), as e.g. Reaper does.

The bummer is that it's harder to compare sample rates... if you have a 44.1k and a 96k file you can't really do a blind test with any extant software that i know of... switching the device from one rate to another usually takes a little time, sometimes makes little noises, etc, and those would have to be guaranteed the same whether going A->A, A->B, B->B, or B->A. Alternately, you can set the device to 96k and let it upsample lower rate files, which, given the sampling theory, I would imagine is virtually transparent, as opposed to downsampling.

-c
Old 29th April 2014
  #30
Gear Head
 

Oh, I see... I'm sorting-of hoping that ALSA and the device will sort it out. I'll let you know if I have any success. Linux audio is a bit of a minefield!
Loading mentioned products ...
New Reply Submit Thread to Facebook Facebook  Submit Thread to Twitter Twitter  Submit Thread to LinkedIn LinkedIn  Submit Thread to Google+ Google+  Submit Thread to Reddit Reddit 
 
Post Reply

Welcome to the Gearslutz Pro Audio Community!

Registration benefits include:
  • The ability to reply to and create new discussions
  • Access to members-only giveaways & competitions
  • Interact with VIP industry experts in our guest Q&As
  • Access to members-only sub forum discussions
  • Access to members-only Chat Room
  • Get instant access to the world's best private pro audio Classifieds for only USD $20/year
  • Promote your eBay auctions and Reverb.com listings for free
  • Remove this message!
You need an account to post a reply. Create a username and password below and an account will be created and your post entered.


 
 
Slide to join now Processing…
Thread Tools
Search this Thread
Search this Thread:

Advanced Search
Forum Jump