TEST: 5 DAWs 5 MBP Optical O/P.

[Radiogal]

Monday 2011-11-14 The scene: Soundtrade Studios in Stockholm. Live room.

The DAWs: ProTools 9, ProTools 10, Logic 9, Cubase (latest) and Digital Performer 7.24.
All installed on 5 equal Macbook Pros running optical O/P to a switcher to two Genelecs.

A 30 tracks multitrack project, all with equal panning and gain. (pan law etc were carefully compensated for).
The difference on the masterbus O/P between these DAWs (1 kHz test tone) was 0.2 dB from the highest to the lowest difference but that was also compensated for.

We tested 2 different projects. One tight rock song (the multitrack)
30 MONO tracks in a project with tracks panned and faders gained all the same on each and every DAW.

And one jazzy more open song. The jazz song was a stereo bounced mix file.

No plugins whatsoever.

This test took 2,5 hours and the material was tested on 30+ members of the Swedisch Sound Engineer Society. All professionals.
We compared different looped material in sections, played on MBP nr 1,2,3,4 and 5 and even randomly.

The original projects were recorded 96/24 in PT and converted to 48/24 in PT and this files were imported to the DAWs.
The test also included a listening to 6dB masterbuss overdrive to see how that was processed internally in each DAW.

After every listening part we all got to say what we prefered and why.

The verdict:
The MBP nr 1 always was the most liked and than sometimes also in combination with nr 4 and nr 5.
MBP nr 3 sounded "compressed". Most bass.
MBP nr 2 nobody liked this one really. Only occasionally by a few persons. Sounded flat and pressed.
MBP 4 and 5 didn´t stand out, sounded equal and recieved some likes on a piano part and some dislikes in other parts of the music.

I was one of those that listened and didn't know what DAW was played on what MBP.
I heard the engineers say/yell "nr 1 is the best" all the time. I agreed.
It had the most depht, best definition and fullest sound. It became really noticable.



Well. After 2,5 hour of listening we finally got the results.

MBP 1= Digital Performer
MBP 2= ProTools 9
MBP 3= Logic 9
MBP 4= ProTools 10
MBP 5= Cubase (latest version)

A lot of jawdropping amongst the engineers... "Whut?????!!!"
DP was by far the most liked in this DAW sound quality test!

The Swedish Sound Engineer Society claims this DAW test is not to be a scientific research in any way.
However the goal was to find out if there is any sonical difference between DAWs.
The answer to that was undoubtably: Yes! That we all agreed on.
Not an enourmous difference but still significant for the trained ears that attended this test.

So now you know

[Kubilay]

Do dig this, especially the reported jaw-dropping after the reveal.

[PaPi61]

I heard the engineers say/yell "nr 1 is the best" all the time. I agreed.

This little experiment only proves how strong the power of suggestion is.
People who took the test should not have been influenced by anyone.
I wouldn't be surprised if the Swedish distributor of Digital Performer secretly sponsored this event.

[Dot.]

interesting test, however i dont really believe in it to be accurate. we all hear things different, and I tried doing a/b tests before, more you listen to same thing harder to tell difference 1s your ears get used to it, unless the difference is huge.

as i said interesting idea tho im just surprised nobody liked pro tools 9, i use it and i like it.

[mikeka]

Originally Posted by PaPi61

This little experiment only proves how strong the power of suggestion is.
People who took the test should not have been influenced by anyone.

Good point if everyone really was just "yelling" out their choice at the same time! Dunno how the test was actually conducted but one would think everyone would submit their choice privately.

Originally Posted by PaPi61

I wouldn't be surprised if the Swedish distributor of Digital Performer secretly sponsored this event.

Not so. The OP asked for DP to be included in this already scheduled shootout.

This is interesting to me as I've been under the impression that DAW software is basically created equally soundwise, (workflow and feature set notwithstanding) and that it's more about the converters used.

[Radiogal]

Originally Posted by mikeka

Good point if everyone really was just "yelling" out their choice at the same time! Dunno how the test was actually conducted but one would think everyone would submit their choice privately.

Need to say there SSES members in the Swedish Sound Engineer Society are for sure no "yeah sayers". All these members are pros and have strong personalities, intregaty and really tell what they mean, like and dislike.
There wasn´t anything to fight for really. We were asked what we experienced after each listening. We told what we heard. Different experiences.
But some where talking louder cuz at that time they became sure what they heard and liked/ disliked.
Nevermind how you turn it, the results speaks for itself.

[klaukholm]

edited, I am out of the mudslinging

[ARIEL]

Fascinating test done ! You didnt record the stereo outs of each Mac book into a stand a lone DAW , would love to of been there . Well at least Cubase was flat I use Cubase .

[engelen62]

You will get a group mentality no matter how strong the personalities and how professional the listeners are. Being in the same room is enough to ruin the objectivity. The power of group psychology makes this test non valid.

Another point is that from the picture the listening is far from ideal for anyone.

I dont see how this speaks for itself in any way.

[Radiogal]

Sorry guys. No matter what you all say and complain about this and that bla bla. the fact is this is what we did and this were the results. No scientific as stated in my first post. take it for what it is. i know what I heard and so did all the others. Sorry you weren´t there with all your questions. I can tell you that many of your questions came up in the Live room to as these are critical pros. Over and out.

[JonesH]

Yes, a friend of mine participated and called me up afterwards. He was also a bit taken aback by the results. There are of course a lot of flaws that would need to be addressed in order for the test to hold any value beyond anecdotal but it's still interesting.

Radiogal, Do you know how the sync was set up? Did they use different interfaces locked to the sync of the external interface? I presume the 8260:s were run on digital?

[Dom & Roland]

I first saw this test on a motu forum, which is unfortunate, as having not found anything else about it on the web makes me suspicious.

[Timur Born]

Originally Posted by Radiogal

We compared different looped material in sections

Sorry to say, but this might have been a mistake! Looping can be one single source of timing (thus sound) differences (I saw it in practice with Ableton Live). So the better approach would have been to copy&paste the loop a few dozen times and play it in one take straight.

Furthermore I wonder how DAWs can be described as sounding "flat" or "most bass" when the whole thing was only monitored via a single path of amp+speakers? Most music sounds "flat" (aka less hyped) on my monitors while "most bass" on my stereo (or for the sake of it through an simple bass+highs EQ via my monitors). The number of participants should level this one out some though (everyone hopefully is using different speakers at their own studio and thus is used to somewhat different sound).

The latter is not meant to be an argument against "different" sounding sources, but an argument against "best" sounding ones.

[Timur Born]

Originally Posted by Radiogal

And one jazzy more open song. The jazz song was a stereo bounced mix file.

Does this mean the song was bounced in all tested DAWs and then the results of the bouncing was listened to? Or does it mean you got one bounced stereo file, put that into all DAWs and just played it through them (as glorified playback software)?

[JELmusic]

Makes me wonder if a lot of these posts would be similar (the ones 'questioning' the test validity) had Logic or PT been the preferred/chosen DAW in this test.

I have 4 of the 5 DAWs that were tested, and I have to say listening over a number of years (well, PT 8, then recently 9) that the results in this particular test are pretty consistent with my own 'unscientific' testing.

[GoldMember]

Originally Posted by Radiogal

A 30 tracks multitrack project, all with equal panning and gain.



Well. After 2,5 hour of listening we finally got the results.

MBP 1= Digital Performer
MBP 2= ProTools 9
MBP 3= Logic 9
MBP 4= ProTools 10
MBP 5= Cubase (latest version)

So now you know

where i can download the Bounced files?

[oldeanalogueguy]

Originally Posted by Radiogal

Monday 2011-11-14 The scene: Soundtrade Studios in Stockholm. Live room.

The DAWs: ProTools 9, ProTools 10, Logic 9, Cubase (latest) and Digital Performer 7.24.
All installed on 5 equal Macbook Pros running optical O/P to a switcher to two Genelecs.

A 30 tracks multitrack project, all with equal panning and gain. (pan law etc were carefully compensated for).
The difference on the masterbus O/P between these DAWs (1 kHz test tone) was 0.2 dB from the highest to the lowest difference but that was also compensated for.

We tested 2 different projects. One tight rock song (the multitrack)
30 MONO tracks in a project with tracks panned and faders gained all the same on each and every DAW.

And one jazzy more open song. The jazz song was a stereo bounced mix file.

No plugins whatsoever.

This test took 2,5 hours and the material was tested on 30+ members of the Swedisch Sound Engineer Society. All professionals.
We compared different looped material in sections, played on MBP nr 1,2,3,4 and 5 and even randomly.

The original projects were recorded 96/24 in PT and converted to 48/24 in PT and this files were imported to the DAWs.
The test also included a listening to 6dB masterbuss overdrive to see how that was processed internally in each DAW.

After every listening part we all got to say what we prefered and why.

The verdict:
The MBP nr 1 always was the most liked and than sometimes also in combination with nr 4 and nr 5.
MBP nr 3 sounded "compressed". Most bass.
MBP nr 2 nobody liked this one really. Only occasionally by a few persons. Sounded flat and pressed.
MBP 4 and 5 didn´t stand out, sounded equal and recieved some likes on a piano part and some dislikes in other parts of the music.

I was one of those that listened and didn't know what DAW was played on what MBP.
I heard the engineers say/yell "nr 1 is the best" all the time. I agreed.
It had the most depht, best definition and fullest sound. It became really noticable.



Well. After 2,5 hour of listening we finally got the results.

MBP 1= Digital Performer
MBP 2= ProTools 9
MBP 3= Logic 9
MBP 4= ProTools 10
MBP 5= Cubase (latest version)

A lot of jawdropping amongst the engineers... "Whut?????!!!"
DP was by far the most liked in this DAW sound quality test!

The Swedish Sound Engineer Society claims this DAW test is not to be a scientific research in any way.
However the goal was to find out if there is any sonical difference between DAWs.
The answer to that was undoubtably: Yes! That we all agreed on.
Not an enourmous difference but still significant for the trained ears that attended this test.

So now you know

maybe
maybe not

i suspect other factors are at play too
and could change the results
or not but i suspect they would


depending on how things were set up for another try

[valis]

Originally Posted by Killahurts

Sorry, but this is one of the most unscientific statements I've ever seen. Having people listen in different rooms would be like having people look at a painting in different rooms, with different lighting, etc. Whereas having them listen in the same room at different times would never yield a quantifiable result either.

Actually I think he was referring to the interactions on a personal level (expressing opinion, body language, etc). I would agree if that is the case too, as having people keep their reaction to themselves and simply recording results on paper/iPad etc would remove group think to some degree IMO.

[oldeanalogueguy]

Originally Posted by Radiogal

Sorry guys. No matter what you all say and complain about this and that bla bla. the fact is this is what we did and this were the results. No scientific as stated in my first post. take it for what it is. i know what I heard and so did all the others. Sorry you weren´t there with all your questions. I can tell you that many of your questions came up in the Live room to as these are critical pros. Over and out.

yes
those were the results
but are they repeatable
are they meaningful

i would like to see a better controlled and executed
experiment done with those other DAWs and include a couple of low end ones too for giggles

one abx triple blind test had people picking a coathanger with connectors soldered on over some big name cable like mogami or monster

[Killahurts]

Originally Posted by valis

Actually I think he was referring to the interactions on a personal level (expressing opinion, body language, etc). I would agree if that is the case too, as having people keep their reaction to themselves and simply recording results on paper/iPad etc would remove group think to some degree IMO.

Perhaps that was what he was referring to, fair enough. And I agree that written, "blind" tabulation would be better.

Still, the results of this test are interesting, to be sure..

[klaukholm]

edited, I am out of the mudslinging

[klaukholm]

edited, I am out of the mudslinging

[ssiicckkoo]

I wont comment on weather or not I find any validity to this "test". But if a DAW is doing extra processing without my knowledge in an attempt to make my mix sound "better"...then I'll soon be looking for a new DAW. I don't need a calculator giving me hyped numbers. Just give me the sum!

"Spend time on any forum devoted to any Digital Audio Workstation (DAWs) software or music production and you will see users making unsubstantiated claims about the audio quality of this or that DAW. Protagonists will say a given DAW is clearly and audibly superior to another. To be frank, that's just nonsense. Any DAW software that uses at least 32 Bit floating point calculations will be capable of processing audio without introducing unwanted distortions, frequency response alterations or any other unwanted effect that would be 'clearly audible' so as to sway opinion. We call the ability to process audio without making unintended changes 'transparency'. Today, from a transparency perspective all DAW software is created equal. If you do hear some difference then it's coming from a setting, effect or option somewhere. Not from some inherent quality of the 'audio engine'."

[klaukholm]

edited, I am out of the mudslinging

[klaukholm]

edited, I am out of the mudslinging

[Radiogal]

Originally Posted by Timur

Does this mean the song was bounced in all tested DAWs and then the results of the bouncing was listened to? Or does it mean you got one bounced stereo file, put that into all DAWs and just played it through them (as glorified playback software)?

The SSES board did organize this DAW test. I wasn´t involved more than being a member asking the board to include DP in this DAW test.
DP was installed on a SSES owned MBP. I let them use my DP license for this.


They told us the bounce was done in PT and converted to 48/24 in PT. All DAWs got this PT file.

[ssiicckkoo]

Originally Posted by Radiogal

It was not me setting up this test. The SSES board said the bounce was done in PT and converted to 48/24 in PT. All DAWs got this PT file.

I think what he's asking is, was each DAW session mixed down to a single wave file? Or were the multitrack sessions just played back to you all within each DAW?

[Radiogal]

Originally Posted by ssiicckkoo

I think what he's asking is, was each DAW session mixed down to a single wave file? Or were the multitrack sessions just played back to you all within each DAW?

One project (rock) contained the whole 30 tracks multitrack session. All DAWs got these 30 tracks imported.
The were panned and the fadersettings were set equally on all DAWs as explained in the firts post.

The second song (jazzy) was a bounced in PT into a stereo mixfile, this file was imported into all DAWs as explained in my first post.

[HomeProducer]

Originally Posted by Radiogal

Monday 2011-11-14 The scene: Soundtrade Studios in Stockholm. Live room.

The DAWs: ProTools 9, ProTools 10, Logic 9, Cubase (latest) and Digital Performer 7.24.
All installed on 5 equal Macbook Pros running optical O/P to a switcher to two Genelecs.

A 30 tracks multitrack project, all with equal panning and gain. (pan law etc were carefully compensated for).
The difference on the masterbus O/P between these DAWs (1 kHz test tone) was 0.2 dB from the highest to the lowest difference but that was also compensated for.

We tested 2 different projects. One tight rock song (the multitrack)
30 MONO tracks in a project with tracks panned and faders gained all the same on each and every DAW.

And one jazzy more open song. The jazz song was a stereo bounced mix file.

No plugins whatsoever.

This test took 2,5 hours and the material was tested on 30+ members of the Swedisch Sound Engineer Society. All professionals.
We compared different looped material in sections, played on MBP nr 1,2,3,4 and 5 and even randomly.

The original projects were recorded 96/24 in PT and converted to 48/24 in PT and this files were imported to the DAWs.
The test also included a listening to 6dB masterbuss overdrive to see how that was processed internally in each DAW.

After every listening part we all got to say what we prefered and why.

The verdict:
The MBP nr 1 always was the most liked and than sometimes also in combination with nr 4 and nr 5.
MBP nr 3 sounded "compressed". Most bass.
MBP nr 2 nobody liked this one really. Only occasionally by a few persons. Sounded flat and pressed.
MBP 4 and 5 didn´t stand out, sounded equal and recieved some likes on a piano part and some dislikes in other parts of the music.

I was one of those that listened and didn't know what DAW was played on what MBP.
I heard the engineers say/yell "nr 1 is the best" all the time. I agreed.
It had the most depht, best definition and fullest sound. It became really noticable.



Well. After 2,5 hour of listening we finally got the results.

MBP 1= Digital Performer
MBP 2= ProTools 9
MBP 3= Logic 9
MBP 4= ProTools 10
MBP 5= Cubase (latest version)

A lot of jawdropping amongst the engineers... "Whut?????!!!"
DP was by far the most liked in this DAW sound quality test!

The Swedish Sound Engineer Society claims this DAW test is not to be a scientific research in any way.
However the goal was to find out if there is any sonical difference between DAWs.
The answer to that was undoubtably: Yes! That we all agreed on.
Not an enourmous difference but still significant for the trained ears that attended this test.

So now you know

Can I ask a simple question? Did you do a nulltest? (Phase invert?). Sounds like a much easier job. Before knowing that the nulltest has failed, I would not even consider to make a listening test. And logic compressed? Are you serious?

[lllubi]

reason ?
dither ?