Lossless Formats for Your Sound Library

[Shan]

Originally Posted by dsteinwedel

After getting a lesson in how both .zip and FLAC work, I'm a lot more comfortable (assured, in fact) saying that it does do what it says it does: Fully loss-less file compression. The audio data is not compressed or altered--the bits are 'reorganized' (for lack of a better term) in a more efficient manner.

Ya, you'll find FLAC and other lossless formats will null out with the source .wav or aiff files. A few tests can give certainty the new lossless user.

Shane

[Dode]

Just to mention, for the OS X users, the existence of a small free utility based on MacFuse that allows transparent usage of FLAC files with any DAW :
TwistedFLAC
It even works with Soundminer !
BTW, also have a look to its TwistedWave editor, one of the best IMHO, and the dev is extremely reactive with new features and bug fixes.

I'd love that all major audio DAWs and tools natively support FLAC, but i asked for this so many times (Pro Tools, Nuendo, Soundminer) in the past, and it never came out... I clearly lose hopes, and will not hold my breath anymore...

I know disk space is less and less of an issue, but for archival or exchange purposes, or even for dealing with large 24bit/96kHz surround libraries, it could be VERY usefull. Same goes for WavPack, which is nice too IMHO.

Time will tell...

Bye.

[danijel]

Originally Posted by Dode

Just to mention, for the OS X users, the existence of a small free utility based on MacFuse that allows transparent usage of FLAC files with any DAW :
TwistedFLAC

That is wicked. Sort of a compressed file system, but specialized for audio. I would love to see read/write stats - with a fast processor, it might turn out that it is faster to read a 50% smaller file and decompress it, then just read a full raw WAV. I'll give it a try as soon as possible. Thanks!

P.S. hope the performance issues with NTFS-3G don't come from MacFuse.....

[apple-q]

Originally Posted by danijel

That is wicked. Sort of a compressed file system, but specialized for audio. I would love to see read/write stats - with a fast processor, it might turn out that it is faster to read a 50% smaller file and decompress it, then just read a full raw WAV.

How could it be faster to decode a flac to BWAV than directly play back a BWAV?
Does anyone know if FLAC strips meta-data from BWAVs? iXML, soundminer, bext-chuck all stay intact?
Side note: I´ve noticed that flac doesn´t always give you 50% reduction. If you have highly compressed audio all maxed out flac reduction is way lower. Similar to zip. If you zip-compress a bwav that contains a lot of silence (FX stem for example) a multi-megabytefile can sshrik down to a few kilobytes because the BWAV contains lots of redundant bits. With high density audio flac is also less effective. The overall result doesn´t necessarily save you 50% but less.

[danijel]

Originally Posted by apple-q

How could it be faster to decode a flac to BWAV than directly play back a BWAV?

Reading from disk takes time, so if the file is compressed a lot (like most SFX libraries could be, with all the blanks in-between the actual sounds), it might turn out faster to read less data and then decompress it. Of course, this is not likely to be usable for project Audio folder (because one wouldn't want the real-time decompressing to use up the CPU during playback), but might be usable for importing SFX from the lib.
RLE compressed TGA or TarGA (in video world) has the advantage over the uncompressed TGA in both file-size and read-write speed, although the RLE decompression is probably much faster than FLAC:
Run-length encoding - Wikipedia, the free encyclopedia

[apple-q]

Originally Posted by danijel

Reading from disk takes time, so if the file is compressed a lot (like most SFX libraries could be, with all the blanks in-between the actual sounds), it might turn out faster to read less data and then decompress it. Of course, this is not likely to be usable for project Audio folder (because one wouldn't want the real-time decompressing to use up the CPU during playback), but might be usable for importing SFX from the lib.
RLE compressed TGA or TarGA (in video world) has the advantage over the uncompressed TGA in both file-size and read-write speed, although the RLE decompression is probably much faster than FLAC:
Run-length encoding - Wikipedia, the free encyclopedia

What do you mean by "read" and "decompress". AFAIK, the process of de-compression will write a BWAV to disk. So how could that be faster than reading a full res BWAV that is already there on my disk?

[Dode]

Originally Posted by apple-q

Does anyone know if FLAC strips meta-data from BWAVs? iXML, soundminer, bext-chuck all stay intact?

It depends, as stated someone earlier, of the use of the "--foreign-metadata" option in the flac command line. TwistedFLAC/TwistedWave use it, so you're safe on this side with these tools. This is a huge advantage for FLAC IMHO, postproduction wise (i personally use metadatas a lot, BWF, iXML, and Soundminer v3/v4).

Side note: I´ve noticed that flac doesn´t always give you 50% reduction. If you have highly compressed audio all maxed out flac reduction is way lower. Similar to zip. If you zip-compress a bwav that contains a lot of silence (FX stem for example) a multi-megabytefile can sshrik down to a few kilobytes because the BWAV contains lots of redundant bits. With high density audio flac is also less effective. The overall result doesn´t necessarily save you 50% but less.

Yes, it depends of the audio content. Try flacing a gigabyte null file, it's fun to see the size of the resulting file ;-) .
IME the average compression ratio is about 40% (between 30 and 70% depending on the material), which is quite good in most cases.

I never thought of what Danijel stated above, quite interesting though... CPU vs hard drive speed ?

Do you think we can "force" the major DAW/Editor manufacturers to support FLAC (opensource, no cost for them !), with a petition or something ? Maybe Gearslutz could push it in any way ? Or Charles Maynes himself ? :-)
If yes, count me in, again...

Same goes for WavPack, which has other advantages (but is not opensource IIRC).

Bye.

PS : concerning TwistedFLAC, remember it's the first release, and that many things could be added/improved, like a multi-folders option, etc. Just ask in its forum, Thomas is very responsive.

[danijel]

Originally Posted by apple-q

What do you mean by "read" and "decompress". AFAIK, the process of de-compression will write a BWAV to disk. So how could that be faster than reading a full res BWAV that is already there on my disk?

If we're talking about copying stuff from a (FLAC encoded) SFX HDD to a (non-encoded) project HDD via PT import (the real-world example), it can't be faster, so the reading speed argument doesn't matter anyway. But the whole process can perhaps take the same amount of time, which is still good, considering you save disk space without drawbacks.

Let's take an extreme example: you have a completely silent (and, thus, very easy and fast to compress/decompress) 24bit stereo file of one hour in length in your SFX library that you want to import and copy to your project's Audio folder, so that the result is a normal BWAV file that PT can read and play back.

- in case your SFX lib is regular BWAV: reading the 1gb file (1 hour, 24 bits, stereo) from SFX drive takes 20s (50mb/s). In parallel, the file is written to the project's Audio HDD, which also takes 20s, so the whole process takes 20s.

- FLAC case: reading the file from SFX drive takes 10ms (because the filesize is 1kb), decompressing is done much faster than your Audio HDD can write, so it is done in parallel with the writing of the file to your project's Audio folder. The writing itself takes 20 sec (50mb/s), because the decompressed file size is still 1 gb. So, all in all, the process takes (again) 20 seconds, although the reading took only 10ms and the decoding took a minimal amount of time.

On the other side of the extreme, it just might turn out that it takes triple the time to read and decompress a FLAC file with "unsuitable" content, compared to simply reading the BWAV.

Now, this is why I said I would like to see the read/write stats on different kinds of files, to see if the FLAC reading+decompressing speed would be comparable to BWAV reading speed on an average SFX library.

[iluvcapra]

Originally Posted by danijel

On the other side of the extreme, it just might turn out that it takes triple the time to read and decompress a FLAC file with "unsuitable" content, compared to simply reading the BWAV.

Considering audio reading is primarily I/O bound and decoding linear interpolation of a stream can easily parallelized, you'd probably see an improvement on the average, though FLAC files have a big problem with efficiently seeking into the file to a particular time index, unless the file is heavily hinted. This sorta has to be instantaneous for use in a DAW.

Originally Posted by danijel

M/S is NOT mono compatible

I don't think you're doing it right. Trying summing the output of the decoder matrix, instead of muting the S channel. "M" doesn't stand for "mono"

[charles maynes]

Originally Posted by danijel

If we're talking about copying stuff from a (FLAC encoded) SFX HDD to a (non-encoded) project HDD via PT import (the real-world example), it can't be faster, so the reading speed argument doesn't matter anyway. But the whole process can perhaps take the same amount of time, which is still good, considering you save disk space without drawbacks.

Let's take an extreme example: you have a completely silent (and, thus, very easy and fast to compress/decompress) 24bit stereo file of one hour in length in your SFX library that you want to import and copy to your project's Audio folder, so that the result is a normal BWAV file that PT can read and play back.

- in case your SFX lib is regular BWAV: reading the 1gb file (1 hour, 24 bits, stereo) from SFX drive takes 20s (50mb/s). In parallel, the file is written to the project's Audio HDD, which also takes 20s, so the whole process takes 20s.

- FLAC case: reading the file from SFX drive takes 10ms (because the filesize is 1kb), decompressing is done much faster than your Audio HDD can write, so it is done in parallel with the writing of the file to your project's Audio folder. The writing itself takes 20 sec (50mb/s), because the decompressed file size is still 1 gb. So, all in all, the process takes (again) 20 seconds, although the reading took only 10ms and the decoding took a minimal amount of time.

On the other side of the extreme, it just might turn out that it takes triple the time to read and decompress a FLAC file with "unsuitable" content, compared to simply reading the BWAV.

Now, this is why I said I would like to see the read/write stats on different kinds of files, to see if the FLAC reading+decompressing speed would be comparable to BWAV reading speed on an average SFX library.

the issue I see with using compressed files as a native format is that it requires a lot of additional cpu cycles to do things- since disk space is relatively cheap, that doesnt make a much sense to me...

[danijel]

Originally Posted by iluvcapra

I don't think you're doing it right. Trying summing the output of the decoder matrix, instead of muting the S channel. "M" doesn't stand for "mono"

Depends on what you consider "mono compatible". We had a leeeengthy discussion here:
M/S in Protools session - 2 track or 3 track ?
I also posted some example recordings - they are still available from that thread.

[apple-q]

Originally Posted by danijel

If we're talking about copying stuff from a (FLAC encoded) SFX HDD to a (non-encoded) project HDD via PT import (the real-world example), it can't be faster, so the reading speed argument doesn't matter anyway. But the whole process can perhaps take the same amount of time, which is still good, considering you save disk space without drawbacks.

Let's take an extreme example: you have a completely silent (and, thus, very easy and fast to compress/decompress) 24bit stereo file of one hour in length in your SFX library that you want to import and copy to your project's Audio folder, so that the result is a normal BWAV file that PT can read and play back.

- in case your SFX lib is regular BWAV: reading the 1gb file (1 hour, 24 bits, stereo) from SFX drive takes 20s (50mb/s). In parallel, the file is written to the project's Audio HDD, which also takes 20s, so the whole process takes 20s.

- FLAC case: reading the file from SFX drive takes 10ms (because the filesize is 1kb), decompressing is done much faster than your Audio HDD can write, so it is done in parallel with the writing of the file to your project's Audio folder. The writing itself takes 20 sec (50mb/s), because the decompressed file size is still 1 gb. So, all in all, the process takes (again) 20 seconds, although the reading took only 10ms and the decoding took a minimal amount of time.

On the other side of the extreme, it just might turn out that it takes triple the time to read and decompress a FLAC file with "unsuitable" content, compared to simply reading the BWAV.

Now, this is why I said I would like to see the read/write stats on different kinds of files, to see if the FLAC reading+decompressing speed would be comparable to BWAV reading speed on an average SFX library.

I think it´s already a one way road. Disk space is already cheap and it´s very unlikely that it´s going to get more expensive or that drives are going to get smaller in capacity. So for me the bottom line is: why bother? Why shrink my files by 40% when one month later a drive twice the size of my current one is about half the prize?
Even if the transfer speed is 1 nanosecond what would that help? Personally when I browse my library I spend 99% of the time to find the sounds I need. Transfer speed is a complete non-issue. The most important thing for me is that I can hear the sound right away when I hit play in soundminer. I don´t want to jump through hoops and activate some virtual volume that transcodes the files in the B/G. Do you guys ever listen to the sounds or is it all about transfer speed for you?

[charles maynes]

the issue is offine archiving. There is no sense in using compressed files in a session. It makes all the sense in the world to use for your library materials so you can manage off-harddisk archives.

With 96k material, if it is nature recordings or sound effects, I would doubt one would ever hear the difference if there was one. In the tests I have done, the sound is either identical or very nearly identical.

[iluvcapra]

Originally Posted by apple-q

Do you guys ever listen to the sounds or is it all about transfer speed for you?

I would probably stay away from the FUSE solution, since Pro Tools won't really dig it and who knows how long that'll be supported on Mac OS X. For an archive or library server, however...

[Dode]

Originally Posted by iluvcapra

I would probably stay away from the FUSE solution, since Pro Tools won't really dig it and who knows how long that'll be supported on Mac OS X. For an archive or library server, however...

In the Fuse solution i mentionned earlier (TwistedFLAC), it's absolutely not an issue, as original FLAC files will always be there. It's just a realtime conversion.
But it could be a problem if one day FLAC vanished, as did many formats in the past...

I agree that disk space is generally not an issue anymore, but for archiving or distributing purposes, 40% is not insignifiant. I'm dealing with 24bit/96kHz surround material ATM (mostly long ambiences), and gaining half the time to copy them from drive to drive, or transfering them through a LAN, will not be insignifiant either, for me at least.

Well, i guess it's a matter of personal usage and needs.

Bye.

[apple-q]

Originally Posted by charles maynes

the issue is offine archiving. There is no sense in using compressed files in a session. It makes all the sense in the world to use for your library materials so you can manage off-harddisk archives.

What are "off-harddisk archives"? Do you backup your library to AIT-tapes and before that you FLAC everything? Or where would such a Library else live but on a drive or an LTO-Tape? (which has built-in compression anyway BTW)

Lets say you have 1 TB library. After Flacing it it will be around 600GB. So then you go out to find a 600GB drive to backup?! Or if there aren´t any a 500GB and a 200GB?

P.S.: I could be wrong here but I think the main application for flac and apple lossless is to distribute full-res audio over the internet (I wouldn´t be surprised if 90% of flac usage is to spread pirated CDs at full resolution, to be honest). But with todays data-connections becoming faster and faster even those days are counted if you ask me.

To me it looks like flac simply adds more steps in between and hoops to jump through at a minimal benefit.

Maybe I´m ignorant, but I don´t understand the benefit for the everyday professional work regarding the original question of this thread.

[iluvcapra]

Originally Posted by apple-q

What are "off-harddisk archives"? Do you backup your library to AIT-tapes and before that you FLAC everything? Or where would such a Library else live but on a drive or an LTO-Tape? (which has built-in compression anyway BTW)

Aside from a copy of my library on HD (since it's half as big, I can squeze twice as much project archives on that drive), I keep a copy of my entire library FLAC'd on LTO and a copy of my personal sounds (the ones I recorded and not on any CDs anywhere) FLAC'd and stored on Amazon S3 -- this is my "offsite" one.

[Haydn]

Originally Posted by crussom

As disks get lager and cheaper, I record more and more at higher sample rates. I've been recording all of my FX at 96K for the last 5 years. Recently I've moved to 192K for some things. Recording at higher rates give a lot of flexibility when it comes to slowing/pitching sounds.

Are you using microphones with frequency responses that extend way above 20K?

I never thought it would be worth it for me to record at those sampling frequencies as my microphones don't pick up much above 20k. I'll do some tests this weekend to see if down pitching sounds better when using high sampling freqs with my equipment.

Thanks,
Haydn.

[charles maynes]

Originally Posted by apple-q

What are "off-harddisk archives"? Do you backup your library to AIT-tapes and before that you FLAC everything? Or where would such a Library else live but on a drive or an LTO-Tape? (which has built-in compression anyway BTW)

Lets say you have 1 TB library. After Flacing it it will be around 600GB. So then you go out to find a 600GB drive to backup?! Or if there aren´t any a 500GB and a 200GB?

P.S.: I could be wrong here but I think the main application for flac and apple lossless is to distribute full-res audio over the internet (I wouldn´t be surprised if 90% of flac usage is to spread pirated CDs at full resolution, to be honest). But with todays data-connections becoming faster and faster even those days are counted if you ask me.

To me it looks like flac simply adds more steps in between and hoops to jump through at a minimal benefit.

Maybe I´m ignorant, but I don´t understand the benefit for the everyday professional work regarding the original question of this thread.

I backup to multiple copy DVD-R it has been as robust as any of the Tape formats I have worked with in recent years- it also allows me to avoid retrospect's propritary file format which has caused issue in the past. I also archive to two set of hard drives.

[cjguitar]

Now that Apple has open sourced ALAC, and it's been over a year since this discussion, I wanted to re-visit this topic and see what has changed. Has anybody else adopted this workflow and what have you found? I'm seriously considering it now, but I want to find an easy way to convert my library and retain all of my Soundminer Metadata. Any ideas?

[iluvcapra]

I wouldn't recommend ALACs for archival, I'd also not necessarily adopt an open source solution unless you were personally comfortable with compiling the source yourself, because 5-10 years from now a Tarball of the source might be all you have.

I use FLACs for archival, FLAC also has some nice features for archiving the WAV out-of-band data, like Broadcast-WAV metadata and unidentified chunks that you would need if you wanted to restore a session bit-for-bit (before FLAC had this function I used WAVPACK). Also I do routinely archive a Tarball of the FLAC source. The drawback of FLAC is that Pro Tools, and Nuendo and Logic, don't read it without a lot of setting up on or host, and the metadata isn't importable by Spotlight on a stock Mac OS X host.

[JSt0rm]

Originally Posted by iluvcapra

I wouldn't recommend ALACs for archival, I'd also not necessarily adopt an open source solution unless you were personally comfortable with compiling the source yourself, because 5-10 years from now a Tarball of the source might be all you have.

I use FLACs for archival, FLAC also has some nice features for archiving the WAV out-of-band data, like Broadcast-WAV metadata and unidentified chunks that you would need if you wanted to restore a session bit-for-bit (before FLAC had this function I used WAVPACK). Also I do routinely archive a Tarball of the FLAC source. The drawback of FLAC is that Pro Tools, and Nuendo and Logic, don't read it without a lot of setting up on or host, and the metadata isn't importable by Spotlight on a stock Mac OS X host.

What exactly is the worry? Bit rot?