Lossless Formats for Your Sound Library

[iluvcapra]

Hello all,

As an experiment, I recently converted a swath of my library to Apple Lossless, and I wrote up my results and some thoughts...

The gist is that you can save at least 50% of your hard disk space, and still have all of your metadata and easy workflow with your Pro Tools.

Anybody else doing this or working with lossless files in the library?

[Fajita]

Lossless is relative. My "lossless" is BWAV. Because disk space is cheap now.

[iluvcapra]

Originally Posted by Fajita

Lossless is relative. My "lossless" is BWAV. Because disk space is cheap now.

Have you found it makes any difference?

[georgia]

Ditto... 48K 16bit BWAV...

1 terabyte sata - $250


cheers
geo

[Fajita]

Originally Posted by iluvcapra

Have you found it makes any difference?

the bigger the sound system (theatre, PA, etc) the more obvious the compression becomes.

BWAV is linear PCM, no compression, no loss.

BWAV includes the meta data too, and that's great for those of us that use Soundminer or any other catalog program.

[santacore]

Since disk space is so cheap I haven't thought about this. I guess as we do more hi-res. recording it might become more of an issue. Hopefully hard drives will continue to get cheaper to balance it out. For now, I think I will stick with BWAV. It seems like any compressed format has more potential for trouble down the line.

[danijel]

Originally Posted by Fajita

the bigger the sound system (theatre, PA, etc) the more obvious the compression becomes.

BWAV is linear PCM, no compression, no loss.

BWAV includes the meta data too, and that's great for those of us that use Soundminer or any other catalog program.

You didn't read the text on his link

Anyway, too bad FLAC is not better supported with Mac. How do you index/search the .m4a files?

[postprosound]

My library is in 24/48. Even if it sounds exactly the same, drives are cheap. The one big problem i have with converting to something like apple lossless, is it's not native to Pro Tools. Thus the files need to be converted every time i want to use it/transfer from Soundminer. Then the issue of portability and sharing sessions becomes an issue, as some of our editors work at home.

[Fajita]

Originally Posted by danijel

You didn't read the text on his link

Anyway, too bad FLAC is not better supported with Mac. How do you index/search the .m4a files?

Our libraries are already in SoundMiner... you can add FLAC support to Mac with a quicktime plugin. Not sure how reliable that is.

[iluvcapra]

Anyway, too bad FLAC is not better supported with Mac. How do you index/search the .m4a files?

I just use Spotlight, but I wrote a special application that searches just audio files, and a more-librarianish UI than the Finder, and that's really been quite sufficient.

the bigger the sound system (theatre, PA, etc) the more obvious the compression becomes.

All I can say is some of you don't seem to be hip to this and need to do some reading. A lossless coder gives you an output that is numerically identical to an output. I'm sortof surprised more people haven't heard of this, since I think MLP was first marketed in what, 1999?

For now, I think I will stick with BWAV. It seems like any compressed format has more potential for trouble down the line.

This is an issue, particularly for ALAC, so for my library I still have my WAVs for the time being, but I'm progressively burning archival versions of all my sounds to FLAC, along with copies of the FLAC source code. So, if in the distant future I need to reconvert them again, even if FLAC has disappeared I should still be able to recompile my copy of FLAC from the sources.

Considering how I've seen people treat their library drives and retrospect backups, I think I'm being sufficiently conservative by comparison.

[jjb]

Disk space is CHEAP. It's under $100 for a 1TB SATA drive these days, and under $50 for a SATA dock to make it easy to swap them out as needed. Why deal with the added processing time?

[iluvcapra]

Originally Posted by jjb

Disk space is CHEAP. It's under $100 for a 1TB SATA drive these days, and under $50 for a SATA dock to make it easy to swap them out as needed.

I'm kinda paranoid about hard disk drives in general; one of the reasons I do this is so I can keep as many copies of my library on different drives (an DVD-Rs) as possible.

Originally Posted by jjb

Why deal with the added processing time?

This is sortof a non-issue; it certainly doesn't have any effect on Pro Tools while playing, since Pro Tools is just going to mung the file down to WAV for its purposes, and the munging itself doesn't seem to take any longer than with a stereo WAV file. I don't see how the "added processing time" actually changes the use case.

I like the copying, since it means that none of my project folders/sessions have any link dependcies that go into my library drive- every project has its own copy of the media it needs.

[gmarinov]

i think few file formats have had the luxury to be universally recognised.. in 5 years' time i wouldn't want to deal with a piece of software which won't open FLAC/ALAC (note: not "doesn't know how to").

also imagine computer-less scenarios for dealing with audio files straight out of your library. not saying there are many cases, but who knows what you'll run into.. no wonder WAV/BWF is the recommended standard for archiving..

at some point i thought having music in ATRAC files wouldn't be a problem..

yes, these are all non issues *today*.

[iluvcapra]

Originally Posted by gmarinov

i think few file formats have had the luxury to be universally recognised.. in 5 years' time i wouldn't want to deal with a piece of software which won't open FLAC/ALAC (note: not "doesn't know how to").

also imagine computer-less scenarios for dealing with audio files straight out of your library. not saying there are many cases, but who knows what you'll run into.. no wonder WAV/BWF is the recommended standard for archiving..

This does appear to be the biggest hang-up, though I'm honestly more concerend about my media failing before FLAC or ALAC being orphaned.

The biggest win seems to be in portability, you can walk around with a horrifyingly large library on a USB thumb drive using this-- I wish I'd had my library like this years ago. Archival seems to be a bit dicier.

[crussom]

What DAWs support opening lossless audio? What happens when you process/pitch the sounds, do you get weird artifacting? Will these lossless formats still be supported in a couple years? I'd rather just stick with Bwav.

Like others have said, disk space is really cheap. I have my library on an internal 2TB drive. I have it cloned to a external 2TB that I keep offsite. Soon, I'll probably add a second clone to keep onsite, just to save me the hassle of a 20 min drive to get my backup if my original dies. I think I have less then $600 invested in both of my drives. I check my backup regularly (like every couple months) to ensure it is still alive. I really don't see myself losing data. It is highly unlikely that both of my drives will fail at the same time.

As disks get lager and cheaper, I record more and more at higher sample rates. I've been recording all of my FX at 96K for the last 5 years. Recently I've moved to 192K for some things. Recording at higher rates give a lot of flexibility when it comes to slowing/pitching sounds.

[iluvcapra]

Originally Posted by crussom

What DAWs support opening lossless audio? What happens when you process/pitch the sounds, do you get weird artifacting? Will these lossless formats still be supported in a couple years? I'd rather just stick with Bwav.

Just accept as given that I'm always converting audio to PCM when I spot it into a session, the DAW never plays the lossless file. Since my whole library is interleaved, and most of it is at 16 bit, this is basically where I am now, since I'm on Pro Tools and Pro Tools has to convert just about anything in my library before it'll play.

Does anyone else work like this? I just think the whole .L/.R business is an atrocity tutt and I won't let it anywhere near my library files.

Originally Posted by crussom

I've been recording all of my FX at 96K for the last 5 years. Recently I've moved to 192K for some things. Recording at higher rates give a lot of flexibility when it comes to slowing/pitching sounds.

Right, which is why FLAC and ALAC both support up to 32-bit integer PCM at least up to 192, and FLAC supports LPCM sample rates into the megahertz.

I've appreciated the comments on this thread, I'm definitely keeping PCM backups of my personal sounds; but I think some of you talk as if LPCM sample streams had fairy dust in them or something.

[charles maynes]

Hey all- ILuvCapri is one of the smartest, and most clever guys I know. He has been looking at this for a long time, and I have to say I am in his debt on many fronts. The idea of being able to reduce the size of a sound library is not trivial when speaking of backing up multiple terabytes of effects. And the pat answer of drive space being cheap doesnt speak at all to offline backup. The issue I am afraid of with a compressed format is the possibility of the format becoming outdated, but that is a concern with any fileformat, as was the case with the original sd2 format going by the wayside.

At any rate, I wanted to step in and say to those who dont know ILuvCapri that he is indeed the real deal.

[charles maynes]

Originally Posted by iluvcapra

Just accept as given that I'm always converting audio to PCM when I spot it into a session, the DAW never plays the lossless file. Since my whole library is interleaved, and most of it is at 16 bit, this is basically where I am now, since I'm on Pro Tools and Pro Tools has to convert just about anything in my library before it'll play.

Does anyone else work like this? I just think the whole .L/.R business is an atrocity tutt and I won't let it anywhere near my library files.



Right, which is why FLAC and ALAC both support up to 32-bit integer PCM at least up to 192, and FLAC supports LPCM sample rates into the megahertz.

I've appreciated the comments on this thread, I'm definitely keeping PCM backups of my personal sounds; but I think some of you talk as if LPCM sample streams had fairy dust in them or something.

I am doing the same as mentioned here except that much of my library is now at 96k/24bit- I would never ever use a sound file that was not in a mastered and archived (twice) form.

[dsteinwedel]

Originally Posted by iluvcapra

I've appreciated the comments on this thread, I'm definitely keeping PCM backups of my personal sounds; but I think some of you talk as if LPCM sample streams had fairy dust in them or something.

No--it's that the concept of cutting file size in half while not losing data or fidelity is ... foreign. It's new, it's different, it's unknown, and hence it's scary. As they say, "there's no free lunch."

I run a 2 (3-ish) backup system. My daily working FX drive is a mirrored raid system. My backup drive (single) gets updated once weekly and lives off-site. I've been contemplating both a 3rd backup disk (at yet another location) and a cloud backup as prices are starting to get realistic for terabyte storage.

I'm kinda paranoid about hard disk drives in general; one of the reasons I do this is so I can keep as many copies of my library on different drives (an DVD-Rs) as possible.

Doing FLAC for DVDs makes sense (223 DVDs in a TB), though I've had more problems with burned DVDs/CDs going bad than I have hard drives failing.

-D

[iluvcapra]

Originally Posted by dsteinwedel

No--it's that the concept of cutting file size in half while not losing data or fidelity is ... foreign. It's new, it's different, it's unknown, and hence it's scary. As they say, "there's no free lunch."

This is what I get for having Claude Shannon as a personal hero.

Actually, I've discovered OPTAR and I think I'll be using that for all of my backups from now on!

[Shan]

Originally Posted by iluvcapra

The gist is that you can save at least 50% of your hard disk space, and still have all of your metadata and easy workflow with your Pro Tools.

Anybody else doing this or working with lossless files in the library?

Lossless is great for session transfers etc via the net and archiving. Monkey's Audio and FLAC are my go to formats. A DAW like Reaper can use these formats natively in the DAW itself.

Shane

[Fajita]

Originally Posted by iluvcapra

This is what I get for having Claude Shannon as a personal hero.

Actually, I've discovered OPTAR and I think I'll be using that for all of my backups from now on!

the treehuggers will love it!

[iluvcapra]

Originally Posted by Shan

Lossless is great for session transfers etc via the net and archiving. Monkey's Audio and FLAC are my go to formats.

What's Monkey's Audio about? Do you have a lot of problems with having to walk people through the decompressing on the other end? Have you had any experience using it with Pro Tools sessions? I'm often looking for a way to ship compressed audio to clients but I hate having to go through a 10-step explanation of how to download Max or something...

DigiDelivery apparently does lossless compression when it sends deliveries, but the methods it employs are a mystery.

I use FLAC with --keep-foreign-metadata to do my session archives too. Pro Tools totally ginks if you don't compress the WAV files with all the embedded chunks.

[Shan]

Originally Posted by iluvcapra

What's Monkey's Audio about? Do you have a lot of problems with having to walk people through the decompressing on the other end?

Nah. The front end is straight forward enough for most people and it does batch conversion both ways.

Have you had any experience using it with Pro Tools sessions?

Yes. I've never had any issues with it myself.

DigiDelivery apparently does lossless compression when it sends deliveries, but the methods it employs are a mystery.

Digidelivery had it's day. I was never a fan of it myself.

Shane

[cjguitar]

It just doesn't make since for a lot of us to convert our libraries to Apple Lossless, as most recorders out there record to .wav or aif natively. It adds another step to the mastering process, and I'm not even sure if there's an easy way to batch convert hundreds of thousands of files. I, for one, will stick to LPCM for all library work.

[charles maynes]

Chris are you making copies in PT from a master library, or using the masters in your edits? I know for me, I master all multi-channel recordings into interleaved files to reduce the load on directories, and doing a lossless conversion to reduce the file size to 50 to 60% of its uncompressed size is not trivial to me. And anding one more step until it gets a library index, at least to me is a triviality- (much in the way doing two sets of offline backups is). I have my library drive set for read only as well and would never, ever use a master in a session.

I know the last time a did a DVD backup of the library, it took about 400 dvd-r's and months to complete. I do that level backup usually every 12 months, and maintain a second copy on hard drives as backups as well. And if the time to compress to one half the size means I gain back dozens, if not hundreds of hours creating offline backups- the math says it is a good use of time....

My session copies for projects I would backup offline as uncompressed files though- in duplicate, of course.

[dsteinwedel]

No--it's that the concept of cutting file size in half while not losing data or fidelity is ... foreign. It's new, it's different, it's unknown, and hence it's scary. As they say, "there's no free lunch."

I spoke with my programmer about this yesterday. After getting a lesson in how both .zip and FLAC work, I'm a lot more comfortable (assured, in fact) saying that it does do what it says it does: Fully loss-less file compression. The audio data is not compressed or altered--the bits are 'reorganized' (for lack of a better term) in a more efficient manner.

Here's a quick lesson on what's going on: How Does a .ZIP File Work? - Yahoo! Answers

So Capi/Charles, are there programs you recommend that will do bulk encoding/decoding for archival purposes?

Cheers,
Dave

[charles maynes]

Soundgrinder, and Soundgrinder pro...

[FullFrequency]

Not exactly a library thing, but.....

One of my biggest clients, who happens to be one of the Uk's largest independant production companies requests FLAC as the master deliverable via FTP. Yes they don't even come in for sign off!!! - they just send me an email with notes., LOVE FLAC

FF

[iluvcapra]

Originally Posted by dsteinwedel

I spoke with my programmer about this yesterday. After getting a lesson in how both .zip and FLAC work, I'm a lot more comfortable (assured, in fact) saying that it does do what it says it does: Fully loss-less file compression. [...]

So Capi/Charles, are there programs you recommend that will do bulk encoding/decoding for archival purposes?

I used Sound Grinder Pro for my library, because it has the best metadata copying of anything I've seen. For my archival, I use the flac command line program with a script that automatically verifies (both inside flac and by doing a decompression and binary comparison) to verify I'm getting exactly what I put in. Actually here's the script...

gist: 244986 - A Shell Script for FLACing a broadcast WAV file and binary-comparing (including all wrapper and metadata) with original- GitHub

I use this for session stuff, where I don't care about library metadata as much as accuracy.

I actually used to use plain-old ZIP for predubs, and you could gain HUGE amounts of hard disk space back because predubs were mostly silence, and ZIP would runlength encode all the zeroes. ZIP is a very safe format because almost every computer on Earth ships with the ability to decompress ZIPs and the file format hasn't changed substantially since 1993.

I'm not as concerend about losing a lossless file to corruption as much as I'm concerend about losing the program I used to compress it. Based on comments here and thinking about it, compression is a great way to make my library more portable, but it can only be a small part of a larger strategy of making a library indestructible.