How To Create A Lossless Music Archive (FLAC)

Some people may be looking for information on how to archive their CDs or wondering why one would even be interested in doing so. This post will try to address those questions as thoroughly as possible. There is a lot of information here and it could probably be broken down into several posts, but that can be harder to keep track of and I think it's better to have everything in one place.

First off, why would one create a lossless audio archive? One reason is convenience. Having a CD quality file on a hard drive allows users to do many things quickly with their music without having to continuously re-rip it. Transcoding, which I will cover below, is much easier and convenient once you already have your files properly archives. One of the primary reasons that I do it, is with the pace of change in terms of digital music formats and technology dealing with the consumption of audio media, archiving your music in FLAC enables me to easily keep pace with these changes with little effort.

Of course for every benefit there are some drawbacks. If you have a lot of CDs or if you do a lot of legal music trading (i.e. Grateful Dead, Phish, etc.) your music library might be very large. This means you will need a sizable hard drive (or perhaps several). This process can be very time consuming as well. Fortunately, for myself, I find this entire process cathartic, and, as you will probably see, I am OCD enough that I enjoy the attention to detail it sometimes requires.

It is probably worthwhile to say something about file formats at this juncture to ensure that there is no confusion for those who are new to the world of digital media. To keep things simple I'll deal with 3 different file formats: WAV, FLAC, and MP3.

WAV is the format that you end up with when you rip a CD to your hard drive. One minute of music in WAV format takes up about 10MB. The complement of WAV on the Mac system is called AIFF. However, Mac has no problem handling WAV and since it is the standard and works across platforms I use WAV as my default raw audio format.

FLAC stands for Free Lossless Audio Codec. When encoding a WAV file to FLAC the same minute of music will take up anywhere from 4.5MB-6MB. Depending on the original WAV files, the FLACs of the same files will use 40-60% of the original drive space. So a 30 minute CD would be around 300MB in WAV format and circa 150MB in FLAC. Although FLAC is the standard in lossless audio compression, Apple's iTunes does not support it. Apple has their own proprietary lossless audio compression format, but I prefer FLAC. I have conducted checksum tests with Apple's lossless codec and they routinely fail whereas I have had no problems with FLAC. (On rare occasion one might see reference to SHN files as well. SHN or Shorten is a lossless file format that pre-dates FLAC, but is not as flexible as FLAC and has been replaced by FLAC in recent years.)

MP3 is the standard lossy audio compression format. The size of MP3 files depends on the bit-rate with which they are encoded. Because I have lossless copies of everything, I encode all of my MP3s at 128kbps, which is the standard rate. The same 1 minute of music encoded in MP3 format at 128kbps takes up about 1MB, or 10% of the original file disc space. This reduced file size does not come without consequences. The audio quality of the MP3 is inferior to both WAV and FLAC. With modern production techniques most people cannot tell the difference, but trust me, it's there and it doesn't sound as good. Therein lies my problem with Apple's iTunes - customers are paying good money for an inferior product. I will always opt to purchase a CD and create my own MP3s. That way I can create a perfect, CD-quality, lossless version of the same album. (While MP3 is the standard in lossy audio compression other formats exist, such as Apple's AAC format. Again, I deal with the standard format because they work error free on any platform.)

Ok, so now that I've talked about the advantages and disadvantages of creating a FLAC audio archive and outlined the three basic file types, it is fit to proceed to discuss the software involved in creating such an archive. My primary machine is an Apple MacBook running SnowLeopard, but I will include some links and comments for a few key PC programs as well.

Audio Rippers:

XLD: The best audio ripper is Exact Audio Copy, and unfortunately there is no version for the Mac. XLD is the closest thing to EAC on the Mac. It's got a simple interface and is pretty straight-forward. XLD can rip to a variety of file formats, not just WAV, and can automatically grab album artwork, too. Most importantly, it taps into the AccuRip database to ensure that rips are 100% accurate. XLD is also actively being developed. Make you have your preferences check for new versions because updates come out all the time for this software (See Right).

Rip/Max: These are two of several tools offered by programmer Stephen Booth. I find Max's ripping interface to be less user-friendly that XLD's, and I like XLD so much that I have not used Rip. However, it's good to know there are options and Rip continues to be developed.

Encoder/Decoder/Transcoder:

xACT: Is able to convert files between multiple formats, although depending on what you're trying to do, transcoding may take several steps. One of the best things about xACT is its ability to generate and verify file checksums (which will be discussed below) including, MD5, ST5, and FLAC Fingerprints.

Max: Supports an even wider spectrum of audio formats and while it lacks the checksum capabilities of xACT as a transcoder, this is my go-to program.

Metadata:

xACT: Also has metadata capabilities, but I find its FLAC Tagging to be rather cumbersome. One of the nice things about xACT's tagging is that it allows users to batch tag the song title and track number of FLAC files. However there is a glitch in the software that I have consistently ran into where, if you are trying to tag more than one album at a time you must tag one. Quit the program, re-open it, and tag the next one, and so on. There seems to be a problem with clearing the clipboard with the auto-title function.

Tag: If you've planned ahead with your original ripping settings the shortcomings of xACT can easily be circumvented with Tag. When setting up XLD to rip users can set the default format for the filenames of the ripped files (See Picture). I use the string "%n - %a - %t" which uses the CD data (which can be edited before ripping) to provide the track number, artist, and track title for each song ripped. This will look like "01 - Grateful Dead - Promised Land." Tag has a feature where the program will "guess" certain criterion based on a similar string. More on this below.

Playback:
As I already mentioned, iTunes does not include FLAC support. Feel free to contact Apple and request FLAC support in the future (http://www.apple.com/feedback/itunesapp.html). While I may not be a huge fan of iTunes as a whole, it is the best music database program and it would be great to be able to apply that organizational structure to a FLAC library.

Cog: A standard drag and drop music player. There's not a whole lot of flash to this program, but it doesn't use a lot of system resources and plays a wide spectrum of audio formats. There are no playlists or equalizers, which would be nice, but for those who just want a solid, reliable music player, Cog is tough to beat.

Play: I fiddled around with Play a while back and found myself appreciating the simplicity of Cog more. However, this is another Stephen Booth program and a definitely a worthwhile audio player.

Miscellaneous:

Album Artwork: Both Amazon.com and Wikipedia are good sources for tracking down album artwork, and I can go into a trick for getting high quality iTunes artwork on your hard drive, but the Album Art Exchange is a wonderful place to find quality album artwork.

Fluke: Fluke claims to enable users to play FLAC files in iTunes. I've never got it to work, and, quite frankly, I'd prefer native support for FLAC so I didn't invest much time in making it work. However, installing Fluke has a positive Finder benefit by allowing users to preview FLAC files in the Finder. Simply select a FLAC file in the Finder and press the space-bar. You'll see a preview screen and the file will start playing (the same work for WAV and MP3, and well all files). Press space-bar again to close the preview.

FLAC & Spotlight: Once you have your FLAC library built installing this little utility will allow Spotlight to search the metadata in your library. A really useful feature.

Hardware: I would recommend an external hard drive. The size depends on how big your music collection is. At present I have my entire FLAC library on a 1TB external hard drive and it takes up almost 700GB. I have the entire library backed up on a second 1TB drive as well. I also have a 320GB external hard drive where I store my iTunes library.

For PCs, the best audio ripper is Exact Audio Copy. A panacea encoding program can be found in Trader's Little Helper. I've found FLAC Frontend and MD5 Summer useful over the years as well.

Archival Workflow

Alright. Now that the software issue has been covered I'll describe my actual archival work flow. I've done this hundreds of times and although it can be a time consuming process, especially if you have scratched CDs, the final product is worth it, in my opinion. I will go through the entire process and explain what all the checksums, transcoding, tagging, etc. is all in the successive section.

Step 1: Insert CD in disc drive. Also create a destination folder. I have a folder for the artist, and a separate folder for each album. If it is a multi-disc set, each disc gets its own folder within the album folder. I prefer to label mine chronologically. In the example pictured, you can see that I label the folder with the Year, Album Title, and the file type. WIthin the parent album folder I have folders for each disc where I keep that disc's relevant data.

Step 2: Open XLD. Edit disc metadata to your liking. Make sure that the drop down menu at the bottom of the window is on "Include pre-gap for all tracks" to ensure that your rips are verified with the AccuRip database. If XLD finds artwork for you, you can save it right from the program in the target folder. When you have everything to your liking, press decode. In the picture you can see that I have XLD set to rip to both WAV and encode those WAV files to FLAC from the get-go. This is something the user can set up in the preferences. Note: Although it is an option, I opt to not add an MP3 file set at this point. This is done on purpose and will be explained later. If you do not have XLD set to automatically save the Rip log, make sure to save a copy of the Log file in the parent album folder. An example of a log report is provided.

Step 3: Once you have your set of WAV and FLAC files, open xACT and choose the "Checksum" tab.

Step 4: Select all of the WAV files and drag them into xACT. Press checksum and you will be prompted with a dialog box. Select "shntool," name the file, and save it to the parent album folder. Once you have generated the ST5 checksum you can delete the WAV files.

Step 5: Select all of the FLAC files and drag them into xACT. Press checksum and you will be prompted with a dialog box. Select "Fingerprint," name the file, and save it in the parent album folder.

Step 6: Open Tag, and drag-and-drop all of the FLAC files into the program. A drawer should pop open and the files should be visible in a list. Select all of the files in the list.

Step 7: Select "Guess Tags" from the Tools menu, or simply press command-G. A dialog box will appear. Fill out the appropriate tags. Remember the formatting must be identical to the actual file name otherwise the tags will not be properly created. Once you have everything properly filled out, press "Guess" and Tag will automatically populate the specified fields.

Step 8: Fill in any remaining metadata fields you desire. Remember to select all of the files to change the metadata for all files at once. When a file's metadata has changed it will be bolded in the tray window. Make sure to save all of your changes before exiting.

At this point you have created a verified and verifiable, losslessly compressed digital copy of your original CD. At this point I like to create an MP3 version of my CD and add it to my iTunes.

Step 9: Open Max. The first time you open Max it's a good idea to set your preferences. Select the appropriate encoding format under "Format" tab. Be sure to set your encoder settings how you like them. Like I mentioned before, I encode MP3s at 128kbps because I can always go back and make a higher quality one now that I have a FLAC copy. You can specify the MP3 default file name structure under the "Output" tab, and you can automatically add encoded files to "iTunes" under the "iTunes" tab.

Once you've established your preferences, select "Convert Files" from the File Menu and add your FLAC files. Press "Convert" and you'll have MP3 files in a few short minutes. Add these files to iTunes if you have not already done so.

Note: Depending on how particular you are about your ID3 tags in iTunes you may need to go and edit the MP3 tags. Fortunately, this is a since in iTunes, and one of the reasons I'd like to see native FLAC integration. iTunes has many more tag fields than the standard FLAC file and you would have to go back and fill in Album Artist field and any other options a particular album might require. Also, if you insert comments in your FLAC tags, for some reason Max does not write those to the MP3 files. If the comment was important, you would have to re-add that. FLAC files can technically have Album Art added, but Tag does not seem to support this function. There is no tag field for lyrics either.

Why did I wait to create MP3 files rather than just have XLD create them for me? The answer is in the metadata. WAV files do not support metadata so the information you edit in XLD only goes so far. For other formats XLD will write metadata. If you have basic tagging needs, this will probably suffice for you and even save a few steps along the way. I tend to be pretty particular about my metadata tagging and prefer the above method because it provides greater flexibility. Using XLD's auto-tagging doesn't make much sense if I plan to revise or add additional tags to my files, so I do not use the option in XLD.



Additional thoughts, loose ends, and explanations

As you might be able to tell, this is somewhat of a cumbersome work flow and unless you work from home and can do all of this in the back ground, or you have a lot of time on your hands, creating a FLAC library, especially of a large CD collection would take a long time. Unfortunately, there is no single program that can do everything I need it to. One of big things seems to be generating ST5 and FLAC Fingerprint (FFP) checksums.

Ideally, there would be one program that would have all of the components of this work flow that could be customizable as the user needs. If that were the case it would be possible for the end user to write a script to cover all the steps laid out above. Because this work flow spans several programs that task becomes increasingly difficult. (If there is anyone reading this who knows about AppleScripts and can help with this, please let me know.)

Throughout all of this I may come off as critical of Apple. Don't get me wrong, I love Apple and most of their products, however, they are not without flaw. I don't understand how they can have multiple lossy codecs native to iTunes, but ignore the accepted standard in lossless compression (FLAC) in lieu of their proprietary ALAC. FLAC is open source, so it shouldn't be an issue of cost. It's really beyond me because they don't do anything with that codec. If they sold music on iTunes in ALAC it would make sense (and do the music a favor, too), but business isn't about what's right, it's more about what will make money.

Some may say, why go through all this trouble when I can just rip a CD with iTunes? Doesn't iTunes have error-correction? The short answer is, yes. However, specialty ripping programs are capable of verifying the integrity of a rip and are much more thorough than iTunes. If you're an audiophile, iTunes is not doing you any favors.

Others might wonder if they can use a program like Max to convert their MP3s to FLAC. Obviously, the answer is, again, yet. However, there really is no point in doing so. Think of the WAV (or FLAC) to MP3 conversion in terms of analog equipment. In the days of making mix tapes if you dub a track from one tape onto another, the audio on the new tape loses a generation of quality, and is inferior to the original. The same thing happens when you encode from WAV/FLAC to MP3. Remember that WAV files process at 44.1kHz/second. That's really fast. Compare that to an MP3 where the number drops from 44,100, to 128. Even a 320kbps MP3 pales in comparison to the original file. And while most people might not be able to tell the difference, if you encode a file to 128 MP3 and back to WAV, you would be able to tell the difference. The situation is the same between MP3 bit rates. A 128 MP3 can be convert to 160, 256, or even 320, but it will not improve in quality at all. Sure, the file size will grow, but the actual audio quality will decrease in this conversion. An MP3s audio quality is, at best, as good as its lowest encoded bit rate.

Finally, let me address the issue of checksums. Many folks on the internet like to use CUE files, which makes no sense to me. A CUE file is only really useful when a CD is ripped as one single track and a playback program needs to know where to insert the track splits. Since most people would never think to rip a CD as one long track, the use of CUE files is moot.

The benefit of having checksums created at the time of the initial ripping/encoding is that you have a means to detect file corruption, and the ability to verify the quality of the files that you have. Originally I used MD5 checksum files, but later learned of their limitations. Once you encode a file and generate a MD5 checksum for it, any change to the file, including metadata, will cause the file to fail it's checksum verification. For this reason, I switched to ST5 checksums, which look at the actual wave form of the audio and not it's container format. I use ST5 checksums for WAV files. Using MD5 for FLACs is a big problem because of metadata tagging. Everyone wants their music tagged their own way, but changing that information causes the MD5 to fail. By using FFPs one can avoid this problem. The FLAC metadata is essentially tacked onto the end of the file and the FFP checks the audio portion, but ignores the metadata section. If you make a typo in your metadata and find it after you generate your checksum you can simply edit the metadata if you have FFPs, whereas it would be necessary to regenerate a new MD5 for the file. The rip LOG simply verifies that a CD was properly ripped.

So used in concert, the LOG, ST5, and FFP verify the quality of your music and make it verifiable at the same time.

For more on checksums, see: http://www.thetradersden.org/forums/showthread.php?t=17951

Have I forgotten something? Is anything unclear? Provide any and all feedback in the comments section.