The Music Project

I was talking to my friend Brian Goetz recently, and he reminded me of a blog entry he posted a while back. He's digitizing his entire music collection, and he's done all the research. This is appealing to me because I recently switched over to zero reliance on the shiny plastic disks once I've ripped them. All my music consumption is now electronic, mostly via iPod (the car was the last frontier, and now that's switched over as well). I have a large-ish CD collection (~1500 CDs), so translating them to electronic form is no small feat. And I only have to do it once for the rest of my life.

Based on my conversations with Brian, I realized the following:

• Ripping and encoding are completely separate activities
• The most important thing is to get a reliable, error corrected rip
• You don't want to get stuck with a proprietary format (as much as I love Apple, I don't want my music collection tied up in one of their formats forever)
• Lossless is the way to go, so that you don't loose any data in this process
• Cross-encoding allows you to rip your music in a "core" format and selectively support proprietary formats and reduced file sizes (and lossiness)

His original entry is about his Windows and Linux based infrastructure. I replicated his same setup (with minor differences) on the Mac.

A Home

First, you need a home for all these files. I recently got a network attached storage device to hold our growing collection of digital photos. I'm my family's tech support, and if anything ever happens to those photos, I'm a dead man! After doing a lot of research on quality, Mac compatibility (my household is 100% Mac now), I ended up with the NETGEAR RND4250 ReadyNAS, which comes with 500 GB of storage (2 mirrored 500 GB drives), with a capacity of 2 TB. I ended up bumping it up to a TB when I realized how big the combination of music + pictures would be, and the Netgear handled it beautifully.

Ripping

I didn't realize this, but most rippers don't take advantage of the error correction bits on the CD, and they are wimps: they give up way too easily. The trick is to find a ripper that is relentless and tries its hardest to get all the bits off the CD. The ripper/encoder I ended up using is Max, which supports Leopard in it's latest (allegedly) unstable form, and previous Mac OS X versions in it's previous (stable) release. I caveat "unstable" because I used it a lot and it was rock solid for me on Leopard. One of the nice things about Max is its support for different rippers: fast & carefree, or paranoid. The latter is the one I want, based on the cdparanoia project. You can configure this ripper to never give up until it gets a clean read of the disk. Several of my CDs trundled for 6-10 hours before Max finally reported success, including a couple that I'd given up for dead.

Encoding

OK, so now I have a good rip, I need to encode it (this post makes it sound like separate steps, but Max handles both for you). As I stated earlier, I don't want to get trapped by a specific format. I ended up (like Brian) choosing FLAC. FLAC is a open standard encoding for music that offers lossless compression, which is what I wanted. The FLAC spec also allows for more aggressive compression without loss of data, depending on your patience. It's designed to take longer to encode but have no impact on playback time. I choose the most aggressive because I have time, hardware (a MacPro with 4 processors), and I want to conserve space if possible. But, iTunes (which is how I play and sync my music) doesn't support FLAC. Max to the rescue: it will let you do parallel encoding. I set Max up to encode the ripped music files to both lossless FLAC and lossless MP4 (Apple's format). The only downside is that it won't allow you to choose different directories for the encoding. The FLAC files I'm placing on a RAID mirrored network-attached storage drive (remember, I never want to do this again!). So, I ended up writing a little Rake file to handle automatically moving the files from one place to another. I rip them all to the RAID drive, then let the script move them (preserving directories) to the other. The script is here, if anyone wants it (no warranty expressed or implied -- you'll have to change all the directories, and if you use this to erase your hard drive I'll shed a tear for you, but might just laugh).

task :copy do
count = 0
skipped = 0
FileList["**/*.m4a"].each do |f|
artist, album = recording_info_based_on f
if File.exist?  "#{DEST}/#{artist}/#{album}/#{File.basename(f)}"
  puts "\tsomething is amiss; I'm skipping: #{f}"
  skipped += 1
else
  FileUtils.mkdir "#{DEST}/#{artist}" unless File.exist? "#{DEST}/#{artist}"
  FileUtils.mkdir  "#{DEST}/#{artist}/#{album}" unless File.exist? "#{DEST}/#{artist}/#{album}"
  puts "#{artist} - #{album} - #{File.basename(f)}"
  count += 1
  FileUtils.cp f,"#{DEST}/#{artist}/#{album}"
end
end
puts "copied #{count} files\nskipped #{skipped} files"
end

def recording_info_based_on filename
File.expand_path(filename) =~ /.*\/(.*)\/(.*)\/.*/
return $1, $2
end

I also made a rake task to report any that ended up missing from the original FLAC directories to the AAC files (just in case something went amiss during a copy process, or I screwed up and deleted something by mistake). I want to make sure that the convenience Apple-format files match the canonical source (the FLAC) files. So, this is the "missing" rake task:

task :report_missings do
count = 0
FileList["**/*.flac"].each do |f|
artist, album = recording_info_based_on f
dest_file_name = File.basename(f).sub /flac/, "m4a"
unless File.exist? "#{DEST}/#{artist}/#{album}/#{dest_file_name}"
  puts "missing #{f.sub /\.flac/, ''}"
  count += 1
end
end
puts "found #{count} missing files"
end

Result

It took me about 2 months of ripping while I'm around my computer, running 2 computers (my laptop and desktop) in parallel. In the end, though, I ended up with 453 GB of music files, the FLAC ones safely tucked away on a mirrored drive and the M4A ones on my desktop, ready to be synced to my iPod (or a subset of them, anyway). Now, when I get a new CD, I rip it using Max to the NAS and either copy the files by hand (if it's just one CD) or use the Rake file to move lots en-masse. Storage is now dirt cheap, and I've leveraged almost a terabyte of it keeping the music files in 2 formats. I also recently bought a portable 500 GB drive so that I can keep all my music with me on the road. It's a copy of the desktop M4A files, but it's easy just to mirror the Music directory from the desktop to the portable drive.

I achieved my goal: an open archival format that I hope will be around for a very long time, and a convenience version for the way I happen to consume them today. And the shiny disks? I put them all in binders, so that if I ever need one of them (or it's sleeve), I can rummage around in the (mostly) alphabetical CD volumes. I didn't put a huge amount of effort creating an expandable storage that makes it easy to keep them in strict order because that would take lots of effort and it isn't something I expect to have to do often. If it turns out I got back to them all the time, I'll invest the time then.