Amazon S3

Posted on August 29, 2007

I finally bit the S3 apple at work today. I've been using S3 for a bit with Bandwagon for backing up my iTunes files, but I hadn't seriously considered how to best use it at myYearbook.com until I had a call today with one of Amazon's Business Development folks. It was during the call that instead of using it as a CDN as I had been thinking, and buying the optical backup system I had been contemplating, that it was a perfect disaster-recovery backup mechanism.

Since I could sync up with our S3 buckets on a fairly regular basis and they can be served publicly off of S3 in a pinch, I could focus on putting systems in place to keep S3 in sync instead of installing, managing and using an optical backup system. In addition, I didn't have the up-front purchasing cost for the system that we would have to put in place to be able to back up the many terabytes of data we have. Where media, power and rack space are a commodities that I'd pay for on an ongoing basis, the cost per megabyte when all is said and done is fairly similar and I have the option of restoring to any location with an internet connection instead of having to purchase additional hardware for out of data-center restorations.

Installation of s3sync.rb was very easy, and while the documentation is terse, it took me all of about an hour to compile and install ruby and have the app working. A few shell scripts later, some crontab additions and we have a regular, off-site backup taking place. Restoration is equally as easy, as s3sync.rb works similarly to rsync. All things considered, off-site backup has never been easier.