Quick advice on storage and backup

Editor’s Note: Since I’ve turned Facebook comments off, I’m experimenting with turning them on directly in the blog. Feel free to ask any questions you may have about this subject at the bottom of the page. No log-in required.

I’ve given this advice several times recently, so I figured I’d turn it into a quick blog post.  If you know someone who is struggling with decisions on how to store or backup media files, please share.

For most people, reliable storage and backup has gotten really cheap and easy to implement, even for people with lots of photos and video.

The trick is to make use of the newer high-capacity drives and affordable cloud backup services. So here’s what I end up telling most people who ask. Note that this advice is designed for people with a data set smaller than about 10TB who don’t need multi-user access.  And, this advice assumes that you have a media collection you want to store and protect, and that it exceeds the capacity of your computer’s internal drive.

General Advice
This holds true for most photographers and (since everyone is now a photographer) also for “regular people”.

  • Use modern big drives – 10-14 TB drives are newer, better designed, more reliable and much faster than older ones. They are also very affordable. If you are using multiple smaller drives, it’s time to replace with a single larger one.  Even if your archive is only a few terabytes, go for one of the large drives.
    Currently, I recommend G Technology drives. 10TB is the current sweet spot for prices ($300). You can bump up to 14TB for a total of $450. (I’ve linked to the USB3.1 version of the drive. You can also get these drives in Thunderbolt, but it won’t be any faster with conventional spinning disks.)
  • Use as few drives as possible – If you can easily get all your stuff on one single drive, do it. It’s much easier to backup and restore than an array of older drives. You don’t need to remember what is where.
  • Get a drive to make a “twin” onsite backup – Again, simpler is better, and big drives are your friend. This is what protects you against drive failure. Much easier to make, keep current, and restore from a local backup than a remote one.
  • You also need an offsite backup to protect against fire or theft – There are two main method to do this. You can get a second backup drive and keep it offsite. You can also use a cloud service to make a 3-2-1 compliant backup. I do both, but let’s handle them independently.
  • Backblaze cloud backup – Backblaze is a great cloud service that I depend on for my own work. The personal version of the service offers unlimited file backup for $60 a year. Uploaded files are encrypted for privacy. You can add external drives to the backup, and it happens automatically in the background. If you are a Photoshelter Pro customer, you can also use their service as your cloud backup. Unlimited storage is included with the Pro accounts.
  • Additional drive for offsite backup – An offsite backup drive provides excellent protection, and quick restoration in the event of a problem. Store it offsite – in general, someplace easy is better than someplace really secure. If you are worried about the data falling into the wrong hands, you can format the drive with encryption. Keeping an offsite drive updated can be difficult, and there are almost always gaps between what’s currently on your primary storage and what’s on the backup. This is why I currently favor Backblaze, especially for working files.
  • Avoid spanned drive devices (like RAID, multi-drive NAS, Drobo, etc.) unless:
    • You really need them for a particular reason. Most people don’t really need single storage volumes larger than 14TB.
    • You understand how to maintain them or have a good tech service to use. These are little computers running Linux, and in general require maintenance, monitoring and updates. I know many people who have experienced total failure of spanned disk devices.
  • If you outgrow the single-drive units, get an additional set – This is getting outside the scope of this post, but worth mentioning. If you can’t fit everything on one drive (say you have 20TB data), then I suggest getting an additional set of drives, rather than going to a RAID device. In most cases, it’s easier, cheaper and safer.

This advice is drawn from content published in The DAM Book 3.0. If you’ve got a larger or more complicated storage and backup job in front of you, you’ll find a lot more discussion over there.

9 thoughts on “Quick advice on storage and backup”

  1. Sorry to hear of the failure.
    Yes, recovery from a bad single drive enclosure (or JBOD enclosure) is typicallyfaster, easier and cheaper than recovery from span drive failure.

  2. So, I am one that has experienced a disaster(s) with a spanned drive. For several years, I have kept my small photo library on Drobo devices, set up as RAID 6. I’ve owned four of them. The first was a very slow 4-bay USB unit that was upgraded to a faster SATA connected 5-bay unit. It failed and was replaced by a similar 5-bay thunderbolt connected unit. Now, it has just failed after three years. I still have a 5-bay NAS unit running which was not being used much, but of course, the disk pack from the failed unit is not compatible. I am currently using it as a temporary backup during this current rebuilding project. The failed spanned device is temporarily being replaced with a single 8tb external drive.

    During the operation of these devices, I’ve had several single-disk failures, and the concept of no downtime to rebuild from backups was very nice. I am currently using Western Digital Red drives (as recommended by Drobo). Perhaps using enterprise-class drives would have eliminated some of the drive failures?

    Using a spanned drive or not, I am done with Drobo. A three-year lifespan for such a critical product simply isn’t acceptable in my opinion.

    I think your advice about using a single large drive as primary storage, in a RAID 1 configuration, is good. However, my experience was really an ‘enclosure’ failure, and the single large drive concept would be susceptible to that as well. Of course, the benefit would be that the single drive would still contain readable files, unlike the spanned proprietary data which is not even readable different models of the same manufacturer.

  3. Albert,
    Yes, in order to fully restore, you want both the files and the catalog. All settings are stored in the catalog, so they can be applied to backup versions of the files to get you back to where you were.

    While its possible to save some settings into the files themselves, you can’t save all Lightroom work into the files (Collection data, Virtual Copies, snapshots, Publish services data all lives in the catalog only.)

    Yes, replacing old drives like these is well advised, if budget allows.

  4. Hi Peter
    I have a mirroring system with two 3.6 TB HDD so that a copy is automatically made to the second disk.
    I now have almost 23K images, and these HDD already are 3-4 years old, so I’m looking to get the 10 or 14 TB that you have suggested.
    I also use Backblaze for online back up. All my LR images are stored as DNG. Backblaze backs up both the image files (source and LR DNG) and the LR catalogue. However, I noticed that, when I tried to restore a rendered file from Backblaze, it did not include the LR adjustments. I suppose that, when faced with a disk failure, I’ll need to restore from Backblaze both the LN DNG image file and the catalogue to see the rendered version of the image. Is this right?

  5. Roza,
    As I point out in the book, RAID used to be desirable for one of several reasons:
    1. You needed a large volume, and disks were just too small to hold entire projects.
    2. You needed more speed for working files since drives were comparatively slow.
    3. A multi-drive configuration can help provide resilience in the event of a single drive failure.

    Items 1 and 2 are less necessary now, due to large drives and to very fast SSDs for working files. And item 3 turned out to be more complicated. I have personally hear more photographers lose entire RAIDs than have a RAID save them from drive failure. If you understand how to maintain it, great, but too many people don’t realize that RAID is not set-it-and-forget-it.

    If your collection exceeds the size of these large drives (14TB), then you need to use multiple volumes. In general, the recommendation I have for people who can’t use cloud, is to consider splitting your archive into single volumes that can fit on one drive (and therefore one set of primary and backup drives).

    For many people, much of the stuff they need to store is unchanging data (old raw files, archived projects, original video footage, etc). If you can split that into a separate drive set, it can be easy to archive safely.

    There are a number of approaches, and in the book I outline how to look at your own needs and create the simplest system you can that provides the capacity, performance and safety you need.

    As to duplicating offsite, if your internet is too slow, then moving drives offsite by carrying them may be the only real option.

  6. Great summary – clarifies a number of issues really neatly, thank you Peter.
    I’m interested in why you advise against RAID. I am looking at revamping my storage at present and had been thinking about a RAID 1 to mirror onsite, with a duplicate offsite (swapped out fortnightly). Cloud storage is out of the question because of the woeful internet connectivity available at my location (ranges from 0.3 – 2Mbs upload). Do you have a recommended software for duplicating the offsite copy?

  7. Mark,
    Definitely SSD for boot, applications, catalogs and works in progress. Spinning disk is still needed for a lot of photographers/videographers due to cost issues. And the new large capacity spinning disks have made some real leaps forward in speed due to multiple independent read heads.

  8. Good sound advice, Peter.

    What do you think of using SSDs for your primary?

Leave a Reply

Your email address will not be published.