Problems with NAS, looking at FW400 or USB2 alternatives

[David Anderson]

David,
I have about 1.5 terabytes of RAW/DNG files.  Each of these goes through several states before it is archived and backed up.  This would make the backup drive array pretty large.  I'm assuming that I would need RAID instead of multiple volumes for this. (If I configured such a system last year, it would probably be a 16 drive box, or some large tape backup system.

This is a lot of hardware expense and administrative overhead to backup a few megabytes of changing metadata.

The simpler system, in my view, is to segregate the changing metadata from the large data set of unchanging image data.

Am I missing something?
Peter

Peter,
My comments were really addressed at the original poster, who, like me, has relatively modest image storage requirements. The OP has about 110GB, while I'm still only at 35GB. That's a long way from the 1.5TB that you have! The cheap LaCie 300GB Firewire 800 external hard disks I use currently store everything I need to backup. With the progress of disk storage technology, I confidently expect to buy 500GB replacements for about the same money in a year or two and thus keep pace with my growing archive.

I don't think you are missing a thing. You have simply made a cost-benefit analysis of multi-terabyte backup hardware options, such as disk arrays and tape autoloaders, and concluded that the key thing is to backup the original images with at least the first-pass ACR and metadata edits - as well as the IVMP database. You are faced with a far more significant financial decision than the OP or myself. In your position, I'm sure I would make the same decision as you have.

David

[wombat2010]

At a simplistic level, if you have 100GB of images then you would need a 500GB external disk to hold 5 historical versions of every file. You would then set Retrospect's grooming option to keep no more than 5 versions of any file and old versions would be automatically erased from the backup set as new ones are added. But this is a worst case scenario. You would not be changing every image on a regular basis. Many would never change. The changes made to Raw files are stored in small sidecars, so the Raw files never change. Updates to ACR settings and metadata for DNG files, however, would involve a DNG file update every time. Have you had a problem with grooming not working as expected?

Thank you for your response, David.  You make a good point about the space needed to keep 5 backup versions of 100 GB of images (and I'm only at about 55 GB, but I back up other things, too, like music, downloads, and documents).  I think I thought a backup set for a given day/week included all files from that date, so that if you had 5 sets, each a week apart, earlier sets would only be smaller by the number of images you added the week after Retrospect created the set.  Also, excellent point about RAW files and their sidecars--even had I understood the previous point, I am not sure it would have occurred to me that, while I may tweak RSP settings, my NEFs never change.  I am not on the DNG train at the moment, so I don't have those issues.

Assuming I do move to keeping 5 versions of files, how would you recommend breaking them up?  One a week old, and then just the last 4 days?  A month, a week, and three days?  If one were to keep only 3 sets, would your suggestion change?

I know nothing about SyncbackSE, but Retrospect also has its 'Duplicate' facility that lets you store your files in a normal non-proprietary format.

Because you understand backup software so well, if you ever have the time and inclination to browse SyncbackSE's feature set, I would love to hear whether you think it is a viable backup option.  It is clearly not as robust and flexible as Retrospect, but it has a "friendlier" GUI, which is what attracted me to it.  Its list of features can be viewed at http://www.2brightsparks.com/syncback/sbse-features.html.  It's not a huge list.

I did not realize what Retrospect's "Duplicate" function did.    Thanks for clarifying.

I am fairly paranoid about losing my images, but I feel quite comfortable with my current setup of a pair of inexpensive 300GB external Firewire hard disks (one held off site) and two sets of DVD-Rs that hold all my original images and at least two versions of my derivatives (all held off site).

I am fairly paranoid, too.  I have a similar routine, with one 300 GB external eSATA drive, a Linux server on another floor of my house acting as a backup server, and two off-site sets of DVDs that I try to keep fairly current.  Actually, keeping current images offsite is definitely my big weakness, because I don't like burning DVDs of my working folders all the time, so all of my backups are on-site, which makes me nervous.  If my house burned down, my recent shots likely would be lost.

Thanks again, David.  Your comments and suggestions are very helpful.

Stephen

[David Anderson]

Assuming I do move to keeping 5 versions of files, how would you recommend breaking them up?  One a week old, and then just the last 4 days?  A month, a week, and three days?  If one were to keep only 3 sets, would your suggestion change?

It doesn't work that way. This particular Retrospect 'grooming' option is simply a counter, and the process is automatic. If the counter was set to 5 and you made daily changes to the file and backed up every night for a week, then all 5 versions stored by Retrospect would be less than one week old. At the other extreme, if you only ever made one change to a file since it was first backed up a year ago then Retrospect would simply store those two versions. BTW, you can get lots of Retrospect advice and help at this user forum: http://forums.dantz.com/ubbthreads/ubbthreads.php/Cat/0

Because you understand backup software so well, if you ever have the time and inclination to browse SyncbackSE's feature set, I would love to hear whether you think it is a viable backup option.  It is clearly not as robust and flexible as Retrospect, but it has a "friendlier" GUI, which is what attracted me to it.  Its list of features can be viewed at http://www.2brightsparks.com/syncback/sbse-features.html.  It's not a huge list.

I doubt that I'll ever devote time to checking out SyncbackSE in detail. I've still not found time to study all the features of Retrospect! SyncbackSE may do a perfectly decent job, but I feel more comfortable with a package that is a single user version of a tool created for a demanding small and medium business environment. That more 'industrial strength' capability does, however, lead to a product with a steeper learning curve. In my business career I used Arcserve backup software but when I last looked that was not priced for the home user market. Retrospect seemed to fill the gap nicely.

I am fairly paranoid, too.  I have a similar routine, with one 300 GB external eSATA drive, a Linux server on another floor of my house acting as a backup server, and two off-site sets of DVDs that I try to keep fairly current.  Actually, keeping current images offsite is definitely my big weakness, because I don't like burning DVDs of my working folders all the time, so all of my backups are on-site, which makes me nervous.  If my house burned down, my recent shots likely would be lost.

If you follow Peter's recent point, then you will realise that it is not a "big weakness" for your offsite DVDs to lack the latest ACR and metadata changes. While undeniably important if you have put a lot of effort into them, they are far less important than the image data itself. Like you, I don't burn DVDs on a frequent basis - but I do protect my most recent shots by backing them up to my two Firewire drives, one of which is always offsite. I bought the second Firewire drive within days of the first, as I wanted the extra protection that two external drives provides. With only one such drive there is always the risk that your backups are delayed (because of the extra effort to retrieve the disk from the offsite location) or the risk that you lose the disk through fire or theft (because it was left onsite). Having two disks also provides protection in case of disk faliure.

David

[wombat2010]

It doesn't work that way. This particular Retrospect 'grooming' option is simply a counter, and the process is automatic. If the counter was set to 5 and you made daily changes to the file and backed up every night for a week, then all 5 versions stored by Retrospect would be less than one week old. At the other extreme, if you only ever made one change to a file since it was first backed up a year ago then Retrospect would simply store those two versions. BTW, you can get lots of Retrospect advice and help at this user forum: http://forums.dantz.com/ubbthreads/ubbthreads.php/Cat/0

Thank you again for the explanation.  That makes perfect sense.  Thank you, too, for the link to the forum; I think I visited there once before, but I will check it out again as I come up with new questions.

I doubt that I'll ever devote time to checking out SyncbackSE in detail. I've still not found time to study all the features of Retrospect! SyncbackSE may do a perfectly decent job, but I feel more comfortable with a package that is a single user version of a tool created for a demanding small and medium business environment. That more 'industrial strength' capability does, however, lead to a product with a steeper learning curve. In my business career I used Arcserve backup software but when I last looked that was not priced for the home user market. Retrospect seemed to fill the gap nicely.

I know what you mean about investigating software.  I have managed to avoid trying more than one DAM application for that very reason--I investigated, trialed, and have stuck with idImager (I'm a Windows guy), which is a very solid program.  I may just stick with Retrospect, too.  I chose it for the same reasons you did, but it can be a little intimidating at times.

If you follow Peter's recent point, then you will realise that it is not a "big weakness" for your offsite DVDs to lack the latest ACR and metadata changes. While undeniably important if you have put a lot of effort into them, they are far less important than the image data itself. Like you, I don't burn DVDs on a frequent basis - but I do protect my most recent shots by backing them up to my two Firewire drives, one of which is always offsite. I bought the second Firewire drive within days of the first, as I wanted the extra protection that two external drives provides. With only one such drive there is always the risk that your backups are delayed (because of the extra effort to retrieve the disk from the offsite location) or the risk that you lose the disk through fire or theft (because it was left onsite). Having two disks also provides protection in case of disk faliure.

You're right.  The risk for me is not the lack of saved RAW edits; rather, it is that sometimes I go a week or more before getting my RAW files themselves backed up offsite (although they are backed up on two different drives at home).  I should just get another external drive and keep it offsite, even if I only use it for my working directory.  They are certainly cheap enough now.  I may also try online storage for my current working files--it's quite cheap and has the added advantage of making files accessible anywhere.  It would have been unfeasible with cable internet access, but with FiOS, it may be just fast enough to be workable (I consistently get 2 Mb/s upstream).  I just haven't gotten around to trying it yet.

Thank you again, David.  I am sure others who frequent the forum have appreciated your thoughtful responses as much as I have.

Stephen

[Steve Miller]

Okay, so I got my my LaCie 250 GB drive that will serve as my primary backup. After re-reading the thread, I have several thoughts and questions that come to mind. Before getting to those, I have one setup question for any Mac users that want to co-exist in a Windows environment. The drive came formatted in FAT32. As I was told by LaCie, the limitations of that file system (if that's what it's called) are that no file can be larger than 4 GB (shouldn't be a problem) and that filename's are limited to 31 characters. Since reading Peter's book, I've made sure to keep filenames to 31 or less characters. The problem is that many of my old files have longer names (I had been told to keep it below 64). LaCie told me that using NTFS would make the drive read-only for Mac (obviously no good). They recommended HFS+ Extended (Journaled) to give the best of all worlds, noting that I would need MacDrive on my PC to read the FW drive. Since I already have this program for the Win side of my MBP, this seemed like the way to go. Just wanted to confirm this choice with any of the gurus here.

Now back to my original post.

I ran into problems using Retrospect. EMC said that I would have no problems using the Mac version, controlling a Windows PC through the network via the Win Client, and backing-up to a FW connected to the PC. (Remember, as I make my transition to all Mac, while waiting for a UB version of PS, all photos reside on my PC, with the Working Files also on the Mac. More on workflow later.) Well, it turned out that Retorspect didn't let me duplicate via the network from the PC to the FW connected locally to the PC. It said that I couldn't copy from a source to itself. Apparently, since I was initiating the Duplication from the Mac (as that is the OS for which I'll buy the software), it viewed all PC drives as the same source, regardless if they were internal or external drives. David, I'll call EMC today, but I posted this here in case you have a solution.

Since my immediate concern was getting a good copy of all of my photo files on the FW drive, I did a simple copy from Windows to the drive. Good news is that it was easy and I now have a complete, off PC copy (aside from the 24/7 NAS that's in danger of blowing up). I was surprised by the speed. I copied all 110 GB in large-ish chunks. My Original RAWs are about 30 GB and took about 30 minutes to copy. While this was almost six times faster than my previous experiences with the NAS (16.7 MB/s vs. 3 MB/s), for which I was very grateful, it was still much less than the specified speed of closer to 40 MB/s. Should I be getting faster speeds? Could a slowing PC be the cause? Love the new drive and its relative speed, but I'm curious.

David, another Retrospect question if I may. You noted that you backup your files. My impression is that Retrospect's backup uses a proprietary format. Does that mean that you don't actually see each individual file on the backup drive (i.e. can't be read in a normal browser)? Personally, I don't like proprietary formats, and as will be seen later, depend on being able to use the backup to go from one machine to another. Am I correct about Retrospect and the way in which it backs up files? If so, I assume I need to use the Duplicate function?


Thanks in advance,

Steve

P.S. I'll follow this up with a hardware/workflow related post. Peter, if you feel it is better placed in another sub-forum, feel free to move it.

[David Anderson]

I ran into problems using Retrospect. EMC said that I would have no problems using the Mac version, controlling a Windows PC through the network via the Win Client, and backing-up to a FW connected to the PC. ............Well, it turned out that Retrospect didn't let me duplicate via the network from the PC to the FW connected locally to the PC. It said that I couldn't copy from a source to itself. Apparently, since I was initiating the Duplication from the Mac (as that is the OS for which I'll buy the software), it viewed all PC drives as the same source, regardless if they were internal or external drives. David, I'll call EMC today, but I posted this here in case you have a solution.

While I have successfully used Retrospect to make a backup via the network of data stored on a remote PC, I've always been backing up the data on a locally attached external drive. I've never tried backing up to an external drive attached to the remote PC. If I get some free time I might try this out, but I can't promise to do this today.

....I was surprised by the speed. I copied all 110 GB in large-ish chunks. My Original RAWs are about 30 GB and took about 30 minutes to copy. While this was almost six times faster than my previous experiences with the NAS (16.7 MB/s vs. 3 MB/s), for which I was very grateful, it was still much less than the specified speed of closer to 40 MB/s. Should I be getting faster speeds? Could a slowing PC be the cause? Love the new drive and its relative speed, but I'm curious.

Your 16.7 MB/sec translates to 1002 MB/min, which is not hugely different from my own experience (ref figures quoted in my earlier posts). Rather like the standard Ethernet network speeds, I suspect that the disk manufacturers' claims have to be taken with a big pinch of salt!

David, another Retrospect question if I may. You noted that you backup your files. My impression is that Retrospect's backup uses a proprietary format. Does that mean that you don't actually see each individual file on the backup drive (i.e. can't be read in a normal browser)? Personally, I don't like proprietary formats, and as will be seen later, depend on being able to use the backup to go from one machine to another. Am I correct about Retrospect and the way in which it backs up files? If so, I assume I need to use the Duplicate function?

Yes, the Retrospect backup files are in a proprietary format and cannot be read by other apps. This caused me some initial concern, but now I have stopped worrying about it. Even if EMC were to go out of business, I would still have the ability to restore all my backup data to a normal format using the version of Retrospect I had at that time and could then back it up again using some other product. In your case, the worst case is that you have to buy both a Windows and Mac version of Retrospect in order to restore on either machine (though my previous understanding of what EMC have told you is that one licence would be enough). As per your assumption, the Retrospect Duplicate function does not store the data in a proprietary format (but it lacks the standard backup method's ability to save multiple versions of each file).

David

[Steve Miller]

Following on from my post above, I wanted to get people's comments on my incorporating the new backup drive into my workflow.

I currently keep all original and derivative photo files on my desktop PC. I have enough HD space to last until next spring, at which time I expect to get a Mac Pro as CS3 should be available. I have CS2 on my PC and MacBook Pro (the Win version - via Boot Camp). I also have iView Media Pro on both systems, though as will be shown, only plan on using it on my Mac. Because of my PCs location in my house, I prefer to do all non-color critical work on the MBP, saving any serious ACR/editing work for the PC/ACD. To date, I haven't really used iVMP as I still had a bit of organizing to do before I imported a ton of files.

My current workflow is to:

1. Bring images onto the PC side of my MBP using Downloader Pro (and now with the FW drive, will copy these originals to the "From Camera" sub-folder in my Working Files directory).
2. Do all keywording/rating/etc. and a crude ACR edit. All of these files sit in subfolders within my Working Files directory.
3. I then copy these files onto the FW drive, thereby updating the original files backup.
4. This Working File directly will then be copied over to my desktop PC, where I will then finish all ACR/editing, then convert all RAW files to DNG.
5. Once done with all of this, I will move the "final" edited RAWs and DNGs to the Archived Photos directory in Original RAWs and Original DNGs sub-folders on my PC (sorry, can't seem to get over the hump of deleting the RAWs).
6. I'll also copy the Original RAW/DNG files in their appropriate folders to the FW drive (yes, I use Peter's bucket system).

At this point, I have a copy of the original RAW and DNG on my PC and the FW drive (also have off-site DVD copies of Originals and need to do so for Derivatives). I'm finally at the point now where my buckets are organized and I can start to use iVMP. Since this is the perfect type of task to do on a notebook, I am beginning to do the work on the MBP. I'm a little uncertain how best to incorporate it into my workflow. So far this is what I'm doing:

1. Import all of the Original DNGs from the FW drive into iVMP on my Mac (I'll get to the derivatives eventually).
2. Start the task of adding keywords/ratings to my older photos (I've been pretty good over the past 18 months).
3. Setup catalog sets where appropriate.
4. Sync the metadata with the FW drive originals.
5. Backup the FW drive to the PC so that I have updated files in each place (this is where I can see a problem because I'll have to make sure that I have the "current copy of the PC folders/files on the FW before syncing the metadata from iVMP to the FW drive).

Something tells me I'm making this whole thing more complicated than need be, perhaps because I want to utilize the MBP for non-color intensive work.

Any thoughts/comments are much appreciated.

Steve

[Steve Miller]

As per your assumption, the Retrospect Duplicate function does not store the data in a proprietary format (but it lacks the standard backup method's ability to save multiple versions of each file).

David

I don't mind not having multiple versions of each file, if anything, it would probably confuse me more than help. Do you know if Retrospect gives you the option of doing an incremental duplicate? i.e. when duplicating from one drive to another, does it copy everything or just those files that have changed on the source drive?

Thanks,

Steve

[David Anderson]

Steve,
Yes, the Duplicate process is incremental.

David

[peterkrogh]

So far this is what I'm doing:

1. Import all of the Original DNGs from the FW drive into iVMP on my Mac (I'll get to the derivatives eventually).
2. Start the task of adding keywords/ratings to my older photos (I've been pretty good over the past 18 months).
3. Setup catalog sets where appropriate.
4. Sync the metadata with the FW drive originals.
5. Backup the FW drive to the PC so that I have updated files in each place (this is where I can see a problem because I'll have to make sure that I have the "current copy of the PC folders/files on the FW before syncing the metadata from iVMP to the FW drive).

Something tells me I'm making this whole thing more complicated than need be, perhaps because I want to utilize the MBP for non-color intensive work.

Any thoughts/comments are much appreciated.

Steve, I'm not sure how much of what you outline is regarding straightening out old work, vs bringing in new work, but it sounds basically right.  A couple points:

1. I would use iView MediaPro to help me straighten out the old files.  MUCH easier than trying to do this in the Finder/Directory/Bridge. 

2. I don't sync back to DNGs wholesale, unless there is a reason (file, format or application Migration, for instance).  All the informational metadata can be safely and efficiently wrangled from within iView.  Although the risk is small, every time you touch a file you introduce the possibility of corruption.  Push information back into the files because there is a reason to do so.  Backing the information up is not a good reason.

3. The Macbook should be fine for color work, if you calibrate it.

Peter

[Steve Miller]

Peter,

Thanks for your comments. I have to take care of the day job, but will try to come back later with a couple of follow-up questions.

Steve