Problems with NAS, looking at FW400 or USB2 alternatives

[Steve Miller]

Haven't posted in a while, but I've been reading threads here and there...

I currently use a LaCie NAS for backup of my photos and music. The NAS also serves as a source for all music that goes through my Sonos system (for anyone looking for a FANTASTIC wired/wireless digital music system for your home, I can't recommend this highly enough). The NAS works perfectly for the music system, I'm guessing that speed isn't an issue when you're just reading a music file off of a network. On the other hand, backing up photos to the NAS is so slow, that it has become a deterrent to timely backups. My wired network is 100 Mbit, which should in theory allow for throughput speeds of 12 MB/sec. Well, in the real world, it takes at least three hours to backup 30 gigs of photos (part of a 110 GB collection), or less than 3 MB/sec. Needless to say, this ain't going to last.

I'm debating about upgrading my router to 1000 Mbit, though given the poor actual to theoretical performance I'm currently experiencing, I'm wondering if there's a problem with some of the internal wiring (which, given that it's behind walls, I can't fix). Any thoughts? Frustration and doubts are leading me to switch over to a straight forward external HD for photo backup (I would still use the NAS for music so I don't have to leave my desktop on 24/7). My desktop PC and my MacBook Pro have FW400 and USB2.0. I've seen conflicting data as to which is faster in practice. Peter or others, any advice on FW400 vs. USB 2.0? I assume I shouldn't even entertain eSATA since the MBP doesn't have it?

I was leaning toward a simple single drive enclosure, but am now thinking that maybe I should build my own two-drive bay with 2x320 GB Seagate drives for better redundancy. If I do this, can I set them up as RAID 1, or do I have to backup one drive to the other? Also, one thing I love about Macs vs. my PC, is how quiet they are. Will these drives be noticeably loud?

Any help is appreciated.

Steve

[peterkrogh]

Steve,
With current drive sizes up to 750 GB, I would not be thinking of RAID for a collection your size.  I'd get FW or USB2.  If you are on MAc, go for FW (FW 800 if your machine supports it).  USB2 is fine for either, really, but there's slightly more functionality with FW, since it can be bootable.

Either should be fine as a general file backup.
Peter

[David Anderson]

Steve,
I briefly considered getting a NAS system but eventually decided that simple external Firewire hard disks were likely to offer the most cost effective and practical image backup solution. I alternate my backups between a couple of LaCie 300GB Firewire 800 disks (the D2 Hard Drive Extreme) and recommend them very highly. They are robust and very portable. One key advantage is that they can easily be transported offsite for security. I always keep one of mine stored in my garage, which is not connected to my house. If the house were to burn down, the garage would not be affected. I think you would be much more likely to keep your NAS system, or any multidrive enclosure, located permanently in your house - with no protection against fire or theft. For me, that would make it an unacceptable backup system.

Usiing Dantz Retrospect Professional software on my old PC, which had a Firewire 800 card, my backups ran very quickly. At present, my new Dell Precision 670 Workstation only has the built-in Firewire 400 interface but the backup speed seems quite similar (perhaps because the PC is much faster). The Dantz software verifies file integrity after copying and only transfers new files on each successive backup. The biggest backup session recorded in my Retrospect log file took 2.5 hours. That was for 250GB (with a large part of that being copied for the first time after a major hard disk rearrangement). My normal backup sessions, that just add all the new files in my C: drive and my images-dedicated H: drive, take an average of only 15 mins. Incidentally, Retrospect uses what they call a Progressive backup system that lets you restore any of the 10 or more previous versions of a file. It only wipes out the older versions as you reach a specified storage limit.

Being well aware that any hard disk can fail, I also have DVD backups of all my images - using Peter's 'bucket' system. 

David  

[Steve Miller]

Peter and David,

Thanks for your replies. I ended up talking to Brian at Macgurus.com yesterday (Peter, I saw them highly recommended from one of your other posts). He couldn't have been nicer and more helpful. It seems like they do very little with FW or USB 2.0, focusing almost exclusively on SATA solutions. Knowing that I had a MacBook Pro, for which there aren't yet any SATA cards, he suggested just going with an off the shelf FW400 external drive, then down the line, when the SATA cards are available for my MBP, looking at a dual SATA drive enclosure. He also highly recommended that I don't use my LaCie NAS drive on a 24/7 basis, because they have no fans and will eventually overheat and fry the HD. All of a sudden, I felt like I shouldn't wait too long to replace my current system.

I ordered a 250 GB LaCie drive from newegg and plan on using it as my main back-up for all photos and music (also have DVD copies of all originals offsite - note to self: make DVDs of Derivatives as well). I'll keep the NAS as a music server, figuring that if/when it blows up, I'll find an alternative way to feed the Sonos. David, thanks for your comments about Dantz, now EMC, Retrospect. It looks like it's a Windows only product. Do you happen to know of any good backup software that has the same functionality and ease of use you described, but works on both Mac and PC platforms?

Thanks again,

Steve

[David Anderson]

Steve,
I only have experience living in the sunny uplands of Gatesville and have never been tempted to bite the Apple and cross over into the Dark Side . Sadly, therefore, I can offer no advice on backup software for the Mac. If both your PC and your Mac kit is all connected via your LAN, perhaps you could consider upgrading parts of it to Gigabit and letting the PC handle the backup of the Mac via the network? Gigabit adapters, hubs and switches are getting cheaper all the time. My Fast Ethernet switch has two Gigabit ports while both my desktop PCs and my laptop PC have Gigabit adapters (they come as standard in most new kit). This means I can always get two PCs talking at Gigabit speeds. I just rearrange patch cables on my patch panel if I want to select a different pair out of my three PCs.

Retrospect offers network client software for Windows, UNIX and Mac machines.

[David Anderson]

Steve,
Further to my last post, I have just run a test of backing up a My Documents folder on my laptop PC over the network using EMC Dantz Retrospect Professional V7 (the product's official full title!). This first required the installation of a Retrospect Client on my laptop. The results were as follows.

15394 files with a total size of 8.33GB were backed up in 19 mins 33 secs. The quoted transfer rate for the job was 933.3 MB/min. I don't know for certain how this rate is calculated, because 8.33GB divided by 19 mins 33 secs gives only 426 MB/min. My best guess is that the data is indeed initially transferred at the quoted rate, but the total elapsed time is then extended by subsequent data verification so that the effective transfer rate is much less.

This quoted rate of 933.3 MB/min is understandably a bit less than I get for backups of data on the desktop PC directly attached to the Firewire drive. A roughly comparable local backup of the My Documents folder (with lots of small files like on my laptop) had a quoted rate of 1145 MB/min. Backing up local digital image folders with a smaller number of larger files (relative to the My Documents folder) can get quoted transfer rates of 1800-2800 MB/min. Again, the overall effective rate after verification is less than half the quoted rates.

I will run a network backup test using an image folder as soon as I get a chance. It would be interesting to find out if the transfer rate over the network improves for such files.

David

[David Anderson]

Steve,
Yet another update on my network backup tests - which I'm doing for my own benefit as well.

As well as 'Progressive' multi-version backups, Retrospect also has a straightforward 'Duplicate' option. As before, repeated Duplicate sessions for the same folders only copy the latest files so it's a slick process. I have just run a new Duplicate operation for the My Pictures folder on my laptop. The result was 35.3GB of image files copied and verified over the network in a total of 1 hour 13 mins. That broke down into 33 mins at a transfer rate of 1093.6 MB/min for the copy process and 40 mins at a transfer rate of 904.5 MB/min for the comparison process.

This particular My Pictures folder contains 12076 files, with a lot of JPEGS but also a fair number of larger Raw files.  

Hope you find this useful. It certainly beats your current 3 hours for 30GB.

David

[Steve Miller]

David,

Thanks for the testing data.

I spoke to EMC yesterday and found out that they also have a Mac version of their software. Essentially, I can either buy the Windows version and manage Mac-to-external hard drive backups via the Client software on my MBP through my network, or vice versa if I by the Mac version. Since I am worried that their is a data transfer problem on my network (thus the slow speeds I'm realizing), I'm hesitant to do any backups via my network. The tech assured me that while I would be setting up the backup via my network, the actual copying/backing up of data from MAC or PC to the locally connected external drive would obviously be local. Probably very straightforward for most people, but I'm new to this stuff.

To your data transfer speeds mentioned above. Are those speeds for a setup similar to what I described above? i.e. are you backing up to a local or network HD? I'm guessing a network drive given the slow speeds (your average speed of 16 MB/sec is much faster than what I achieve, but slow compared to local FW speeds - supposedly closer to 40 MB/s)?

All of this theoretical speed talk will be put to rest this weekend after I get my external drive and download the Retrospect demo.

Thanks again,

Steve

[David Anderson]

Steve,
My backup tests were done as follows. The data to be backed up was on my laptop PC, which was running the Retrospect Client software. The main copy of Retrospect was on my desktop PC. The LaCie 300GB FW external drive was connected directly to my desktop PC via a FW400 interface (the drive is compatible with FW800, so could run faster).

For comparison purposes, I have just run the Duplicate process again, using the same data, but this time with the files residing on my desktop PC. This local backup took 46 mins overall - 22 mins at 1623.0 MB/min for the copy, 24 mins at 1488.2 MB/min for the compare.

David

[David Anderson]

Steve,
Just to complete the story, I have now moved my LaCie Firewire 800 PCI card from my old desktop to my new desktop and re-run the local duplication of the 35.3 GB My Pictures folder. This took a total of 39 mins - 19 mins at a transfer rate of 1814.5 MB/min for the copy and 20 mins at 1907.1 MB/min for the compare.

You will note that these FW800 times are better than when using the FW400 interface but the improvement is nothing like the factor of two that is implied by the names of these interfaces!

David

[peterkrogh]

I would offer a note of caution regarding the use of backup software to update all changed files.  I personally like the idea that my backed-up files do not get updated, so that any disk, file, or program errors that get introduced to the primary files do not get propagated to the backup files.

Since I can wrangle all the informational metadata with iView, I don't feel the need to update the backup version of the image files themselves.  You might be inadvertently corrupting the backup in this process, if something has happened to the primary.

Although this arrangement keeps me from saving ACR settings that have been changed since the original backup was made, I consider that an acceptable risk.  In most cases, ACR settings get updated because a Master File is being made.  In these cases, the new Master (annd its derivatives) will become the version of the file I use for most purposes.  I expect this to change at some point in the future, when DAM software can harvest and use Image Adjustment Settings as the metadata that they are.

Peter

[David Anderson]

Peter,
I think you are underestimating the capability of good backup software combined with good backup procedure. The discussion here has been about Retrospect. If you use its 'Backup' option, then every changed version of a file is added to the backup rather than overwriting the older version. Retrospect offers different user-selectable options for 'grooming' the backup set when it reaches a preset maximum size limit. One of those grooming options is to specify how many versions of a file will be retained (I think the default for this setting is 10). You can restore any one of these historical versions of a given file.

Speaking as a recently-retired IT Manager, I would make the point that no serious backup process should ever just retain the latest version of a file. In my corporate life I set up so-called Grandfather-Father-Son tape rotation schemes that ensured you could restore data from any day in the past week, any week in the past month and any month in the past year. Retrospect's 'Progressive' backup scheme provides a similar, though obviously not identical, level of protection. I have DVD-R backups in addition to my Retrospect-controlled external hard disk backups, though I don't attempt to update these DVDs just because ACR settings have changed.

In other words, there is nothing to stop you backing up everything you do - and backup to external firewire hard disks is fast enough not to be inconvenient (which means there is more chance that it gets done on a regular basis).  

David  

[wombat2010]

Thank you for the reply, David.  The problem I run into (I use Retrospect, but I am playing with SyncbackSE, too) is that backup sets quickly get out of control size-wise when multiple backup sets are kept.  I am trying to decide what is a reasonable balance between safe, reliable backup sets and the resources it takes to get there.  I am not running a photography business, but I would be crushed if I lost all or a chunk of the pictures of my family.  Losing my "creative" shots, on the other hand, may not be such a great loss.    (One thing I like about SyncbackSE, that Retrospect doesn't have, is that its archives are in the orignal file format--the advantage is that I can look at pictures on my backup server or take a backup drive with me to someone else's computer and pull up pictures.)

I would appreciate hearing what you or others feel is the right balance for their own work, understanding that everyone has different needs and resources.  Thanks.

Stephen

[peterkrogh]

David,
I have about 1.5 terabytes of RAW/DNG files.  Each of these goes through several states before it is archived and backed up.  This would make the backup drive array pretty large.  I'm assuming that I would need RAID instead of multiple volumes for this. (If I configured such a system last year, it would probably be a 16 drive box, or some large tape backup system.

This is a lot of hardware expense and administrative overhead to backup a few megabytes of changing metadata.

The simpler system, in my view, is to segregate the changing metadata from the large data set of unchanging image data.

Am I missing something?
Peter

[David Anderson]

The problem I run into (I use Retrospect, but I am playing with SyncbackSE, too) is that backup sets quickly get out of control size-wise when multiple backup sets are kept.  I am trying to decide what is a reasonable balance between safe, reliable backup sets and the resources it takes to get there.

Stephen (wombat2010),
At a simplistic level, if you have 100GB of images then you would need a 500GB external disk to hold 5 historical versions of every file. You would then set Retrospect's grooming option to keep no more than 5 versions of any file and old versions would be automatically erased from the backup set as new ones are added. But this is a worst case scenario. You would not be changing every image on a regular basis. Many would never change. The changes made to Raw files are stored in small sidecars, so the Raw files never change. Updates to ACR settings and metadata for DNG files, however, would involve a DNG file update every time. Have you had a problem with grooming not working as expected?

I know nothing about SyncbackSE, but Retrospect also has its 'Duplicate' facility that lets you store your files in a normal non-proprietary format.

I am fairly paranoid about losing my images, but I feel quite comfortable with my current setup of a pair of inexpensive 300GB external Firewire hard disks (one held off site) and two sets of DVD-Rs that hold all my original images and at least two versions of my derivatives (all held off site).

David