Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
Print Page - Buckets & LR workflow

The DAM Forum

Software Discussions => Lightroom => Topic started by: Mike Guilbault on February 15, 2007, 06:24:21 AM



Title: Buckets & LR workflow
Post by: Mike Guilbault on February 15, 2007, 06:24:21 AM
So, for a new library of images... if one is going to use LR for their DAM software (I'm not asking to get into a LR vs iV debate - I already know the answers to that one), are buckets and the general workflow described by Peter in his book still the best method.  Since the images are not 'managed' anymore, obviously a folder structure needs to be established - and I'm assuming buckets is the way to go.  And, if you're going to use DNG as your format du jour, when will be the best time to do the conversion?


Title: Re: Buckets & LR workflow
Post by: Chris Bishop on February 15, 2007, 08:02:49 AM
My understanding, as a novice, is hte buckets are to aid backing up, nothing more. If you basck up to DVD-that limits your bucket size, Bluray? limits are larger but still limited by the backup media.
How do you plan to backup when using lightroom, and to what. that is your need for a bucket and it's size. Back uo t external HDD's and you probably only need one bucket (500GB!!) a month(!!)
Chris Bishop


Title: Re: Buckets & LR workflow
Post by: Mike Guilbault on February 15, 2007, 08:43:56 AM
I intend to have a HDD backup drive to my main image drive, but I'm also looking at DVD for archival backup.


Title: Re: Buckets & LR workflow
Post by: johnbeardy on February 15, 2007, 09:09:57 AM
If Peter had envisaged hard drives as the units of archival backup, he wouldn't have called it the "bucket" system - how about "beer barrels" or "swimming pools"?

John


Title: Re: Buckets & LR workflow
Post by: peterkrogh on February 15, 2007, 11:02:35 AM
Mike,
I would say that the answer is yes, although I have honestly not checked it out thoroughly to see if it breaks anywhere besides the other issues I have pointed out (poor multi-computer workflow, for instance.)

One could backup by mirroring a drive, but that does not offer protection from corruption being introduced to the primary archive, and being replicated to the backup.
Peter


Title: Re: Buckets & LR workflow
Post by: samdring on February 15, 2007, 01:11:39 PM
Peter

Methinks mirroring software is a tad more capable than that these days


Title: Re: Buckets & LR workflow
Post by: peterkrogh on February 15, 2007, 10:42:47 PM
Samdring (please use real name)

How would mirroring software detect file corruption?  I recently had an instance where I had 3000 overlapped files on a drive.  It took DiskWarrior 5 days to run through and find all the overlapped images.  Wouldn't mirroring software simply copy these corrupted half-files over to the new drive, overwriting the good backup files with corrupted ones?  Is there some kind of error-checking in some of this software that could tell if the source file has been corrupted?

I can see how archiving software might offer protection, but that would carry its own problems, principally in the form of a gigantic data set on the backup media (multiple copies of each file - in my case, an extra 10 terabytes or so).
Peter


Title: Re: Buckets & LR workflow
Post by: Mike Guilbault on February 16, 2007, 06:07:19 AM
My second HD is for my daily backup and DVD's for an archive.  I'm not so worried about that - it's worked for me for the past 7 years.

My question was more pointed towards fitting LR into the DAM workflow.  For example, in Peter's book, images were adjusted in ACR before being converted to DNG.  Would this be the same using LR?  Is this the point where you would archive the images to DVD (from the bucket/folders you've created).  And what about virtual copies (versions)?  How do they now fit in?


Title: Re: Buckets & LR workflow
Post by: danaltick on February 16, 2007, 07:07:40 AM
Mike,

Yes.  At least that is how I intend to do it; however, due to the lack of full DNG support (along with other B4 deficiencies), I'm holding off for V1 before using LR in my workflow.  Once your current DNG archive bucket is full then yes, you would burn a DVD.  I plan to maintain my current working files workflow with LR V1 as presented on Peter's DVD tutorials.  Unfortunately virtual copies are a LR internal feature.  Hopefully one day that will become a part of a standardized schema that gets embedded in the sidecars or the DNG itself.  Until then, third party asset managers such as iView will not be able to utilize it.  Peter, if I'm overlooking something here, let me know.

Peter,

With regard to mirroring (which I no longer do except for the working files structure), do you test the integrity of your buckets after they fill and before backing them up to your secondary archive?  If not, wouldn't you be susceptible to backup corruption the same way mirroring is?  In otherwords, aren't manual copying and mirroring both susceptible to backup corruption?  I think maybe the assumption is being made that mirroring is done unattended on a nightly basis.  However, mirroring could be done as just a replacement for manual copying, but there would be no real benefit to that.  I used to mirror my current bucket on a nightly basis, but realized the best solution was to only backup the current bucket when it was full and instead make working files backups a part of my workflow (like you do).

Dan


Title: Re: Buckets & LR workflow
Post by: peterkrogh on February 16, 2007, 09:10:47 AM
Mike,
While I'm sure you can come up with some workaround that will enable this with V1, it will not be as streamlined as I hoped.

The DNGs will not be built with a full-sized preview, unless you run them through the DNG converter separately.  This means that buckets you make with Lightroom will not be the right size since the DNG size will change when they get a preview.

Virtual copies are, at the moment, alternate settings that are only known to the Lightroom database.  You can't push that information back into the file, so you have to backup the database if you want to save it.  (Also remember that without Photobinders, you can't add databases/libraries together.  So you can't create collections/virtual copies on your laptop and then bring that information into your main library on a different machine.)

Personally, I think the workaround pain involved in creating a secure, reliable system with Lighrom V1 to manage the files will outweigh the benefit for the multi-computer-using pro shooter.  And many of the hoops you will have to jump through in the next several months will not be necessary once later versions come out.

Peter


Title: Re: Buckets & LR workflow
Post by: Mel_Lammers on February 16, 2007, 09:47:02 AM

Mike,
While I'm sure you can come up with some workaround that will enable this with V1, it will not be as streamlined as I hoped.

The DNGs will not be built with a full-sized preview, unless you run them through the DNG converter separately.  This means that buckets you make with Lightroom will not be the right size since the DNG size will change when they get a preview.

Virtual copies are, at the moment, alternate settings that are only known to the Lightroom database.  You can't push that information back into the file, so you have to backup the database if you want to save it.  (Also remember that without Photobinders, you can't add databases/libraries together.  So you can't create collections/virtual copies on your laptop and then bring that information into your main library on a different machine.)

Personally, I think the workaround pain involved in creating a secure, reliable system with Lighrom V1 to manage the files will outweigh the benefit for the multi-computer-using pro shooter.  And many of the hoops you will have to jump through in the next several months will not be necessary once later versions come out.

Peter
Peter,

I apologize if I missed the point above, but in the Outback Photo review, they said they had 30K photos in the LR V1 database and the ran the Backup Library option at startup [enabled in Preferences in V1 I guess since I can't find it in Beta 4.1]. They stated it took 5 minutes but felt it was worth it.

That implies a strategy of "turn LR on at the beginning of day [or when you turn on your computer if you don't keep in on 24/7] and leave it on. I suspect that strategy will drive other tactics for full time automatic continuous backup & etc.

We all are the IT shop for our own data. It is beginning to look like full time continuous backup to both internal and external HDDs is mandatory. This is becoming mandatory for one person/one computer photographers as well as professionals like you [with separate servers etc].

It may be cost effective to move to the upcoming MS Home Server S/W and a dedicated server + some off site capability [in case of hurricane/tornado/fire etc].

Mel


Title: Re: Buckets & LR workflow
Post by: samdring on February 17, 2007, 12:44:38 PM
Samdring (please use real name)

How would mirroring software detect file corruption?  I recently had an instance where I had 3000 overlapped files on a drive.  It took DiskWarrior 5 days to run through and find all the overlapped images.  Wouldn't mirroring software simply copy these corrupted half-files over to the new drive, overwriting the good backup files with corrupted ones?  Is there some kind of error-checking in some of this software that could tell if the source file has been corrupted?

I can see how archiving software might offer protection, but that would carry its own problems, principally in the form of a gigantic data set on the backup media (multiple copies of each file - in my case, an extra 10 terabytes or so).
Peter


Peter

Forename = Sam
Surname = Dring
I use MirrorFolder which has a significant number of options including incremental, archive, not mirror deletes etc - methinks Dan's point also very valid
It is a solid method - my preference is to have one drive real time and another every 12 hours - it works for ME and that is the real issue given that our needs are so very different.


Title: Re: Buckets & LR workflow
Post by: peterkrogh on February 17, 2007, 05:07:10 PM
Sam,
Sorry for the confusion about your name. 
As long as you understand the risks, no problem.  Do you have a system in place that would protect you in the case of a disk error on the primary?
Peter


Title: Re: Buckets & LR workflow
Post by: samdring on February 18, 2007, 04:31:56 AM
Peter
No problems on the name.  My name without a space returns  Samdring Dorje Phagmo, the famous incarnation of the deity Vajra Yogini !!
I suppose I should have given more info.  I am in the throes of changing parts of my system but, in essence:

2 small sata HDD on Raid 0 for system (Ghosted)
2 large sata HDD on Raid 0 for Images & Data
1 sata enclosure taking hot-swappable HDD
1 firewire external

I write metadata to raw on downloading putting everything I need in at that stage. Yes I know this is dangerous for Nikon and some others but not a problem with any Canon [5D, 20D and previously 10D] in my experience.  I realtime mirror the raws ONLY selecting an option to backup files before overwriting during synch.  This backup option is rarely used 'cos I don't change the raws very much but it would have been used in the scenario you portrayed.  I separately mirror the 'databases' of the many differing converters I have used/do still use which give me the settings and a third mirror does my conversions (tiff, jpeg etc).  This third mirror does NOT have the backup option selected - there would be too many changes.

These realtime mirrors are then repeated daily in the same format to a different drive and, given your post, am seriously considering increasing that to weekly.

This sounds cumbersome but can be set up initially in about 10 minutes or so and runs smoothly without intervention.  I have three copies at any one time and the swappables will allow off site single copy.  I think my raws are well protected which, if on major failure, could be used to re-convert if necessary.

If anyone spots chinks in my armour, I would welcome comment please


Title: Re: Buckets & LR workflow
Post by: peterkrogh on February 19, 2007, 07:04:06 PM
Sam,
So, this :

>> I realtime mirror the raws ONLY selecting an option to backup files before overwriting during synch.

indicates that any file that has been altered will get archived to an additional backup before the mirror backup is overwritten?

If so, seems reasonably well protected.  Do you end up creating a large data set of the changed Raw files?  Do you depend on the backup software database to sort this out in the event of drive failure?
Peter