Workflow with derivatives

[Mark Beckelman]

Peter -- currently, I download my RAW camera files to specific folders arranged by client name.  When it comes time to work on one of the raw files, I do a save as and place the new derivative file in a new folder that is within the main folder that houses the RAW's -- and entitle that folder WIP (for work in progress).

I've read the book, but I guess I need it spelled out for me...

So let's say I'm shooting a job for Widgets, Inc.  Where should my raw captures go?  Where do my eventual deriviatives go?

Mark Beckelman

[peterkrogh]

Mark,
Because me derivative files often get made long after the RAW files have been archived, I don't like to keep the files together.  It co-mingles new and old work, and can really complicate the backup process.

I have an entirely different directory structure  for derivatives.  It is outlined on pages 66-68.
Peter

[danaltick]

Mark,

Take a look at my bucket structure in my migration log thread.  You will see that my originals and derivatives are actually in parallel subfolders.  The masters are under the derivatives subfolder.  The thing to keep in mind here is that once your files go into the archive it's now time to think "catalog virtual sets".  Use your catalog to locate those derivatives and create a virtual set called WIP under a Utility Set if you like.  For more ideas on this reference Chapter 7.  The archive is structured for long term archival; whereas your catalog can be structured for work in progress, long time archival, or whatever you choose.  Hope that helps.

Dan

[Mark Beckelman]

Peter (and Dan) -- thanks for the clarification.  It makes sense.

This is one of those things that seem really abstract, but I'm sure once I get a handle on it, I'll look back and realize it was incredibly simple. (kind of how I felt about color management years ago)

I'm sure your workshop will further clarify things.

I'll just keep muddling along until the light switches on...

Mark

[skahn]

I would like to continue this thread a bit.  As I have gotten into the DAM workflow, I find that the original/DNG side of the process is fairly straightforward, once understood, and the Camera Raw conversion of groups of images can be quick and efficient.  Working with derivatives is more complex, especially if the job (for example) is ongoing and may require multiple edits of prints over time; i.e., some projects don't get to be finally put to bed for perhaps a couple of years.  So some  images of the job may get archived (as derivatives) and cataloged, just to be superceeded by new versions.  If I hadn't archived the first derivatives, then the current versions would replace them.  When the new files are archived, the images from the project wind up being scattered in different places.  I gather that the catalog should be used to keep track of these files, otherwise it would soon be really difficult finding specific images located in different buckets.

I think this problem arises out of the transition from folder-to-catalog thinking.  I'm still in the inbetween.  Using the catalog creatively and consistently is looming as the next big subject to tackle.  But here always seems to be new scenarios that don't exactly fit the structure.  Keeping a broad, generalized overview of the catalog and its rules seems to be the goal.

It is easy to waste tons of time keeping the logic of the catalog consistant...  I am struggling with this process.  Much easier to do this with new work that enters the system guided by the principles of the DAM workflow according to Krogh.  Legacy work seems to present far more prolems.  I'm resigned to leave most of it outside of the new digital workflow and catalog it separately.
Steve

[peterkrogh]

Steve,
A couple of things.  First, you sem to have a pretty good handle on it.  There's no perfect way yet, given the limitations of hardware at the moment.

1. If you have a workflow/setup that fully protects working files, and have the capacity (in terms of size) to keep a lot of images in your working files folders, then you might wait to put away ongoing projects that make a lot of derivative files.  Sometimes my Master Files sit in the working files folder for months before they get archived.  This can lead to a pretty large working files folder.

2. If you update a previously archived image with a clearly better version later, you might consider erasing the old file once you archive the new one.

3. I don't know about you, but I am making many fewer derivative files than I have original files.  THese are not so hard to catalog, and since each has a high value in terms of the work I have put into it, the added time cataloging it adds comparitively little overhead.

4. As time goes by, and the RAW file conversion tools get more sophisticated, we will need to create fewer and fewer derivative files, since most adjustments will be handled by the RAW converter.

5. Make sure cataloging your derivative files is a universal step when putting them away.

Peter

[danaltick]

Hi Peter & Steve,

Since my current bucket gets backed up nightly over my LAN just like my working files, my current thinking is to just go ahead and place my derivatives directly in my current bucket and catalog them right away.  If I ever needed to delete an old derivative, I would just do it from my catalog.  This keeps my working files folder from getting too big.

Also, in response to number 4 above, for the average photographer that probably holds true, but I'm not so sure about photographers like Mark Beckelman or Katrin Eismann who do a lot of masking and compositing and montages.  Mark, any comments here?

Dan

[Oliver]

I am having a similar issue with my derivative workflow/cataloguing.  With my wedding clients, I find that I am making derivative products that are increasingly widely spread over different derivative buckets.  When it comes time to burn these files to a disc for the client, I am finding there is no intuitive way outside the catalogue to find all of the files for that client except by searching through different DRV buckets.  While this is not too difficult with current clients, I can see that this might become challenging and frustrating with clients from 2 or months back. 

I guess keeping everything in the Working folder until the job is totally finished would work as has been suggested.  I just worry that 6 months later I'll end up making christmas cards or something and that derivative file will be saved in a bucket in a bucket that can't be related to the actual job in any other way than the cataloguing software.  Is that the point? Any other thoughts?

Oliver.

[danaltick]

Oliver,

Yes, that's basically the point.  The catalog gives you that virtual layer to group your images however you like without the bucket constraints.  Remember, the catalog spans your entire archive, not just the current bucket.  However, it is still a good idea to maintain some sort of structure within your buckets as Peter outlines in the book.  Just whatever structure you decide to use, try to be consistent with it from bucket to bucket.  That will make it easier to do global searches across all your buckets from the file browser if you ever find yourself without your cataloging application, or need to migrate to a new catalog.  I create subfolders in my derivative buckets for "Masters", "Deliverables", and "Contacts" because those are the only types of derivatives I catalog.  If I want to find all my Masters, I can do a global search across all the buckets for all my Masters.  Peter uses a linear folder approach in the book where he postfixes "_Masters" to his masters folders, etc.  You can do a global search on that as well.  I haven't found any drawbacks to my sub-directory approach as of yet, so I would say pick whichever way you want, but be consistent with it.  

Dan

[Oliver]

Thanks Dan,

In that way it is critical that your cataloguing software be able to see every possible file type that you may deliver to a client in the event that a year down the road a client needs a second copy of your product.  In my case that would be a slideshow created with Proshow and images tweaked in Photoshop.  My cataloguing software needs to be able to burn a disc from a virtual set otherwise I am stuck going through a bunch of different directories hunting for DRV files in to add to that disc.  It's the mindset that's killing me.  Letting go of being able to open up a directory in explorer and quickly find what I need to give to the client.  I know this can only work so long and I am determined to get the catalogue system working so thank you all for your input.

Oliver.

[danaltick]

Good cataloging software like iView can burn CD's, create web galleries, and much more from your virtual sets.  It truly is an extra degree of freedom that a file browser cannot offer...take advantage of it.

However, after discussions with Peter a while back, I do not archive my proofs.  They can be easily re-generated from the DNG embedded previews as discussed in the book.  Take advantage of that too.  It's one of the great benefits of DNG's.  I keep my proofs in a separate directory and I actually have a separate iView catalog for that directory just so I can still have all the benefits of a catalog with my proofs.  I just don't archive them, and I usually delete them when I'm done.  Hope that helps.

Dan

[peterkrogh]

Oliver,
I would suggest keeping the ongoing wedding work in the working files folder until it is ready to deliver, at least for different stages of delivery (proofs, vs album finals, for example).

As I said somewhere on this forum, some files will stay in my working files derivative folders for months, until they are ready to put into their bucket. 

As to the Christmas card issue (a single image pulled out from a shoot and reworked months later), I just make a new folder in the latest bucket for these files.

I suggest that you put a client name in the folder name, so that you can find it with the windows explorer.
Peter

[skahn]

Wow.  I don't check the forum for a couple of days and lo and behold...

I am rapidly learning how to use the catalog (iView) as home base.  Instead of searching my folder structure in Bridge, I am more and more utilizing the virtual sets to locate, evaluate and launch actions.  I recommend saving OFTEN.  I did a lot of catalog setup work a few weeks ago and iView crashed before I had saved.  ugh...  As I become more dependent upon the catalog program, it gains value and importance in the workflow and should be BACKED UP OFTEN.  Buckets all look the same from the outside and are virtually opaque without the catalog software.

Peter, I, to, have been trimming back on derivative files and multiple versions as increased storage capacity and 4MB of RAM have encouraged me to keep most of my layers intact in one file.  This makes for quite large files, but many fewer versions.  Use of a RIP such as Image Print allows me to scale the master file to different print needs.  I think the more dependent I get on the catalog software, the finer tuned and personalized the workflow will become.  Like understanding what kind of virtual sets are generic and which are job-specific and how to reduce needless redundancies.

As far as  burning backups via iView...which I have not done yet as I am still awaiting the Sonnet SATA PCI-x card and JBOD enclosures...the saved DVD archive file (Toast archive) can tell us what is on the DVD/in the backup/bucket.  These could be searched (?), catalogged (?) as an alternate way of finding archived work.  I never used to pay much attention to these files, but I am now...

Peter...you certainly started the ball rolling!!!

Thanks again.

steve

[Mark Beckelman]

Hi Peter & Steve,

Since my current bucket gets backed up nightly over my LAN just like my working files, my current thinking is to just go ahead and place my derivatives directly in my current bucket and catalog them right away.  If I ever needed to delete an old derivative, I would just do it from my catalog.  This keeps my working files folder from getting too big.

Also, in response to number 4 above, for the average photographer that probably holds true, but I'm not so sure about photographers like Mark Beckelman or Katrin Eismann who do a lot of masking and compositing and montages.  Mark, any comments here?

Dan

Dan -- sorry for the delay in responding.  I very often go back to my original raw files to reprocess, as I'm using bits and pieces from these files in composites.  The resulting composite results in a derivative (or is it a new original?) -- which brings to mind the question of how to properly name the file.  Date it? (yy/mm/dd_xxx.psd)  Suggestions welcome...

[danaltick]

Interesting.  I'm hesitant to answer this one.  Peter may have some inputs here.

Dan