The DAM Forum
Welcome, Guest. Please login or register.
August 27, 2014, 08:04:55 PM

Login with username, password and session length
Search:     Advanced search
28016 Posts in 5137 Topics by 2910 Members
Latest Member: kbroch
* Home Help Search Login Register
+  The DAM Forum
|-+  DAM Stuff
| |-+  Backup Strategies and Tools
| | |-+  Help! How to handle backups reflecting changes to previously archived content?
« previous next »
Pages: [1] Print
Author Topic: Help! How to handle backups reflecting changes to previously archived content?  (Read 3768 times)
Big-T
Newbie
*
Posts: 16


View Profile
« on: April 07, 2008, 03:01:40 PM »

 Undecided Believe it or not I spent a few hours reading/looking for a similar post on this concern/question (both on and off forum):

I am ready for my second copy of peter's DAM book I have used and referenced my one copy so much.

Question: in trying to follow the DAM guidelines refernced, what do you do with the situation where you go back to previously-taken and backed up images and re-edit them or somehow change something with the file (image correction or any kind of change, meta-tag update, etc)? Having to redo a whole cascade of (DVD) media volumes for a change to one (or a few) file is obviously not desirable. I have read posts to this topic that come close to this specific point, but not fully discussed (at least couldn't find).

My backup technique incorporates taking entire image directories on my main working PC and copying those entire image directories (all image file versions - orginal and derivative) onto 2 other drives so making a change to a file in a folder is easy - I just transfer the changed file(s) to the other backup drives having identical contents as my 'master' PC. But, what to do about archived disc volumes that need to reflect most recent change(s) to the file(s)? How does this jive with Peter's backup workflow example?

My environment is one in which I have about 5 years of digital photos online and I may make a change to any file in the past at any time - even going back to when my first child was born. I understand that Peter suggestions archiving files when you think the image files have gone all the way they are likely to go but that doesn't quite fit for my environment as it is all "open" to change at any one time.

I am probably missing some key and simple point. Forgive me Peter (and others) if I am having an "old-timers" moment. I really want this backup workflow to work! Smiley

Thanks in advance for any enlightenment/suggestions/help!

T.
Logged
markpirozzi
Full Member
***
Posts: 179


View Profile
« Reply #1 on: April 07, 2008, 04:02:07 PM »

The recommended plan is to store derivatives separate from originals.  If you redo a derivative, the new version goes in the current derivative bucket.  Using unique names for the originals and appending these names for derivatives should make it easy to find the most recent derivative after you've forgotten you did that second derivative. 

If you store derivatives with the originals the first time around, there is no reason you can't store the second derivative in your current bucket - the key is searching by filename at a later date.  If you add new derivatives in old buckets, their size will go over there limit at some point.

Another case would be redoing old RAW files after archiving.  One option after burning a DVD, would be to copy the RAW into a current bucket and reinterpret the processing. Or just redo the old version in it's original bucket on a HD and hope you never need to restore from the DVD.  Reburning is a individual decision based on how much change there has been to a bucket and the value of your time reworking the files.  It may not be a bad idea to reburn the DVDs after a number of years anyway.

Mark
Logged

Mac OSX 10.7, 2009 MacPro
Big-T
Newbie
*
Posts: 16


View Profile
« Reply #2 on: April 07, 2008, 06:49:40 PM »

Hi Mark, thanks for your reply.

It may be that for me, I continue to do the transfering of the entire image directories (old and new, original and derivatives) onto 2 other hard drives as main backups; then, occasionally - like after a lot of work is done or I introduce a lot of new meta-tags like through iView, - I will burn the dozens of DVDs.

I very much like Peter's methodical volumnizing of his sets of DVDs for his images but this may be something I do less often, again, depending upon the amoun of change my directory of fiels have experienced. I am new to iView and getting started with it now. Perhaps after I have tagged and re-named the thousands of images, I would have that be a milestone point of being sure to backup to DVDs in addition to the copyoing of all the images to 2 other hard drives.

It just seems to me to be a point of inefficiency that when you alter an image previously backed up and just place the new version of the file(s) into the current derivatives folder, you will eventually be backing up old content with new if you no longer need the master file of the image you changed. How would this work if you had to restore all your files from DVD? Would your new(er)/most recent versions of a file end up just be placed into the same directory where its prior versions reside? That is, let's say that you are in the process of restoring your collection from DVDs and on disc 3 you have a file called abc123.psd   . . . . . .  if your latest rendition/version of that file is abc123-BW.psd say from DVD disc 19, would that file on disc 19 just end up falling into the respective containing folder when you are done copying over all your files for the restore? Make any sense? So, when you are done, you have all the files/versions in a folder together now? So on your next snapshot of discs, they will all be there going forward?

Thanks in advance for shedding some light on these questions.

I will be interested to see how others have adopted the backing up previously archived images issue, too.

T.
Logged
billseymour
Sr. Member
****
Posts: 308


View Profile
« Reply #3 on: April 07, 2008, 10:20:36 PM »

T-

You might feel more comfortable about the original/derivative question by thinking about exactly what is being accomplished with the use of (1) buckets; (2) original image filenames that arrange themselves chronologically; and (3) organizing filenames around 'core' portions of the image name. So:

(1) Buckets: You have 'originals' buckets, and 'derivatives' buckets. You fill the original buckets chronologically, and when filled, you move on to the next bucket. The 'derivatives' buckets are filled more loosely, on a 'project completion' basis.

By separating the derivatives into their own bucket, you avoid having to 'redo' your Originals buckets. When you fill them, that's it. Further work done on a image will go into the derivatives bucket.

(2) filenames arranging themselves chronologically- makes filling buckets and seeing where you are much easier.

(3) 'core' parts of filenames. This exploits the idea that in DAM programs like iView/EM/iDimager/etc, the chronological sorting of filenames will cause Originals and Derivatives (which start with the same 'core' name) to arrange next to one another. So:

wcs_080407_001_Original.jpg
wcs_080407_001_Ps001_Master.psd

- will list sequentially. The importance is: the catalog's sequencing puts these two images next to each other, NOT which bucket the images are in. My original might be in Originals_Bucket_01, while my derivative might be in Derivatives_Bucket_05. Doesn't matter. They wind up listed sequentially because the'core' part of the filename is the same, and the catalog lists alphabetically (and chronologically, if you use yyyy-mm-dd date form).

I will leave to someone else to discuss keyword updating, since I still itch to re-keyword my originals. (Which I may still do at some point). Folks here seem to follow the idea that you update keywords in the catalog (rather than doing followup syncing of the original images), and you backup the catalogs so you protect your metadata efforts.

On the Q of backing up to DVD- I don't fully understand why this would be an issue. Main hard drives fail. So do backup HD's. So do the 'second backups'. Why not backup to DVDs as you go (ie, as you fill buckets), and put them somewhere far away from your computer. Wouldn't you be happy you did this if disaster struck all your technology? You should also DVD backup your catalogs from time to time, to preserve the updated keywording (which is in the catalog).

Hope this is of some help.
--Bill

Logged
Sossity Corby
Full Member
***
Posts: 247


View Profile
« Reply #4 on: April 08, 2008, 12:48:08 AM »

So any changes I make to my meta data of my camera originals after the 1st initial typed meta data I entered before/at/upon ingestion to my originals drive makes those originals  become  derivatives, even if I don't make any adjustments to the images themselves? ie; a camera original; 080101_1234_CO.jpg, is downloaded to my originals drive to bucket 001; before I downloaded it, I entered some basic meta data; city,state,country,some keywords,category,my name & copyright, in Down loader Pro. So if I go back & open up this image again in Expression media, to add/update/correct the meta data, ie more keywords,change in contact info, or just forgot to add something in Down loader Pro, it is now a derivative? & goes to my derivatives drive? even if I don't do any actual editing to 080101_1234_CO.jpg itself? how would I rename an image like this to reflect the meta data ( not photo editng change ) change for derivative status? So does it mean any change made to an original image at all; meta data or editing of the photo makes it a derivative? At exactly what point does an original become a derivative? I thought only when one edited an original  image in a photo editor

Sossity
« Last Edit: April 08, 2008, 12:50:18 AM by Sossity Corby » Logged
Chris Bishop
Sr. Member
****
Posts: 472


View Profile
« Reply #5 on: April 08, 2008, 03:15:23 AM »

I look upon any work done prior to conversion to DNG as a temporary situation. it may be in this state for an hour or several weeks.
Ingest, with basic metadata etc
Run through Quick Developer (LR) or Rapid fixer (Bridge) to get as many files as possible to the highest quality.
Add any further metadata
Convert to DNG
Archive in the next available bucket.
Earlier backups etc created on ingestion can now be deleted (if you need the space)
That DNG is not altered again
It is often copied into a derivative workflow where Masters are created.
Does this help?
Chris Bishop
Logged
billseymour
Sr. Member
****
Posts: 308


View Profile
« Reply #6 on: April 08, 2008, 05:31:11 AM »

Sossity-
Hopefully someone will confirm this, but my understanding (and what I do) is that a Derivative is where you have modified/altered the image itself. So, I take my Jpeg Original into Photoshop, fiddle with the contrast and colors, and generate a finished image- that is a Derivative, which I will give a derivative filename when I catalog (ie, wcs_080401_001.jpg original; wcs_080401_001_Ps_Master.psd derivative).

If you are only adding/changing metadata in an image you've already cataloged, the image itself is still 'the same image'. (ie, if you add metadata to an Original, it is still the Original; if you add metadata to a Derivative, it is still the Derivative).

The question on adding metadata (later in the process) is:
a) whether to only update the catalog (so you 'save' the metadata changes in the catalog), or
b) do you also sync-annotate to embed the changed metadata to the Original image files.

I like the idea that, if I sync-annotate, my images have the updated metadata (rather than depending on the catalog to store these changes). I believe that the DAM practice is more 'don't touch the Originals again once you've done your processing and cataloging'- under this model, the later metadata would be saved in the catalog (and the Originals would not be touched).

Does that help? As an example for me: I have gone through a number of cataloged images, to mark them as 'Photoshop-possible projects' (images that I now see I can 'push' into interesting results). I am using a previously unused Label for this.
- I apply my new label in iView, to my Original image (it is still my Original).
- because this is only a temporary Label (I might change my mind), I am not syncing back to the Original image file.

Does this help clarify?
--Bill
Logged
Big-T
Newbie
*
Posts: 16


View Profile
« Reply #7 on: April 08, 2008, 08:16:51 AM »

Thanks everyone for such an active and helpful community of fellow shooters!

I kind of went to bed last night somewhat determined to continue to just do things as I have been - backing up my main 'live' collection of all 5 years of photos on my main PC hard drive over to 2 other drives and making occasional disc backups of the entire collection. Again, for me, I may make a change to any file in my entire collection at any one time, perhaps going back years to an earlier photo. But now, I am back to really rethinking separating the orginals and derivatives, giving them their own directory chain and sequencing of folders and then backing them up to disc, separately (originals and derivative buckets); this versus backing up to disc with originals and derivatives together.

I understand that if I keep the originals and derivatives together that when I do a backup to DVDs, I will largely just be copying back original data and bringing over a very small percentatage of new content from the derivatives. However, I must admit that by having the derivatives in a subfolder of my originals on my hard drive, it is very nice to be able to see all the derivatives of the original(s) all at once in the same folder. I just click on the respective original folder (now titled with just a date of download) and see the originals and then open up a subfolder to view the derivatives of that parent directory.

This may be one "downside" to keeping separate original and derivative buckets -  . . . . .Let's say that I want to see all the derivatives of an original image at one time. Under Peter's method - if I am understanding correctly - if the derivatives of a single original are scattered over 4 different backup DVD discs, I would have to hunt each one down from each of the discs and then bring them together in a single directory. . . .?  . . .OR - this may be important - OR would it be that if I am using iView to catalog the images, this would not be necessary to do if I just want to VIEW them as the catalog program will display them all together because the (as Mark says) they will have a similar "core" filename with just perhaps the very end of the filenames being different (abc123.cr2 . . . . .abc123-BW.psd . . . abc123-BW2.psd. . . .)

If the above is true and the catalog program will relieve me of this task of having to grab images over multiple derivative discs since they will appear in the catalog list by filename, what do  you do if you want to not just VIEW all the derivatives, but actually GET all of them to LOAD in Photoshop? Then I guess I would need to locate each one of the derivative discs within the long span of derivative discs (say disc 02, 09, 35) and then load them into Photoshop one-at-a-time?

Thanks for hearing me out and setting me straight!

Ugh. .  . I am losing more hair on my head over this.

Thanks again for all the replies, friends!

T.
Logged
ScottBuckel
Full Member
***
Posts: 245


View Profile Email
« Reply #8 on: April 08, 2008, 12:42:22 PM »

T -

My personal interpretation of Peter's system, and the one I do is I keep all files on hard drives - orignal dngs and derivative files in separate directories.  I archive to DVD only as a back up to the hard drive stored images.  I keep two copies of all files on different hard drives. 

The idea is to keep you stroage and archive as simple as you can that works for you.

I hope this helps,

Scott
Logged

Scott Buckel
Monrovia, CA
Big-T
Newbie
*
Posts: 16


View Profile
« Reply #9 on: April 08, 2008, 02:03:42 PM »

Scott -

so you also do something quite similar to my approach, but you distinctly separate the originals and the derivatives in different directories on your hard drives.

I may find that by doing this it will be some happy "medium" between desiring to have all files onlline at any time but backing up the originals and derivatives separately, on hard disk directories and DVDs vs. Peter's approach to backup. My case may just be different enough than what many others need or are comfortable with.

Questions:
1) do you follow Peter's approach in labeling your originals and derivatives by DVD archive "bucket" also - having something like "DVD_Original_01. . . .DVD_Original_10" and "DVD_Deriv_01. .  .DVD_Deriv_10"  . . .? 

as well as

2)keeping an "open" derivatives bucket/folder available to collect newly captured edited files or edited older previously-archived originals until reaching a max llimit (like 4/8 GB)?

Thanks for sharing.

T.
Logged
markpirozzi
Full Member
***
Posts: 179


View Profile
« Reply #10 on: April 08, 2008, 05:06:54 PM »

Sossity,

I hope my statement "Another case would be redoing old RAW files after archiving.  One option after burning a DVD, would be to copy the RAW into a current bucket and reinterpret the processing." didn't give you the idea to bring old images to current buckets on a wholesale basis.  I was thinking of reinterpreting a few RAWs now and then.

Everyone,

The goals of the system are to able to be able to find your images and to be able to restore your file system after a loss.  You can modify what doesn't suit you, your time, or your resources.  About 4 years ago when I switched my fledgling digital photo collection to another computer, I saw how hard it would be to restore a filing system of a very large collection not filed chronologically. Plus, you can't see at a glance what folders are missing.  I read Peters book a couple of years later and told myself I need to give up on a subject based system soon.  All you have to remember is that 4 follows 3, etc.  I also resisted placing letter prefixes on file names but I have changed that because I have decided to back up friend's snapshots with my pictures. I prefix their shots their initials.  I place a lower case v in font of their initials and this keeps all the snapshots together.

Big-T

1) Basically yes, except that I use CAP-001, CAP-002... (for captures) and DRV-001, DRV-002... (for derivatives) and label my DVDs as such and keep all the DRV DVDs together and the all CAP DVDs together.  I write a date (v061202) of the burn on the disc.  If I do burn a new one, I write 'Replaced by v080408' on the one being replaced and keep that as another backup copy.

2) Yes.  You can let that bucket get big if you want and then split it up to smaller buckets when you have time.


Peter doesn't push subsequent metadata into the files often, but as I remember he will do that every once in a while when has to do a restore or some other big maintenance chore. Every time you update metadata to a file you increase the risk of corrupting a file. The chance per file is small, but with 50,000 files, the chance of some being affected is not infinitesimal.


Mark

Logged

Mac OSX 10.7, 2009 MacPro
Pages: [1] Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!