Questions from "The DAM Book" 2nd Edition

[peterkrogh]

Dan,
Some people want it to always be up-to-date. That's not possible for me (I do lots of organizing when I'm not connected to the collection), and I don't think it's wise.  If you update on *every* change, that's a lot of handling to the files (over a network for me) which introduces at least some possibility of corruption.

However, there are occasions when I think it's the right thing to do - data migration time, upon a change in the way you do some part of your workflow, etc.  Restoration of the archive would be one of those times for me.

Peter

[danaltick]

Peter,

Makes sense, and gives me a little bit better understanding of the overall picture.

Dan

[danaltick]

Peter,

Another question about the buckets.  I'm still using DVD's (plan to go to Blu Ray this summer), and my new 50D is 15MP, so I fill buckets pretty quick now, and of course am running into the partially filled bucket problem more frequently.  I've actually got a decent Todo list from the book that I plan to post here for others when I get the chance.  One of the Todo list items is to implement the Transfer Me folder where I build the buckets there before archiving.  Once I start doing this, I would like to know if you see any problems with back-filling partially filled buckets in that folder before archiving?  Do you see any potential problems with placing newer jobs in earlier buckets as long as it's done before archiving and burning to DVD/Blu Ray?  Thanks again.

By the way, finished the book.  Just doing some review before writing the review at Amazon.  I actually think I should read it again.  Really good stuff!  Your knowledge of the subject matter is truly inspiring!  The community should be forever greatful for your continued efforts to keep us all up to date in an ever changing and so critical part of this exciting field!

Have a Happy Memorial Day.

Dan

[peterkrogh]

Dan,
the change to Blu-ray has created some fundamental differences. 
The way I think about "ready to be archived" is one.
Although the core - archive images once you get a large enough group to make the effort worthwhile - has not changed, it does have some practical implications. The fact that stuff lingers in the "ready to transfer" area for much longer is an important one.

Ultimately, it makes the process of creating buckets more of a "gather together:" than "split apart" operation.
Peter

[danaltick]

Peter,

Hence the offsite swappers.

When you say "gather together", do you gather your jobs into the Transfer Me buckets to fill the buckets as compactly as possible without concern for putting the most recent jobs in the most recent buckets, or do you still split your jobs across the buckets to preserve the date ordering?  Of course this wouldn't be as big an issue with Blu Ray buckets, but I sometimes get some fairly large gaps in my DVD buckets if I don't split.

Dan

[peterkrogh]

Dan,
I don't worry about trying to keep things in chronological order in the buckets. I have a fair amount of low-priority personal work that I get to when time allows.

I don't send half-full buckets off to the archive - I find it much cleaner to only send stuff off once there is a full bucket - hence the "gather together" statement.
Peter

[danaltick]

Peter,

I came to that realization myself shortly after the post and thought about deleting it, but looks like I'm too late ;-).

Dan

[danaltick]

Peter,

I asked this question back on page 2 of this thread, but it got lost iin all the responses, so here it is again:

Question from Chapter 11: If you perform a data validation hash on the reworked DNG's followed by a transfer validation to the backup archive, wouldn't that eliminate the risk of backup corruption of reworked DNG's?

Dan

[peterkrogh]

Dan,
Yes, I would consider that very safe.  And one day it will be automated.  Right now it's not easy for most people to accomplish.
But you certainly have the concept right.
Peter

[danaltick]

Peter,

My archive is relative small right now (< 20,000 images, 33 buckets).  Given this, I'm fairly tempted to run the DNG converter on all the DNG's and redo the backup with a transfer validation in order to bring them all up to date with the 1.2 spec.  I even have the room on each HDD to duplicate the archive.  Would you be inclined to do this with a relatively small archive, and if so, what approach should I take to just do the DNG's?  Keep in mind my derivatives are in separate subfolders in each bucket parallel to the Raw subfolder on the same HDD.  This may be a reason for not putting them on the same disk.  Should be an interesting test.

Dan

[peterkrogh]

Dan,
I would suggest that the update of the DNG spec is a good reason to consider this "migration". (Migrating to the current spec).  The validation hash tag is valuable enough that you want it to be in all of your DNG files. If you have a lot of DNGs older than 2008, it makes sense to consider doing this. The migration should be pretty straightforward:

1. Make sure catalog of existing buckets is up-to-date.
2. Prepare target drive to receive files (run a low-level format to touch all blocks on the drive)
3. For the paranoid, I mean, thorough, check SMART data on the new target drive.
4. Run the DNG converter on the files, targeting the new drive, with "Preserve Folder Structure" checked.
5. Confirm that no errors were encounrtered in the processing.
6. catalog the new files. Make sure numbers match in folders.
7. If other file-types were in the folders and left behind, copy them manually to the new folders in the target drive.
8. Reconfirm an equal number of files exist in the source and the destination drives.
9. Do a quick visual inspection of the files, looking for any visual corruption (colored bands like shown on page 249
10. You're done. Open a Carlsberg.

Peter

[danaltick]

Wow!  That Carlsberg really provides incentive!

Question:  Because I have plenty enough room to duplicate the archive on each HDD (i.e. primary & backup), I was planning to do it that way rather than having to purchase new hard drives.  Only drawback is I can't format the drives because they have the archives on them; however, the left over derivatives will only need to be moved instead of copied.  Will this be ok?

One other thing, will I need to babysit the conversion, or will the converter log it all for me?

Dan

[peterkrogh]

Dan,
Sure, use the same drive as a target if there's room. It will be a bit slower but...
To answer the second question, no, you don't have to babysit. All errors are logged.
Peter

[danaltick]

Peter,

Do you think this would also be a good time to push the metadata back into the files before doing the conversion?

Also, I don't see any reason why I can't use the backup as the destination for the primary conversion, then perform the validated transfer back to the primary drive.  Should give me the same results, but with better performance.

I'll probably leave the old primary and backup archives on the hard drives for a week or two, until I'm confident the new archives are ok.

Dan

[peterkrogh]

Dan,
Yes, I would sync fresh metadata before the conversion.  The conversion will tel you if anything ill has befallen your images from updating (very unlikely), and these newly validated and updated files should probably also be copied to become the new backup files (particularly since they will now be more easily validated than you older backup files.)
Make sense?
Peter