Migration strategy questions for Peter, Dan, or any takers

[Steve Miller]

Peter, Dan, and any other helpful DAMers,

Apologies in advance for what will probably a fairly long post with many questions.

I'm at the paralysis phase. I've read The DAM Book twice along with many very helpful posts (including Dan's migration thread just behind this one), and have diagrammed a tree structure for Working Files and Archives. Now as I sit here trying to figure out the best way to proceed, I'm getting a brain freeze. For reference, I'm dealing with an archive of ca. 9,000 original and 6,000 derivative files dating back to 2000 (RAW since 9/03). All of these are personal, though I see doing paid jobs in the future so I want to make sure the system is robust.

My first problem deals with all of my old images and what is the most efficient way to add keywords and ratings to them. I found religion in mid-2002 regarding file naming and have been naming all original files since then with the convention YYYYMMDD-subject-counter.ext, then subsequently refined it to YYYYMMDD-subject-originalfilename.ext (where the original filename was something like _MG_1234 and the subject name could be blank for household shots or have the name of the event or place). When I switched to the file naming scheme, I went back and renamed all of my old original JPGs so all 9,000 originals have this naming scheme.

The derivatives aren't quite so neat. Up until I switched the filename approach in mid-2002, I renamed all derivatives with descriptive names (easily identifiable in Explorer) with no dates or numbers. Unfortunately when I look back at those files now (I'm guessing only about 600), I find that the capture date isn't accurate so any renaming programs that I know of wouldn't be able to automatically rename based on my naming scheme. So one question that I have is whether I should bother to rename these, or if anyone knows of a program that would somehow match them to the original?

On the subject of renaming, many of those old names (originals and derivatives) are much longer than 31 characters because at the time I remember reading something about 62 or 64 characters being the limit (maybe this is true for Windows but not Mac?), and since I didn't want to be tied in to a proprietary system, I put keywords in the actual file name (no spaces). My future naming convention will be SJM-YYYYMMDD-20D-1234.ext where 20D is obviously the camera used and 1234 will be the original filename (I'm not quite sure why, but I feel I want to keep this in the name). This would still leave room for adding -mast, -bw, etc. for the derivatives. Quick aside, does the 31 character limit exclude the "." and the extension or is really a 28 character constraint? Given that the old names are often longer than 31 characters, do I need to rename them? If I don't, will I run into a problem if my future images will have the prefix SJM- while all others will start with YYYYMMDD? I was just about to say that a name sort wouldn't produce a chronological listing but since numbers come before letters, all of the numbered files would naturally precede any newly named images taken in 2006, so I don't think that would be a problem.

Now for my chicken and egg questions. Should I rename, keyword, and rank my old files in Bridge or iView? Should I do all of that work in my current folder structure or should I move to the new bucket system, get all of my files/folders in their eventual resting place and then rename/keyword/rank? FYI, while my current folder hierarchy is very organized (identical trees for Original and Derivatives such as D:\Edited Photos\YYYY\YYMM\event), it was horrible for backing up because it was tough to follow when I had enough to burn to disc (I'm not crazy though, I run a RAID 1 and also have a copy on a NAS and an offline HD). Back to renaming, etc., should I do all of the rename/keyword/rank work for the Originals and the Derivatives? Assuming I do the work for both, is there an easy way to do this in iView (I'm pretty sure you can't in Bridge and I'm still demoing iView so I'm less familiar with that app)?

Okay, I guess I've used up way too much screen and server space so I'll leave it at that for now.

Thanks very much in advance for any assistance.

Steve

[danaltick]

Steve,

I do sympathize with you.  And trust me, I'm certain you're not the only one experiencing a brain freeze over all this.  It can be overwhelming.  I happen to have the good fortune of having many years of computer experience as a programmer; however, I doubt very seriously I'm as good a photographer as you are, or probably even anybody on this forum for that matter.  That's why I think this is a great place for us to learn from each other.

I don't really know where to begin in trying to answer all your questions.  However, from reading through it, I'm fairly certain that the whole picture of DAM has not completely set in for you yet.  It truly is a lot of information to grasp.  I've actually read the book two times now and am reading it through again as I do the migration, and keep in mind, I have years of computer background, so I know it's got to be mind boggling for those photographers who don't.  Not to mention those who have archives and archives of film....that scares me just thinking about it.

Ideally you would want to do the migration by moving all of your files through the working files structure that I have listed in my migration thread, but that's an ideal world....one that is not practical for most of you who have thousands of images.  So what's the next best thing then?  For that I'm going to have to refer you back to chapter 9 of the book.  Read it very carefully.  You can skip the part on migrating from film.  I think Peter does about as good a job as can be done explaining how to handle large amounts of legacy images.

Read pages 185 - 191 for information on how Bridge can be used for rating and keywording your images.  However, in your case, I think you might be better off just following the techniques Peter lists in Chapter 9 for getting your images directly into your archive for use with your cataloging software.

Near the bottom of page 162 (last bullet), it says 31 characters (plus the extension).  That means a total of 35 characters counting a 3 character extension.  Keep in mind, this is a guideline, not a hard fast rule.  Once again, use the techniques spelled out in Chapter 9 for renaming your legacy material.

After reading through these pages again, if it doesn't answer most of your questions, feel free to post back with those still remaining.

I do hope this helps and good luck.

Peter may have some follow-up, but I know how busy he must be, so I'm doing my best to offload him somewhat given my schedule as well.

Best Regards,
Dan

[Steve Miller]

Dan,

Thanks for your help. I definitely don't have your programming background, but as a casual user of computers that for five or six years has been using Norton Ghost to keep a clean copy of my clean Windows machines (credit to Fred Langa of LangaList - highly recommended), I'm definitely on board with most of the concepts encompassed in DAM. Since mid-2002, I've been concerned about not being locked into a proprietary DAM solution (though of course had no idea it was called DAM), thus the reason for the old keyword-embedded filenames. I'm ecstatic to have stumbled across Peter's book and this forum because I'm certain that I've finally found a solution. The brain freeze comes with the chicken and egg questions of what to do first. That said, I won't waste anyone's time with additional questions until I re-read Chapter 9 (for the third time). If you don't mind, I'll be back to you soon thereafter.

Best regards,

Steve

P.S. On a tangentially related subject. I notice from your previous thread that you use ExplorerPlus instead of Windows Explorer. There used to be a bunch of these programs out there but none of them seem to have been updated in the past couple of years. Do you like this program? Does it show folder size for folders in the right hand window (to help in knowing when a folder needs to be "closed" and backed-up to DVD)? Thanks again.

[danaltick]

Rick,

Unfortunately ExplorerPlus does not show folder size either.  I have to right click on the folder and select properties to ge that.
I purchased ExplorerPlus a few years ago soon after a new release came out.  And yes, it does have some nice features.  It's definitely a step up from Windows Explorer and probably still one of the better Windows browsers out there right now for the money.  I'm an avid reader of PC Magazine, and they rated it highly a few years back.  However, you are right, they have not updated it in a while and I really don't know if they will, so I can't really recommend it based on that.  I think Novatix may have actually sold it off....not sure.

Dan

[peterkrogh]

Steve,
Not a lot of time to respond, but to add to what Dan said...

1. I think I would keep the old names as they are.  New names will probably just confuse the issue.

2. Can you break your current legacy directory into Buckets?  If this is a central organizing principle, then you will not want to shed the folder-based information too quickly.  You might find it easier to sort out form the old directory, then split into buckets, or to split first, then sort.  I would probably sort with iView to make the buckets.

3. Do your organization in iView, not Bridge.  You can keep "snapshots" of data organization by using virtual sets.  (like when I moved from iView 2.6 to 3.0, I made some sets that indicated what Label everything had.  When I updated, I reimported the metadata, and some label values changed.  Since I had kept the old information by making sets "has 2 star rating 051120" I was able to resolve the discrepancy.)

4. GET COMFORTABLE WITH THE NEW SYSTEM BEFORE YOU TACKLE THE OLD STUFF. (unless the old stuff is at real risk, due to not being backed up)
Hope this helps.
Peter

[Steve Miller]

Peter and Dan,

Thanks again for your suggestions/help. Here's an update from where I was in the original post:

1. Since all of my files fit on a hard drive, and since I had a pretty organized legacy folder structure (parallel original and derivatives folders broken down my year and month and when applicable, event), moving all of my files into the bucket structure is actually very easy. The only delay I'm encountering is that because I like the idea of the "pretty good print", I'm re-editing my RAW shots from last year before I convert them into DNGs. I was very busy with triathlons last year so my "to be converted" backlog piled up and when I did do the conversions, I used C1 or RSP which of course don't flow through to the DNG settings. I've now relocated all of my original files from 2000-2004 and the last six months of 2005 (still need to edit and relocate the first half of last year). I love the bucket system because it will allow me to incrementally backup just the new files added (and now this will be easily identifiable - unlike with my old system).

2. Very good advice to first get comfortable with the new system before going back to my legacy files. Only problem is that I became so overwhelmed with untouched files last year, that I've sort of shut down my desire to take new shots until I can clean up my old mess (not so much the catologing - which can wait - but the editing of my keepers from last year).

3. After re-reading the book, particularly Chapter 9, I decided to first move to the bucket system (in progress as stated above), then once that is set, I'll use iView or Idimager to do all of my ratings, labels, metadata, keywords, etc.

I have two questions:

1. For all of these legacy images, should I do all of the ratings/keywords/metadata for both the originals and the derivatives? If so is there an easy way to do this (i.e. to match an original to the derivative en masse)?

2. I'll pass on the re-naming of my old images in the format YYYYMMDD-subject-counter.ext, but am wondering what kind of problems I may run into since they are much longer than 31 characters. I'm pretty certain that Windows can handle ca. 60 characters so is this a Mac limit? Any risk if I eventually make the move to a Mac? (which I contemplate weekly)

Peter, once again I can't tell you how happy I am to have come across your book. I've been fairly organized (on a pre-DAM level) but had no idea how to efficiently setup my backup routine and use metadata/keywords/etc. and be certain that I wasn't sigining on to a specific company's proprietary solution. Just what the doctor ordered.

Steve

[peterkrogh]

Steve,
as far as question 1, without specific knowledge of your archive, I would say to keyword the RAW first. This will be the more complete archive, and so that's where I would start.  I would expect tools to become available in this year to more efficiently transfer the RAW file metadata to derivative files with the same root name.

As far as 2, I would suggest that you keep it to 31 +.  But you knew that.  What are the potential downsides?  In general, needing to rename as you send files to others, and the possibility that you will have to rename the whole data set at a later date to conform to a system or software application that you will be moving to.
Peter