My-First-DAM-Migration Log (stardate 0601.10)

[danaltick]

Well, I’ve finally begun my first DAM migration and plan to use this thread to keep a log of my progress, resolve issues, discuss alternatives, feedback, etc. as I work through the migration.  Hopefully this thread will prove beneficial to all the other DAM noobs out there .  Keep in mind that I'm Windows XP based, but hopefully this thread will prove beneficial to the Mac guru's as well.

I comment on my starting point in this thread http://thedambook.com/smf/index.php?topic=106.msg761#msg761.

I spent a few extra days doing some reading here www.controlledvocabulary.com.  I wanted to make sure I had a well thought out controlled vocabulary and keywording strategy in place before building and cataloging my infrastructure.  I feel I’ve struck a balance for the time being, and it's safe to proceed.

Last weekend, I installed my internal harddrives and tested them out thoroughly.  My LAN is setup as follows:
    PC 1: primary archives (for both original and derivative), primary WorkingFiles folder, & dual layer DVD burner (my workstation)
    PC 2: secondary archives (PC located in secure building)
    PC 3: secondary WorkingFiles folder
    PC 4: "Transferred to Archive" folder
    Epson P-2000: 40GB offline USB2 WorkingFiles folder

To start the migration, I initially purchased only two additional internal 250GB SATA harddrives: one for the primary and one for the secondary.  I have already installed, formatted, and run Norton Disk Doctor on them.  They are installed in separate PC’s on the LAN as described above.  I purchased only two for the following reasons:
    1) Lower initial cost
    2) Make use of the six-month rule base on current requirements
    3) Lower power consumption (no need to upgrade power supplies)
    4) Less management overhead starting out
    5) Already have space on LAN for WorkingFiles and TransferToArchive folders

Because I’m starting with only two harddrives, I had originally planned to partition them into two partitions: two-thirds RAW, one-third DRV.  However, I realize now I can fully utilize the storage on each drive by structuring the bucket system as follows using just one partition:

                                                 Media_nnn_yymmdd  (limited to bucket size)
                                                             |
                           -----------------------------------------------------------------….
                          |                                  |                       |
               RAW_yymmdd                         DRV                Music, Video, etc.
                          |                                  |
            Descriptive subfolders      Descriptive subfolders

Peter, if you disagree with placing all media into the same bucket (segregated of course), please let me know.  Thanks.  

I also have an offline USB 2.0 storage and viewing device (Epson P-2000), that will double as a proofing device and offline WorkingFiles storage device.  A review of it is here http://www.luminous-landscape.com/reviews/accessories/P-2000.shtml.  Hopefully this will eliminate the need for a laptop in the field....at least for now.

At this point, I still need a DVD burner and a possibly a second offline portable storage device to keep three copies in the field.  However, these two items can wait until after the migration.  As Peter says, always wait to buy technology until it’s needed…that has really paid off for me.

I have also deviated somewhat from the WorkingFiles and TransferToArchives structure mentioned in this thread http://thedambook.com/smf/index.php?topic=66.0.  I will describe this on my next post.

My time is limited, but I will do my best to keep the thread up to date as I move forward.

Best Regards & Good Luck to All,
Dan

[danaltick]

Peter,

I created a flow diagram of my WorkingFiles folders and would like to insert it as an image; however, I'm not sure how to do that.  The insert image icon doesn't supply me with a folder dialog to browse to the image on my PC.  Please help.  Thanks.

Dan

[RDThomas]

I have started the process myself and glad to see this thread.  I do not have nearly as extensive a network but have a singelePC with 4 internal harddrives and one firewire 400GB external.  I have been a digital photography fan for a good number of years with no strategy for DAM besides Windows folders by subject.  I have over 25,000 images and I just can't keep up with knowing what is where in the folder structure so I got Peter's book and read it over the christmas holidays and I have started the process.  I am using a new internal drive as a migration drive for the DAM new system and backup to the external and DVD's as I fill up my buckets.  New work I am using the naming convention of RAW_###_MMDDYY and DRV_###_MMDDYY and as these buckets fillup back up to DVD.  That is working fine and I am using iView and Bridge as peter has described in The DAM Book.  Only thing I am not fully comfortable with is having the DRV files in a different folder/structure.  But this forces me to insert keywords and structure by catagories. 

Now for legacy stuff (my 25000 images).  I am moving one folder at a time to the new drive with its current folder name into a Leg_###_MMDDYY bucket.  I then open bridge and do a review of the images (haven't looked ast some of these in years) and assign ratings, delete the obvious dogs, rename with NNN_MMDDYY(original file date)_Filename, assign ratings, assign keywords, load std boilerplate IPTC data.  I then add this directory to the iView catalog and create (as necessary) and assign to catagories.  I wish the ratings transfered over from Bridge.  I will have to look at Peter's book and see if he had a tip on that.  One other thing I have learned is the value of converting canon Raw files to DNG esp if you have modified the color balance, exposure, etc. in raw and saved.  In iView you will not see these changes in the thumbnail created as iView does not process the XMP file.  When you make a DNG with imbeded JPG it reflects these raw changes you made and shown in the iView preview.

More to follow.  Comments appreciated.

[wombat2010]

Thank you both for posting your experiences as you migrate.  I am getting ready to do so myself (just set up an external drive last night and will install Retrospect Backup 7 over the weekend), so I appreciate seeing how others are doing it.  I'm a visual person (I guess that's good for photography), so seeing the layout of the various file structures really helps.

The snapshots in Peter's book were good, but I really feel I need more detailed, system-wide snapshots (that encompass originals, derivatives, working files, archives, etc.) to understand how they all interact and where they are situated.  For that reason, Dan, I am very much looking forward to seeing your file flow diagram.  In fact, if you are unable to post it, I would like it if you would email it to me at the address in my profile.

One other (semi-OT) thing: For purposes of migrating, organizing, and archiving, do people here think it is more useful to have a program like iView or CS2's Bridge?  At the moment I edit in MS Digital Image Pro 2006 and have only Breezebrowser for organizing (no cataloging software).  Because I only shoot JPEGs (I have only a Sony V3 (for now) and few programs support its RAW format), I suspect I'm better off getting cataloging software first.  I can make do with Digital Image, even though I am itching to use layers and some of the advanced touchup tools.  Thanks.

Steve

[danaltick]

Well, Wednesdays are full days for me, so I didn't get a chance to do any further work on the migration.  Hopefully tomorrow I will have some more time.  I will also attempt to answer your questions then as well.
All I ask from everyone is that you try to make an effort to keep this thread on topic; that being, keeping all questions related to issues/alternatives/recommendations related to this migration effort.  I would like for others to be able to reference this thread easily for information that might assist them in their own migration.  Thank you very much.

Best Regards,
Dan

[danaltick]

RD,

I’ve actually contemplated keeping my Derivatives together with my DNG’s as well.  I can see advantages to this from a grouping standpoint.  However, I can also understand why one might want to keep his originals separate too.  Peter actually cites a number of advantages to that in Chapter 3.  In the diagram above, you will see that my Derivatives and RAW’s are on the same partition, but they are still separate folders.  Because I’m just now beginning this journey, I’ve decided to stay with the book and keep my RAW’s and Derivatives separate.  I’ve even spent some time mulling over different subfolder structures within the archive.  For example, should I attempt to use actual keyworded subfolder hierarchies within the archive similar to virtual sets or just use descriptive subfolder names like Peter does in his book?  After going over several different scenarios in my head, I’ve come to the conclusion once again to just stick with Peter’s technique.  It’s simple, quick, tried, and proven.  I think the main point here is to strike a balance between expediency and maintenance.  I believe Peter has been there and done that as he indicates in the book.  So for now I intend to just make my subfolder names as descriptive as possible (possibly using some keywords) without attempting to create an elaborate set of keyworded hierarchies within the archive.  Reference the book for some examples.   I’m making a conscious decision to place my emphasis on my cataloging software (i.e. iView), making sure to preserve, maintain, and migrate that catalog through time along with the archive.  Remember, with the catalog, you’re free to group your images, as you like.  If I attempt to do this with folders in my archive as well, not only will it be time consuming, but also now I could see myself constantly comparing my archive sets to my catalog sets for consistency, which could be impossible given that one is virtual and the other physical.  Therefore, I’m going to consider the catalog and cataloging software an integral component of my archive.  I can’t afford to get bogged down just updating the archive.  However, I do believe I am going to add a few extra levels to the diagram above, but only to facilitate easier archiving with a little extra grouping.  Stay tuned for a posting of it.

You should use Rank & File to transfer your ratings from Bridge to iView.  Reference Chapter 6 and 7 for more information.  Also, search for Rank and File (or R&F) in the forums for known issues.

Peter also mentions the no-sidecar advantages of DNG’s in his book, including the iView advantage with the previews.  I can’t remember the chapter right off hand though and I don’t have the book handy.

Steve,

I plan to post the WorkingFiles diagram as soon as I get instructions from Peter on how to insert it.  He’s gone to MacWorld now, so we may just have to wait on that.  I’m making a few alterations to it right now anyway, so it’s not quite ready yet.

Steve, you absolutely must have a catalog manager to implement Peter’s DAM system.  In order to understand why, I’ll have to refer you back to the book.  The advantages are covered in several places; especially Chapter 7.  I have actually read the book more than once.  There’s allot of information to absorb, but Peter does an excellent job at presenting it in a straightforward manner.  He also points out the DAM requirements for cataloging software.

I should have an update on my progress in the next day or two as time permits, so stay tuned.

There is one other thing that I would like to make note of concerning my unique LAN setup.  If you will notice, a LAN setup is not cited in Peter’s book in Chapter 4.  I have taken this into consideration and opted for it anyway because my LAN is where my investment is, and it’s already in place.  It is a 24-hour LAN.  Here is a list of what I consider to be the advantages and disadvantages of using a LAN for DAM:
     Disadvantages:
        1) Security risk (viruses, Trojans, worms, theft, etc.)
        2) Environmental risk (lightening, fire, etc.)
        3) Network bandwidth limitations
        4) Greater power consumption
        5) Shorter harddrive lifespan
        6) Possible much longer disaster recovery from DVD’s rather than offsite harddrives if LAN is destroyed or stolen.
     
     Advantages:
        1) Always-online access to backups
        2) Supports automated nightly backups of both working files and archives
        3) More concurrency between primary and secondary archives
        4) Completely automated except for DVD burning

Here is how I am addressing the disadvantages:
    1) 100 Mbit/sec LAN is plenty sufficient for nightly mirror jobs
    2) LAN is behind a hardware NAT firewall, invisible on the internet
    3) Each PC uses a software firewall with program access control
    4) Each PC uses up-to-date anti-virus software with automatic updates
    5) Each PC uses up-to-date anti-spyware software with automatic updates
    6) Each PC’s O/S is backed up nightly to other PC’s on the LAN using incremental backups going back two weeks.
    7) Each PC uses high-end surge protectors and are well grounded and ventilated.
    8 ) Archive harddrives are spec’d for 24-hour operation with very long MTBF’s.
    9) Secondary archive PC is in a separate building
   10) Will burn a bucket DVD at least every two weeks even if the current bucket isn’t full and keep DVD’s offsite.

Basically I’m making a trade-off between the overhead of dealing with an offsite caddie versus the overhead of burning DVD’s more frequently and possible DVD disaster recovery if the LAN is compromised or lost.

Dan

[AlanDunne]

Only thing I am not fully comfortable with is having the DRV files in a different folder/structure.  But this forces me to insert keywords and structure by catagories. 

RD, like Dan (and of course Peter) I am keeping derivatives in a separate physical location file location than originals. This greatly simplifies managing backups and archiving, as is described in The DAM Book. Where I deviate from Peter's practice is as follows. Peter has separate catalogs for originals (several, in fact as his original archive grows fast) and one for derivatives. The number of derivatives is much smaller, especially one you separate proofing from preparing masters.

I am slowly migrating my old collection into IVMP and the rest of the DAM system. Unlike Peter, because my collection is smaller, I am keeping originals and derivatives in the same catalog, even though they are physically stored in different folders and buckets. Since my derivative file names have the original name as the root, it is easy to find the original and derivative. I am finding this makes it easy to find and correct discrepencies in rating and labels (a side effect from poor practices over the past few years). The other reason I am doing this are:

• As my collection grows over time, I expect the catalog size supported by IVMP will grow as well
• I believe that in the not too distant future, IVMP and other catalog tools will have much more sophistcated features for managing the relationship between originals and derivatives

I currently use a "virtual set" to track all the derivatives in the catalog in case I need to have a narrow search focusing on derivativs only. Peter accomplishes this by only searching his derivatives catalog when he knows in advance that it is a derivative that he seeks.

Cheers ... Alan

[danaltick]

Keep in mind too that you can also keyword your derivatives with the keyword "derivatives", and you can then add the keyword "proofs" to all your proofs....the skies the limit with a good cataloging application and keywords can be very powerful.  I'm just now beginning to realize this.

Dan

[danaltick]

Below are the flow diagrams and file browser images for my bucket structure and working files structure.

At the lower levels of the bucket structure I will place my descriptive jobname folders copied from the final folder of the working files structure.  Note that I still intend to use the filenaming conventions listed on p.82 in the book for the image filenames.

The bottom JobX and JobY folders in the working files structure are my descriptive jobname folders.  Once the folders have reached the final working files folder, they will then be dragged to the appropriate shortcut to copy them to the archive (see the browser diagram below).  Dragging in this case will copy and not move because the archive is on a different drive.  Then they will be dragged to the Copied2Archive subfolder and remain there until the current bucket has been burned.  I have opted to do this on my LAN instead of using the "Transferred to Archive" folder on PC4 as previously mentioned.  PC4 will not be needed.  Since the Copied2Archive folder gets backed up along with the working files structure, I am keeping three copies at all times.

The curved lines in the working files structure represent data flow as the folders are moved through the working files structure.

Note - these structures support both RAW and JPEG jobs.

Feel free to post your comments, suggestions, rejections, etc.  Keep in mind, this is my first attempt at this.

Best Regards,
Dan





Here is the working files structure as it would look in your O/S file browser:

[danaltick]

Peter,

If I'm missing some important concept(s) here, please let me know.  I don't want to get down the road only find out I've made a wrong turn somewhere.  I tried to strike a balance between expediency and maintenance with the bucket structure.  I realize hindsight is 20/20 here, and you may not agree with this.  Thanks.

Dan

[danaltick]

Well, I managed to create two GBM Pro mirror jobs and test them out yesterday.  I created one for working files and one for the primary archive.  I scheduled these to run nightly and they ran successfully last night.  I also created a desktop icon for the working files job so I can run it immediately after a camera import.  Since my bucket structure contains both my originals and derivatives, I only needed to create one job for the Media_001 bucket.  Once that bucket becomes full and gets burned to a DVD, I will postpend the yymmdd to the bucket, rename the job to Media_002, point it to the Media_002 bucket, and keep it moving right along with the buckets.  Also when the RAW folder fills up, I will postpend the yymmdd to it as well indicating the date of the last RAW file added to that folder.  If I ever want to list all of my RAW folders across buckets in date order in the future, I can easily do that with my file browser.

Next step for me will be to move my first batch of RAW files through the working files structure...stay tuned.

Dan

[danaltick]

I successfully moved my first batch of RAW images through the working files structure and archived them.  Everything went fairly smooth.  I had only one hiccup with Rank & File hanging up on me.  I was able to fix it by purging the folder cache in Bridge.  Peter is aware of this problem and working to correct it.

My first batch contained only 80 RAW images.  It took rougly 7 seconds per DNG conversion.

One thing I did notice; the JPEG full-sized previews in the DNG's really look awesome in iView's slideshow mode....really nice and sharp.  I don't remember, but can someone tell me if Bridge uses the sidecar files or the embedded DNG JPEG when viewing the DNG's in slideshow mode?  The iView DNG's appear sharper to me.  Is it my imagination?  Thanks.

Dan

[AlanDunne]

Dan,

somewhat ironically, Bridges uses it own cached previews for DNG's, not the embedded jpeg preview in the DNG. I believe the reasons for this are:

- consistancy amongst all file types (not all have embedded previews)
- even with DNG, it is optional to include an embedded preview

I also believe that the Bridge previews are not "full size". If you use the "full size" selection for your embedded preview in the DNG (and I understand that you do), then then Iview is using a full size jpeg preview on the screen and Bridge is not (if I am correct about Bridge not being full size).

When you view the full size preview from say a Canon 20D (which I recall that you have), then Iview is most likely down sampling to display the full size preview in the DNG. It is possible that the down sample process might create the illusion of a sharpening effect. That might explain whay the Iview preview looks sharper than the Bridge preview. Or it could be simply a matter of higher resolution.

Both the embedded preview in the DNG seen in Iview, and the Bridge preview, should reflect the sharpening setting in ACR, so that should not be a difference unless your DNG preview and Bridge preview somehow got out of sync.

Cheers ... Alan

[danaltick]

When you view the full size preview from say a Canon 20D (which I recall that you have), then Iview is most likely down sampling to display the full size preview in the DNG. It is possible that the down sample process might create the illusion of a sharpening effect. That might explain whay the Iview preview looks sharper than the Bridge preview. Or it could be simply a matter of higher resolution.

Alan,

I think it's one of the two reasons you cited above.  I'm using a sharpening setting of 75 as my ACR default.

Dan

[AlanDunne]

Below are the flow diagrams for my bucket structure and working files structure.  These of course can be tailored to meet your specific requirements.
...
Feel free to post your comments, suggestions, rejections, etc.  Keep in mind, this is my first attempt at this.

Dan, since you invited, here are some questions/comments ...

• You seem to be using different folders for "Personal" vs "Job". Is there a reasons for the physical separation? Could you not just keyword to differentiate, or use a virtual set, or are you using different catalogs for personal and job?
• You are using different folders for RAW and jpeg originals. I am currently doing this as well and it does result in some simplification. I am wondering if you ever shoot in RAW+jpeg mode on your camera? If yes, where would you put your jpegs that match the raws? Just so you know, what I am doing is keeping those jpegs with the raws, and images shot only in jpeg in a separate jpeg folder. Since I shoot 85% in raw only, 14% in jpeg only, and maybe 1% in raw+jpeg, it is not something I worry too much about, so the question is more for intellectual completeness.
• You have a "Proofs" folder as a subfolder (once removed) of "Derivatives". Since you are using the embedded full size preview in DNG, would you not just use those for proofing purposes, or do you have any specific requirements that would trigger the need for a proofs folder? Do you intent to keep those proofs long term, or are they throw away once their purpose has been served?

Anyway, thanks for your contribution to this thread. It is most useful to see all the diagrams. I wish I was able to document my work in such a prolific manner as yourself!

Cheers ... Al