DNG, Backup, Metadata synchronization

[David Anderson]

I don't sync back to DNGs wholesale, unless there is a reason (file, format or application Migration, for instance).  All the informational metadata can be safely and efficiently wrangled from within iView.  Although the risk is small, every time you touch a file you introduce the possibility of corruption.  Push information back into the files because there is a reason to do so.  Backing the information up is not a good reason.

Peter,
I'm very close to making the switch to a DNG-based workflow, based on the advice you gave in The DAM Book. One of the good reasons you gave on page 17 for choosing DNG was "because the DNG format is openly documented, all sorts of metadata can safely be written to the file, with no danger of it becoming unreadable or of corrupting the file". That seems a little at odds with what you have just posted here. Have you had some bad DNG experiences that caused you to change your mind?

In the middle of page 18 of your book, you also make the point that DNG's open format makes it possible for any third party app that reads DNGs to also read and add to the metadata stored in the file. I'm therefore a little surprised that you are not syncing the metadata back to the DNG as a matter of course - or do you seldom use anything other than IVMP to read/write metadata after initial ingestion?

If Steve is intending (like me) to keep his original Raw files in addition to his DNG conversions, the risk of DNG corruption when syncing metadata can probably be ignored. What do other people think on this point?

David  

[peterkrogh]

David,
The main point is that any time you touch a file, you introduce the possibility of file corruption.  This goes for DNG, TIFF, DOC, or any other file.  If you are adding metadata to thousands of files on a regular basis, maybe that is millions or tens of millions of touches over the life of an archive.  It's pretty safe, generally, but since there is really not a reason to do it on a regular basis, I advise against it.
Peter

[Steve Miller]

Steve, I'm not sure how much of what you outline is regarding straightening out old work, vs bringing in new work, but it sounds basically right.  A couple points:

1. I would use iView MediaPro to help me straighten out the old files.  MUCH easier than trying to do this in the Finder/Directory/Bridge.  

2. I don't sync back to DNGs wholesale, unless there is a reason (file, format or application Migration, for instance).  All the informational metadata can be safely and efficiently wrangled from within iView.  Although the risk is small, every time you touch a file you introduce the possibility of corruption.  Push information back into the files because there is a reason to do so.  Backing the information up is not a good reason.

3. The Macbook should be fine for color work, if you calibrate it.

Peter

Peter,

1. I absolutely plan on using iVMP to take care of all of my old files.

2. David got to this before I could, but I echo his sentiments. For me, I want the DNG to have all of the latest info/metadata so if I move to another program, all of my work won't be left behind. Given that I doubt I'll give up the security of keeping a copy of the RAW, the risk of corrupting a DNG file is quite low (as David pointed out), I'm just surprised to hear that you are concerned enough to not bother copying the keywording/rating work done in iVMP back to your DNGs. Seems to go against the principle you highlight in the book of making sure to stay in non-proprietary systems.

3. I haven't switched my Windows PS CS license into a Mac license because I assume the speed in non-UB is pretty slow. Because of this, I can only work in PS on the Win side of my MBP (via Boot Camp) where although I could probably profile the monitor, the brightness control on the Win side is horrible (seems like a bug because you can either have it VERY bright, or 10 levels of dark to very dark). So currently my MBP isn't good for color critical work. I am tempted to download the Mac version of CS2 and see how the speed is. If it's fast enough, I could then use the MBP and calibrate the monitor on the Mac side where the brightness setting works (then hook it up to my ACD23 for a nice two monitor setup).

We'll see.

Steve

P.S. Anybody have any experience with CS2 on a MBP? Speed? Monitor calibration with eye-one?

[BobSmith]

Anybody have any experience with CS2 on a MBP? Speed? Monitor calibration with eye-one?

I'm using CS2 and an EyeOne on an Intel iMac.  CS2 is useable but slow... running slightly slower than on my 1.6GHZ G4 Powerbook.  EyeOne and EyOne Match software work just fine.  IView runs quite well on the Intel powered Mac.

Bob Smith

[David Anderson]

David,
The main point is that any time you touch a file, you introduce the possibility of file corruption.  This goes for DNG, TIFF, DOC, or any other file.  If you are adding metadata to thousands of files on a regular basis, maybe that is millions or tens of millions of touches over the life of an archive.  It's pretty safe, generally, but since there is really not a reason to do it on a regular basis, I advise against it.
Peter

Peter,
I fully accept your point that there must be some risk (however small) to an image file every time you update its embedded metadata, but I don't quite understand your "millions or tens of millions of touches" comment. Perhaps my lack of a detailed knowledge of statistics prevents me from getting the point, but at present I don't see the relevance of quoting the aggregate of all changes to all files in an archive. Surely it is only the number of changes to an individual file that matters when assessing risk? In which case, we might really be talking about just three or four metadata updates during the life of an average image.

In addition, I think it is vital to make a distinction between images backed up to a DVD-R archive versus regular 'normal' backups to external hard disks. The former would not be recreated just because of metadata or ACR changes to some older files, while the latter would reflect the latest update status. There is therefore no risk of corrupting the primary archive, while still continuing to have the convenience of up to date (and backed up) DNGs that are fully portable to a range of apps.

Please feel free to poke as many holes in my logic as you can find before I start my voyage into the wonderful world of DNG workflow!

David  

[peterkrogh]

David,
In my case, 170,000ish images (so far in my young digital archive) multiplied by, say 10 metadata changes or additions each over time equals 1.7 million file modifications. Even if it's safe 99.99% of the time, that could still lead to some kind of problem in 170 files.  Of course the ones you are likely to touch most will be your most favorite/valuable.

The main point, is that most of the time this is entirely unnecessary.

As a method off backing up metadata, it's staggeringly inefficient.  Essentially you are scattering a few hundred K of constantly changing metadata into terabytes of unchanging image data.  And unless you bring the entire backup online and re-synchronize, none of it is reliably backed up.  So you would still have to back up the catalog documents themselves.

>

In addition, I think it is vital to make a distinction between images backed up to a DVD-R archive versus regular 'normal' backups to external hard disks. The former would not be recreated just because of metadata or ACR changes to some older files, while the latter would reflect the latest update status.

Not sure what the above means.

while still continuing to have the convenience of up to date (and backed up) DNGs that are fully portable to a range of apps.

 

I agree that pushing metadata back into files is the way to make the metadata portable, but I guess my question is when would you feel like you need to do this?

If I generate images files out of iView from the DNG (one of its main advantages) they contain the metadata that iView has in the catalog.  In my own workflow, it's only when I take the image into Photoshop ( a pretty rare occurrence for me - maybe 1% of my images) that I would need to push metadata back in to the DNG.

When I migrate from iView (which I expect to do at some point - maybe years or decades from now) I will certainly push back the metadata into the files.  But I will need to be doing the push at the last possible moment to make sure that all current information is updates.

Peter

[David Anderson]

As a method off backing up metadata, it's staggeringly inefficient.  Essentially you are scattering a few hundred K of constantly changing metadata into terabytes of unchanging image data.  

Peter,
I'm nowhere near the terabyte league, but I take your point that metadata has trivial storage size requirements compared to the actual images. Indeed that suggests an advantage of sticking to using Raw files, as all the metadata and ACR changes are stored in sidecars and the Raw file remains untouched.  

And unless you bring the entire backup online and re-synchronize, none of it is reliably backed up.  So you would still have to back up the catalog documents themselves.

There was never any question about not backing up the IVMP catalogue (I back up all my data). Perhaps my brain is getting a bit sluggish with old age, but I don't quite understand what you mean about bringing the entire backup online. I suspect that you may be referring to a situation where the size of your total image archive is too big for all of it to be kept 'online', i.e. on the hard disks permanently accessible by your main computer. It is highly unlikely that will ever be the case for an amateur photographer like myself. I use a Dell Precision 670 Workstation that can take up to four hard drives. At present, I have three SCSI hard disks installed, the largest being 300GB. I could therefore easily have 1.2TB of 'online' storage - more than enough for the vast majority of amateurs, and probably enough for a significant proportion of professionals. Almost certainly, 500GB drives will become available at an affordable price in the near future, thus making 2TB on online storage relatively easy to achieve.

In my case, all my images are in an absolute minimum of three places: (1) my DVD-R archive - including the ACR settings and metadata that were current at the time of burning each DVD-R, (2) my normal backup to external hard disk - kept up to date every day or two with any changes to ACR settings and metadata (and also including IVMP catalogue backups), and (3) the hard disks on my PC - by definition, the latest versions. I see no lack of reliable backup in such a scenario. Your circumstances, of course, may be very different.    

>

In addition, I think it is vital to make a distinction between images backed up to a DVD-R archive versus regular 'normal' backups to external hard disks. The former would not be recreated just because of metadata or ACR changes to some older files, while the latter would reflect the latest update status.

Not sure what the above means.

Your concern seemed to be the amount of 'touching' undergone by an image archive. In the case of the archive that sits on DVD-R, I was simply making the point that I never intended to recreate such DVD-Rs even after updating ACR settings or metadata for the live versions sitting on my PC's hard disks. In other words, my DVD-R archive will remain undisturbed, just as you recommend.

I agree that pushing metadata back into files is the way to make the metadata portable, but I guess my question is when would you feel like you need to do this?

I guess my initial answer is that I never thought about not doing it - a view that seemed to be encouraged by both your book and your criticisms elsewhere about Portfolio's inability to re-sync metadata to DNGs (which I took to mean was a daily requirement, not an unusual event in your workflow). I just assumed that that was what everyone did as a matter of course - indeed, I thought that the ability to store the metadata in the image file was meant to be one of the primary reasons for moving to DNG. You are now causing me to rethink my established assumptions about DNG workflow! Thinking hurts at my age.....

Another answer would be that I was planning to experiment with both Portfolio and IVMP (I have both) before deciding which product was normally best for my needs. I had toyed with the idea that perhaps I would find a use for both products - which would obviously require regular resynching.  

If I generate images files out of iView from the DNG (one of its main advantages) they contain the metadata that iView has in the catalog.  In my own workflow, it's only when I take the image into Photoshop (a pretty rare occurrence for me - maybe 1% of my images) that I would need to push metadata back in to the DNG.

There you go again, Peter, upsetting my preconceptions! I would never have guessed that so few of your pictures would be processed by Photoshop. Again, I just assumed that every decent image I want to use for any purpose will be enhanced within PS. Are you saying that 99% of your adjustments are achieved within ACR?

David

[peterkrogh]

Perhaps my brain is getting a bit sluggish with old age, but I don't quite understand what you mean about bringing the entire backup online. I suspect that you may be referring to a situation where the size of your total image archive is too big for all of it to be kept 'online', i.e. on the hard disks permanently accessible by your main computer. It is highly unlikely that will ever be the case for an amateur photographer like myself. I use a Dell Precision 670 Workstation that can take up to four hard drives. At present, I have three SCSI hard disks installed, the largest being 300GB. I could therefore easily have 1.2TB of 'online' storage - more than enough for the vast majority of amateurs, and probably enough for a significant proportion of professionals. Almost certainly, 500GB drives will become available at an affordable price in the near future, thus making 2TB on online storage relatively easy to achieve.

David,
I would suggest that I feel more comfortable keeping the main backup off-line and off-site.  In my case, this is the HD backup, which lives at Darren's house.  Keeping it online as you suggest means that it is subject to many of the same hazards as the primary - theft, voltage surge, virus, human error, even device failure.  By keeping the HDs off-site and off-line, and only synchronized when I have a reason to, I keep them walled off from many of these hazards.

I'm reluctant to configure a system as you do, since it puts too much reliance on the DVD backup.

So for me to sync to backups, I actually need to bring the drives on-site, mount and synchronize.  That's what I mean by bringing the backup online.

There *is* a difference between a hobbiest and a professional here.  You are unlikely to lose your house just because your photo archive goes bad.   ;-)

I guess my initial answer is that I never thought about not doing it - a view that seemed to be encouraged by both your book and your criticisms elsewhere about Portfolio's inability to re-sync metadata to DNGs (which I took to mean was a daily requirement, not an unusual event in your workflow). I just assumed that that was what everyone did as a matter of course - indeed, I thought that the ability to store the metadata in the image file was meant to be one of the primary reasons for moving to DNG. You are now causing me to rethink my established assumptions about DNG workflow! Thinking hurts at my age.....

Pushing metadata back into DNG files is, in my view, an essential ability.  That does not mean you want to do it with every piece of metadata.  You should do it when you need to do it. (Just because you can change your spark plugs, doesn't mean you would want to every time you drive). I continue to suggest that there is no reason to do it every time you change or create metadata, since there's generally no reason to do it on a daily basis.  Backing up the metadata is not a good reason, since this is a terribly inefficient way to back up metadata, and since it introduces the small possibility of corruption, every time you do it.

The best reason to use DNG is not because you want to push metadata into the file on a regular basis.  The best reason, in my opinion, is that you can bundle a fixed rendering of the image along with the RAW data, and that this rendering can be extracted and used.  It can even be extracted and used by software that does not have RAW file parsing capability.

There you go again, Peter, upsetting my preconceptions! I would never have guessed that so few of your pictures would be processed by Photoshop. Again, I just assumed that every decent image I want to use for any purpose will be enhanced within PS. Are you saying that 99% of your adjustments are achieved within ACR?

The things I need to do with my files, in descending order of frequency are:

100%   View Onscreen to make judgements about the images  
50%  Create web galleries for client review (even if the client is me)
5%  Make proof prints (prints that can be made straight from the DNG that has been adjusted by ACR. These might be 4x6 for friends, email JPEGs, or batch conversions for delivery)
1 %  Make Master Files (Send to Photoshop and make the best possible version)

All bu tht elast of these can be done straight from the DNG very quickly and efficiently, without creating lots of derivative files to wrangle. Lightroom will reduce this percentage even further, as more of what I need to do in Photoshop can be done in LR directly from the RAW file, and stored as a non-destructive rendering in a very small DNG.

I know it's hard to get your head around, and I know you're not the only one wondering about all this.  But in any case, that is the reality of the imaging present and future.

It's a good future.

Peter

[David Anderson]

David,
I would suggest that I feel more comfortable keeping the main backup off-line and off-site.  In my case, this is the HD backup, which lives at Darren's house.  Keeping it online as you suggest means that it is subject to many of the same hazards as the primary - theft, voltage surge, virus, human error, even device failure.  By keeping the HDs off-site and off-line, and only synchronized when I have a reason to, I keep them walled off from many of these hazards.

Peter,
I couldn't agree more and I never suggested otherwise. You're a busy guy and I guess you have forgotten several of my earlier posts in the original thread that started this discussion (I'm in no position to criticise, as my own memory is far from perfect - a fact my wife will be only too ready to confirm!). My total archive is small enough to fit on one external 300GB disk and I have two such disks. One of them is always kept off site. Even my other external HD is normally kept locked away in a separate room from my main PC. Every week or two I swap the two external hard drives. As stated in an earlier post, I fully expect to upgrade these external drives to 500GB models within a year or two so that I can continue in this manner for some time to come.

The best reason to use DNG is not because you want to push metadata into the file on a regular basis.  The best reason, in my opinion, is that you can bundle a fixed rendering of the image along with the RAW data, and that this rendering can be extracted and used.  It can even be extracted and used by software that does not have RAW file parsing capability.

How do you back up this rendering data? Does this not mean backing up the DNG itself? If yes, then is there a way to resync metadata at the same time that does not involve another rewrite of the DNG?

As regards your minimal use of Photoshop, I guess I'd better start experimenting a bit more with editing DNGs in ACR to see if I can speed up my workflow in the way you suggest. As always, thanks for your help.

David

[peterkrogh]

How do you back up this rendering data? Does this not mean backing up the DNG itself? If yes, then is there a way to resync metadata at the same time that does not involve another rewrite of the DNG?

I get images good enough to proof with before I convert to DNG.  This way, the minimal corrections are stored in the DNG, and the backup versions.

When I am readjusting, typically that is when I am making a master file.  In these cases, the loss of the proofing settings would generally not be a disaster, since the new derivative file is what I would want to use on most cases.

Additionally, when I create the version that I am going to take into Photoshop, I am typically doing work to the file that I don't save back to the DNG. I back the sharpening down, and pull some punch out of the blacks, since I will adjust both of these in Photoshop.  Keeping the existing proofing adjustments in the DNG continues to make it suitable for batch proofing/rendering with the rest of the shoot.

I hope that before too long, DAM software will be able to harvest and restore this data in the same way it can with informational metadata.

Lightroom will change this workflow, for the better.

Peter

[Rick McCleary]

Additionally, when I create the version that I am going to take into Photoshop, I am typically doing work to the file that I don't save back to the DNG. I back the sharpening down, and pull some punch out of the blacks, since I will adjust both of these in Photoshop.  Keeping the existing proofing adjustments in the DNG continues to make it suitable for batch proofing/rendering with the rest of the shoot.

 Just to clarify a point for David -

After you have made your ACR adjustments, converted to DNG, archived and cataloged, the idea is to leave the DNG alone.  When creating a master file in Photoshop, you'll open the archived DNG in ACR and probably make some adjustments to the embedded settings (as Peter pointed out - sharpening, shadow, etc.).  Once you have made those changes, open the DNG into Photoshop by option-clicking on the "Open" button which will "Open A Copy" without sending these changes back to the archived DNG.  The key is to open a copy.

[David Anderson]

Peter,
You appear to be saying that 99% of your image editing is carried out on the Raw files. I guess you could also proof print or make web galleries from the Raw files, if you don't mind using sidecars, so that is not a benefit enabled by DNG conversion.

So far, I'm getting the impression that the benefits of DNG conversion (in order of priority, starting with the highest priority) are as follows,

1. Enabling a catalogue app such as IVMP to display fully adjusted thumbnails (because such apps can't read sidecars)

2. Avoiding the inconvenience of sidecars

3. Protecting against obsolescent proprietary Raw formats

4. The ability to store metadata inside the image file - but only necessary on rare occasions, as discussed earlier


Am I on the right track? Would you add more bullets to my list? Would you alter the order of importance?

David

[danaltick]

Peter,

I'm curious how Lightroom will change this part of the workflow.

Dan

[peterkrogh]

David,
5. (really 1 or 2) The ability of DAM applications to create derivative files 10 times faster than Photoshop can do when creating from the RAW file data.

Dan,
LR will take it to .1%
Peter

[David Anderson]

David,
5. (really 1 or 2) The ability of DAM applications to create derivative files 10 times faster than Photoshop can do when creating from the RAW file data.

Peter,
That's interesting - and a new one for me. Can you please clarify exactly how you create such derivatives from within IVMP. Which application do you open the file with?

David