DNG Metadata from Bridge vs. iView Media Pro 3

[mikewren]

My backup scheme uses two fileservers, one at work and one at my home office. I use SSH (for secure/encrypted data transfer) and a file sync tool called Unison. Unison keeps a record of the files on each server, and computes the difference between the two, if any.  It then migrates ONLY the most recent PARTS of the file.  Unison does not transfer the whole file, only the changed parts... this saves a huge amount of bandwidth. Unison is based on a backup algorithm called RSync, except Unison is bi-directional, rather than uni-directional.

I've noticed that for DNG files metatagged in Bridge, the metadata is stored at the beginning/head of the DNG file.  In iView, metadata is stored at the end/footer of the file. For DNG files that are created in Bridge, then edited in iVew, the metadata is moved from the start to the end of the file. This is probably why metadata updates in iView are so quick; iView doesn't have to rewirte the entire file to make simple metadata update.

This difference isn't a big deal for usability; Bridge can still obviously read metadata written out of iView.  However, it can become a big deal when working with files only in Bridge.  Simple metadata changes made in Bridge (prior to opening in iView) causes Bridge to rewrite the entire DNG file, which when it comes time for  my nightly backup, causees the entire file to be transferred.

I'm not sure what the point of this post is, other than to share my experiences with this community.  I got so much out of Peter's book as a computer geek searching for a sane metadata scheme (rather than a photog searching for DAM), I wanted to contribute something back.

Maybe Peter can bring this up with the folks at Adobe so newer versions of Bridge will write metadata to the end of DNG files... since it's apparently more efficient to do so.

Thanks for the book and these forums, Peter!

EDIT: This is all happening in windows XP... I have no idea if this would apply to OS X users as well.  I don't have any Macs here with which to test.

[AlanDunne]

Wow, that's some post, especially for a first one. That's pretty subtle stuff. I hope you stick around. I think we can all learn from your knowedge.

Cheers ... Alan

[mikewren]

Thanks, Alan, that really is too kind!  I tried to keep the geek talk to a minimum, but it's a pretty technical point, but one that I think could be important as offsite backup software (and users) become smarter and smarter.

I have a bit of experience with metadata strategies from the lossless digital audio world, and it's really neat to see the same lessons can be applied to digital photography. The current software tools are so close, but not quite there yet.  I spent this past weekend getting to know iView, and the user interface feels clunky.  The search is fast enough on the 10k items in my current dataset, and the GUI is fast, but it seems like i have to press way too many keys and move the mouse too much to do simple tasks such as sorting, adding metadata and then pushing out my updates back to my files.  Yes, I'm using shortcut keys... but it just "feels" like there's a bit of streamlining that can still happen.

[danaltick]

I wonder if this DNG file handling difference between iView and Bridge could be related to the problem I speak of in this thread http://thedambook.com/smf/index.php?topic=278.0.  Here are the two excerpts taken from the thread about the problem:

• One problem I have noticed with Bridge though is the Metadata panel does not work for any of my DNG's, even though the File Info panels work fine, as well as the Keywords panel.  Also, all the Info information is fine in iView.  The Metadata panel in Bridge just shows as a blank panel with the progress indicator in the lower left corner of the panel continuously swirling.  This also happens with any derivatives I make from these DNG's.  And I've noticed this happens even with no ratings or labels assigned to these files in either Bridge or iView, so I know it's not realated to the label problem discussed in another thread.
• I've found what causes the problem.  If I sync annotations in iView on a DNG, it causes the problem in Bridge.  The DNG does not even have to have any labels or ratings assigned.  All I have to do is import a newly created DNG and sync annotations with export.  Also, the File Info panels look fine in Photoshop.

In case this might be relevant, I'm running under WindowsXP and using DNG's made from Canon CR2's with the Adobe DNG Converter.  This appears to be another case where iView is doing something to the metadata that Bridge doesn't like.  The colored labels is the other one.

Dan

[mikewren]

My RAW files are from a Canon 20D.... I haven't applied colored labels, but apply metadata (non star) which I expect to carry over to iView via DNG.  I'll play with redundant Canon RAW files to see if I can replicate your issues.

I've already confirmed that iView saves metadata in a way different than Bridge, however, in my tests, it hasn't (yet?) been an issue.

It's surprising to me that the Adobe DNG forums are a far less useful than what Peter has set up here.  wink wink, nudge nudge 

[mikewren]

OK, a few more days of playing with Bridge and iView... and the results have ended up less consistant than I first thought... sometimes iView keeps metadata at the head of the file, sometimes it doesn't.  Also, the mirroring software I use, Unison, is sometimes un-smart and transfers/mirrors the whole file, and sometimes it only mirrors the metadata changes... and this is not consistent with where the metadata is stored (head or foot of the file).

The important part is that the metadata is getting shared between Bridge and iView, thanks to DNG... and thanks to Peter for singing the virtues of this format to users, but more importantly, software vendors!

[peterkrogh]

Mike,
Who is that Southpark character anyway?  I keep wanting to say "MMkay?" but I know it's nor Mr. Mackie.

so, on a theoretical plane, what we are doing in creating this archive is to manage a very large set of unchanging image data (the RAW image data) and a much smaller set of constantly changing metadata (both the informational metadata, an the RAW file adjustment data.)  Keeping the changing metadata stored and organized in a catalog is the most efficient way to wrangle it, compared to having the write it back to individual image files, and then to re-archive the image files.

The constant mirroring of the files, just to save small bits of text data really strikes me as inefficient. Personally, I only push metadata back into the files because there is a reason to, like I need to create a derivative file with new metadata embedded, or - much less often - because I need to transfer the file between users, computers or applications and I want it to take the metadata along for the ride.

So maybe this re-synchronization that you are describing is not really needed so often.
Peter

[mikewren]

Mike,
Who is that Southpark character anyway?  I keep wanting to say "MMkay?" but I know it's nor Mr. Mackie.

It's a character I created based on my own likeness from this website.  The SMF portal software resized the avatar and you can't see the beer stein too well.

so, on a theoretical plane, what we are doing in creating this archive is to manage a very large set of unchanging image data (the RAW image data) and a much smaller set of constantly changing metadata (both the informational metadata, an the RAW file adjustment data.)  Keeping the changing metadata stored and organized in a catalog is the most efficient way to wrangle it, compared to having the write it back to individual image files, and then to re-archive the image files.

This is correct.  I do this because I have my home fileserver, and a work fileserver, and would like to keep files on both sides in sync with each other.  This presents some unique hurtles, in DAM software.  I'm finding myself having to manually export metadata in iView when I makes changes at home during the evening, for example... then force an import of metadata at work when I arrive the next morning.

The constant mirroring of the files, just to save small bits of text data really strikes me as inefficient. Personally, I only push metadata back into the files because there is a reason to, like I need to create a derivative file with new metadata embedded, or - much less often - because I need to transfer the file between users, computers or applications and I want it to take the metadata along for the ride.

Yes, it would be inefficent, if the whole file were transferred.  The really cool thing about Unison is that only the parts of the file that change are transferred.  For DNG files, this works better than I expected.  The image data is very rarely transferred, while the metadata is the only part of the file that makes the trip through the ether.  Unison computes the differences, and can mirror the differences in 1GB of updated DNG files in under 20 minutes over my home cable Internet connection (40KB upstream)... every night Unison checks 60GB in my image archive.

My issue, I think, is actually one of workflow and software integration.  How to allow my updated metadata to seamlessly mirror between two physical locations (this part I've already figured out) and then how to easily get iView to recognize these updates (more difficult).

My previous workflow was to have a two sets of identical DVD's, one at home and one at work, but it wasn't until I read your DAM book that I realized that digital image files are not static, but dynamic.  Metadata that is more fluid and easy to change is much more useful over time... this is an aspect I completely underestimated.

Anyway, needless to say, my current metadata mirroring solution is a less than ideal.  I'm too lazy and forgetful to tote an external harddisk to and from work.  Have you come across alternative solutions in your travels, or maybe a different way of configuring iView to recognize these external metadata changes.

Thanks as always.


EDIT: I had a brainstorm this morning on the way into work: Maybe the solution would be to mirror the iView cache/catalog file?  I'm not sure of the structure of file, but perhaps Unison would do an efficent job of mirroring updates without having to transfer the whole file.  Then, whenever I open iView at home or work, it would open the most recent catalog.  The downside of course is that I would be opening the catalog across the 100mbit LAN.

FYI: I have Unison scripted in a batch file that is set to run at 2am every morning... the longest I've seen it run was it's first mirroring run, and that only went until 5am.



EDIT #2: Maybe I should just suck it up and use an external drive like everyone else.    Although, I've never been one to believe in KISS for any particular scenario until Murphy has his way with me...