Standardization of adjustment metadata?

[Marc Sabatella]

I realize that DNG allows for embedding a previewing, and that's a great thing, but I'm concerned here about the adjustments used in creating that preview.  While in theory the format allows the settings for the various adjustments used to create the preview to be saved as XMP data within the file, I'm wondering how much standardization there is in this regard.  My perception is that every RAW processing program I have seen has slightly different controls, and indeed produces slightly different results for similar settings of similarly-named controls.  So I am not really confident any other given program would be correctly interpreting the exposure, sharpening, or other settings made by another program.  For example, if I initially use ACR to do my adjustments and then generate the DNG file, but later on I decide I want to work on the DNG file with RawShooter, is it reasonable to assume that the latter would render the file pretty much exactly the same way as ACR did given the settings stored in the DNG file?  If not, it seems to me that some of the advantage of the "openness" of the DNG format is not as great as it would first appear: all the work that goes into making initial adjustments would be lost (except for seeing a preview) if you changed software later.

On a related note, it isn't clear to me if the Adobe DNG converter listens to adjustments made by other programs, or if it only knows about ACR's adjustments.  Obviously, ACR must be passing metadata information through to the converter to embed in the file, perhaps through sidecar files.  Is this interface documented, so other programs can in theory do the same when they invoke the DNG converter?  That is, if I make my initial adjustments with SilkyPix instead of ACR, and then invoke the DNG converter from SilkyPix, is the converter going to be able to understand the adjustments made within SilkyPix (assuming SilkyPix actually tries to communicate them)?  If not, it seems even the embedded preview feature would only be useful if you do your initial adjustments with ACR.

Don't get me wrong, I'm pretty well sold on DNG in general, if for no other reason than they can be compressed whereas my Pentax RAW files are not, and of course just being assured that the files always will be readable at all is comforting.  I'm still trying to sort out the extent to which I really need to use Adobe products in order to realize the full benefit of the DAM principles.  Elsewhere, I had asked about all-in-one solutions like ACDSee Pro, and I appreciated the responses from Peter on that thread.  But even assuming I don't go that route, I'm not convinced Adobe Elements would be as strong a solution for me as using idImager, possibly combined with RawShooter or even the Pentax-supplied RAW software I already have.  If I have to use Adobe software in order to gain the supposed interoperability advantages of DNG/XMP, doesn't that seem a little contradictory?

[johnbeardy]

One important aspect you don't mention is the ability of various cataloguing programs to save metadata to the DNG, and for that information to continue with any derivatives. By saving you time managing the sea of pixels, you can then spend more time on their output quality.

But you are mainly talkiing about the raw processing side of the format and you make some fair points. So far, as far as I know, no other software program is saving its raw adjustments to the XMP metadata, writing its preview to the DNG file, or reading ACR adjustments and applying them as part of its own conversion routines, and potentially comparing the results against the Adobe-embedded preview. But don't forget, it's early days for DNG. There's nothing stopping other raw converters from doing so, and it'd be perfectly reasonable for users to request they do. Right now it's a bit of a guess, but I'd expect some competing programs will both exploit the XMP ACR data and store their own. Why not?

John

[Marc Sabatella]

One important aspect you don't mention is the ability of various cataloguing programs to save metadata to the DNG

Right.  Like I said, I'm sold on DNG in general.  Being able to save keyword and other standardized metadata in the file itself is indeed an important advantage I neglected to mention.  Here's where I have complete faith that even programs that don't *currently* support this will do so in the very near future, because it strikes me as a no-brainer.  Obviously a good thing to do, obviously technically feasible.

you are mainly talkiing about the raw processing side of the format and you make some fair points. So far, as far as I know, no other software program is saving its raw adjustments to the XMP metadata, writing its preview to the DNG file, or reading ACR adjustments and applying them as part of its own conversion routines, and potentially comparing the results against the Adobe-embedded preview. But don't forget, it's early days for DNG. There's nothing stopping other raw converters from doing so, and it'd be perfectly reasonable for users to request they do.

But that would be significant to me only if the format actually supports a *standardized* way of doing these things, so metadata written in one program can actually be understood and correctly intepreted by another.  Writing a preview seems straightforward, so I'm sure you're right that others will jump on board here - applications should be able to write previews directly into a DNG file whether or not there is a published spec on how to pass adjustment data into the Adobe DNG converter, and obviously they can read them.  I'm more concerned about recording the adjustments themselves, and in particular, about *communicating* those adjustments between applications.  If there is no standard for how this data is stored and interpreted, then storing this data in the DNG file seems no better to me than storing it in sidecar files or a database.

For example, it seems it would be *possible* to define a standard for how "curves" should behave - how many points should be settable, how their X&Y coordinates should be expressed in the XMP metadata, what shape the curve should take between set points, exactly how a given X,Y value on the curve should be interpreted in a given color space, etc.  But based on my previous experience participating in the development of technical standardizations, and based on the fact that I've never seen any mention of these issues in any discussion of DNG, it seems quite likely to me that such standardization does *not* yet exist.  Furthermore, it seems likely to me there would be some amount of resistance to such standardization from the various companies involved with RAW processing.  Everyone wants to be able to claim they do things "better" than the competition, and in the context of actually producing an image, a "better" way of interpreting the same metadata can only mean "different from the standard".  The usual result of these conflicts is for the standard to specify only behavior for some sort of "least common denominator", which in this case might mean, for example, only color temperature/tint, a single value for sharpness, brightness, and contrast, and for each application to implement other features (eg, curves, unsharp mask settings) in incompatible ways.  Now, I realize DNG is completely under control of Adobe - it isn't the result of the usual IEEE standardization processes.  But I still have this suspicion that this is not a solved problem.

This is why the overal tone of my initial post was one of skepticism.  Again, not skepticism as to the value of the format in general, but skepticism over the extent to which it really addresses all of the interoperability problems one might encounter in migrating between software packages.  Regarding adjustments, all I can tell for sure is that *if* you use ACR, then your adjustments will be preserved as you upgrade to new versions of ACR.  But that should be the case for *any* RAW processing software that allows saving this information at all, whether in sidecar files or in a database.  The situation appears a little more promising regarding the preview - like I said, even if other programs currently are not capable of writing such previews, it seems likely to me this capability will show up soon enough.  But I'm less concerned about the preview itself than the work that went into creating it.  And again my reason for concern isn't in deciding whether to use DNG or not - that's a done decision for me - but in deciding just what capabilities I really need from the software I am trying to choose.

[peterkrogh]

Marc,
The reality is that RAW file processing will be continually improved and that no two converters will render RAW files the same way.  Improvements like Lightzone's adjustment layer functionality is just not going to translate from one application to another, and that is the way it is and should be. 

An application could make a rough translator for adjustments made in other applications, that would provide you a starting point for correction, but it simply won't render in the same way, especially as these applications get better and better.

What we do need, however, is some kind of predictable rendering, after we have made decisions on how the image should be displayed.  The embedded preview does that, and to a much better effect than you probably realize.  One of the things I do in my lectures is to show 24 inch prints made directly from this embedded preview.  Most professional photographers cannot tell which print is made from the embedded preview, and which is made from a 40 MB TIFF.  So the embedded preview is the rendered version of the file.

At the moment, only Adobe software creates the embedded preview, mostly, I think, because the other companies did not realize how important it would be. I would expect that to change before too long, and every RAW file converter that supports DNG should also write a preview to the file that shows the adjustments you have made.

The problems of predictable rendering are common to all RAW files, but in most cases, there is no solution to the predictable rendering issue, unless you use manufacturer's software.  DNG provides for this, and that is, for me, the most important reason to use it.

Peter

[Marc Sabatella]

The reality is that RAW file processing will be continually improved and that no two converters will render RAW files the same way.

Thanks for confirming my suspicion.

What we do need, however, is some kind of predictable rendering, after we have made decisions on how the image should be displayed.  The embedded preview does that, and to a much better effect than you probably realize.  One of the things I do in my lectures is to show 24 inch prints made directly from this embedded preview.  Most professional photographers cannot tell which print is made from the embedded preview, and which is made from a 40 MB TIFF.

That is indeed impressive, and it makes me feel better about the viability of using that preview for something other than thumbnail display.  I'm not sure it directly addresses my concern about differences in rendering between programs, except by suggesting I wouldn't actually *need* to ever render mosts file a second time.  Interesting.