DNG conversion: a most basic question

[frankgindc]

If a raw file is converted to DNG (say, without the original raw embedded), would that in some way limit the ability of various DNG/RAW converters to render the file, er, uniquely?  That is, would the RAW already be rendered at that stage so that the various converters would be basically "reading" and providing the user with the same thing -- or would there still be room for interpretation between the converters?

For example, if a given 3rd party RAW converter currently is better at, say, highlight recovery or rendering accurate reds than another converter, would the first converter lose some of its advantage in this area if both converters were working on an already converted DNG?

Hope this question makes sense.  I'm just wondering aloud if this might be one reason that 3rd party converter might be reluctant to adopt DNG.

Frank

[danaltick]

Frank,

Peter may need to confirm this, but my understanding is that the proprietary Raw data goes through a lossless compression when converted to the DNG format.  Because it is lossless there should be no impact on the individual Raw editors' capabilities.

Dan

[frankgindc]

But then it seems that there wouldn't be a case where a RAW converter can convert a DNG but not the RAW that it was derived from.  What I mean is not whether or not the functions or "sliders" of the RAW converter would be dimished (or whether there'd be a loss in resolution or data) but whether or not the DNG process does some of the decoding in the process and would then eliminate some variation.  You know the ongoing debates about which 3rd party RAW conversion is better at decoding the Canon/Nikon code...would that become partly a moot point with the DNG?  I was thinking that the DNG process might actually use ACR to create the DNG.

Frank

[danaltick]

Frank,

If I understand you correctly, I really don't know what happens to the metadata imaging instructions when they get converted from a proprietary raw format and embedded in the DNG file.  My guess is the Adobe DNG converter is using the ACR instructions as is, but it may have to do a conversion on other Raw editors' instructions that may cause a slight variation when those instructions are used by the third party editor when editing the DNG file (assuming the third party editor supports DNG), but you can be assured that the Raw data itself is preserved.

Dan

[frankgindc]

Thanks for bearing with me, Dan. 

I might misunderstand what a DNG does, but my understanding is that if you do not include the RAW file and are using a RAW converter that works with DNGs, then the converter (or other imaging programs further downstream) would be reading the data that has already been interpretted (in some way) by the DNG rendering process.   So rather than Bibble, Lightroom, DPP, CaptureOne, or Aperture using their own wizardry to interpret the orginal contents of the RAW file from the camera, with varying degrees of success under different conditions, with a DNG each program would instead be working with a DNG translation.   

Each program might carry that translation forward differently, but I'm thinking that there would be much narrower parameters since they all would be following the data the way its mapped out in the DNG standard (e.g., all converters know that under the DNG standard that some data, say, 10010100 = Red # 5, or whatever).   I'm thinking that this would be different than the greater variation that happens when RAW converters work directly from a camera RAW file (e.g,  Bibble interprets a CR2 file where 10010100 = Red.0001, ACR would say that it's Red.0002, yada yada).

Not to get too far down into the weeds (perhaps it's too late) but there are two potential implications for my totally nerding out on this:

1.  When considering why a 3rd party RAW converter might not want to adopt DNG, if the above is true then it may be understandable if that company thinks they have a leg up on the competition in their ability to interpret RAW files from the camera. 

2.  If the DNG process does standardize the rendering to some degree, then which RAW converter program one uses becomes far less important in determining how images will be interpretted; instead a user would look to program features/slider/DAM/plugins etc. rather than the converter engine.

Well, such are the dangers of thinking out loud.  But do you see what I mean?

Frank

 

[frankgindc]

To break it down, I guess my basic question is this:

When you convert a RAW file to a DNG, do future rendtiions of that DNG file look to the original camera RAW data - which is now merely stored in a new format and package -- or are they looking to the DNG process's interpretation of that RAW data?

Frank

[danaltick]

Frank,

I think the confusion lies in your perception that the DNG converter translates the proprietary Raw data into a so-called DNG format.   To my knowledge, this is not what happens.  Instead, the converter merely performs a lossless compression on the data before embedding it into the DNG file.  Only the interface to that data is what gets translated, and that interface is publicly documented for third party application vendors to use.  So for example, if you were to use Bibble to edit a DNG file, it would first decompress the raw data, returning it to its original state before applying the rendering instructions.  So as far as Bibble is concerned, once it has loaded and decompressed the data, it is now transparent to the editor that the data was ever stored in a DNG job jacket; make sense?

Dan

[Ken]

Looking at the question from a slightly different angle.  Since each manufacturer dos not share there how they interpret their RAW files, I guess I see a related question as, "Is it easier for a 3rd party software company to read a RAW file or a DNG file to render the user an image?"  This may provide a bit of insight (notice I said only a bit) as to why developers may or may not prefer one file format to the other.

--Ken

[peterkrogh]

Frank,
A DNG is a repackaging of the raw data, plus some other stuff.

A DNG contains:

• The entire raw *image* as the camera recorded it into the raw *file* (Two different things)
• The metadata that is stored in the raw file, moved over a s a block.  This data comes in whether or not Adobe software can understand or make use of it - it's simply transfered.
• Any XMP data that is in the sidecar of the file, stored as an XMP block in the DNG  (this includes informational metadata, like IPTC, rendering instructions like ACR settings, as well as a duplication of any EXIF or other metadata that can be parsed out into XMP, which makes it more universally readable).
• A Fixed Rendering of the image, in the form of the embedded JPEG file created according to the user's settings.

Any raw converter that does not work with DNGs is in that state because the software manufacturer has decided not to support the format, not because there is any technical or proprietary barrier to doing so.  There's noting stopping any manufacturer from making their software offer the same range of renderings with a DNG that they can with the original Raw.  All the data is there. It's my understanding that this is what what Apple does in Aperture.

There is a small bit of additional engineering needed to read private makernotes out of the DNG file, since they have been moved over as a block to a new place.  I believe that with older cameras, there were some issues with how this data was copied over to the DNG, due to the use of relative offsets in the file format.  I think camera companies now see relative offsets as bad software design, and no longer do it on shipping models.

Hope that clears it up.
Peter

[frankgindc]

Peter,

One part that I don't understand is this:   If the DNG process doesn't do some portion of the rendering (using ACR or something) -- and the RAW data remains unchanged and uninterpretted -- then why would a RAW converter that is unable to render a raw image from a particular camera be able to render a DNG based on a RAW file from the same camera?  I mean, it seems that if the original RAW data is still what the converter is looking too -- but simply from within a DNG package -- then the converter still wouldn't be able to read it.  So, for example, when a program like Aperture touts that its use of DNG format will enable it to read RAW files from non supported cameras, it seems that this would only be possible b/c the DNG conversion process is walking the RAW file into some common ground that the program understands.   

Frank

[peterkrogh]

Frank,
There's (more than) one thing I forgot to mention.  A DNG also includes a color decoding matrix that lets a program decode the color in a Raw file, even if it has never seen this exact file type before.  You can see this work if you make a DNG from a new camera, and then open it with Photoshop CS2.  CS2 makes use of the matrix in the DNG to decode the raw data, even though it has no idea how to decode the color in a D300 file.

I don't know how many other programs decode color in the same way that Adobe does, and can therefore make use of the embedded matrix. It seems as though Apple is doing this with Apertuure 2.

Peter

[danaltick]

Peter,

Very interesting.  Sounds like a key element in long-term archival.  Prompts the question though; can we expect tomorrow's DNG editors to make full use of today's matrices?  In otherwords, if you have a color representation of the data, is that all you need for future parametric image editing?

Dan

[peterkrogh]

Dan,
I expect that there will be more sophisticated ways to decode color coming all the time.  This is really to be thought of as a fail-safe in case current software disappears at some point in the future.  You'll see some very cool additions to this in future specs, including ways for other software to create and embed their own color translation references tools, and more...

Stay tuned.
Peteer

[Marc Sabatella]

One part that I don't understand is this:   If the DNG process doesn't do some portion of the rendering (using ACR or something) -- and the RAW data remains unchanged and uninterpretted -- then why would a RAW converter that is unable to render a raw image from a particular camera be able to render a DNG based on a RAW file from the same camera?

Many converters have code in them that checks to see what type of camera a RAW image is from, and if it's not on the approved list, it doesn't even *try* to decode the information in the file, because it is afraid it won't know how.  RAW converters assume that they are going to need to be tweaked for each new camera - new cameras might have different formats or different ways the image data needs to be interpreted.  But the reality is, the formats usually don't change significantly between models from a given manufacturer.  Which is why you can sometimes fool converters into dealing with new cameras from known manufacturers by simply using a binary editor to alter the magic field that identifies the type of camera.  Assuming you know enough about the raw format in question to know where this info is, of course, and most of us don't - but you'll often see such hacks posted to forums shortly after release of a new camera.  Eg, "here's a script that will fool RawConverterPro into thinking raw files from the new Canikontex X25 actually came from the old Canikontex X24".  With DNG, converters can take the RAW data at face value since the format is *known* to not change.  Sometimes they are a bit off on color, at least until they actually do learn about the new camera, unless they are able to make use of the color info Peter mentioned.  At least that's my understanding.

Marc Sabatella