Writing star ratings to NEFs

[peterkrogh]

Matthew,
Thanks for clearing that up.  Sounds like a reasonable practice.
Peter

[matthewjheaney]

This sounds very cool.  Can I get a copy to toy with?

I emailed you a zip file containg the COM server (nefhelp.dll) last night.  Let me know if you received it.

-Matt

[peterkrogh]

I did not.
And I don't think I'd have time to look at it. I hit a completion milestone turning in the book, and I'll be totally slammed for the next couple weeks getting the layout edits done.
Peter

[matthewjheaney]

And I don't think I'd have time to look at it. I hit a completion milestone turning in the book, and I'll be totally slammed for the next couple weeks getting the layout edits done.

Oh, sorry -- I was answering Chris's post.  I keep forgetting that all posts go to the end of the queue. Too used to threaded email readers, I guess...

-Matt

P.S. New ed. for The DAM Book?  That would be great -- I read the 1st ed. three times and learned much from it.

[peterkrogh]

Matt,
Yes, a total rewrite - 200 additional pages - been a long 8 months...
Peter

[matthewjheaney]

At some point I should probably write a proper NEF file handler and incorporate that into the toolkit, but this method works if all you want to do is manipulate the ratings value.

An NEF file is just a TIFF file.  The XMP Toolkit SDK can read and write XMP packets in TIFF files, so feature I wanted to test was to attempt to use the built-in TIFF file handler to manipulate the XMP packet embedded in a NEF file.

Out-of-the-box, the XMP Toolkit SDK does not read NEF files, because there is a test in XMPFiles to explicitly disable support for known camera raw files.  So I simply disabled that check in the sources, and now I'm able to read and write XMP packets in NEF files too, using just the TIFF handler in the XMP Toolkit SDK.  Importantly, I'm now able to expand a NEF file to add an XMP packat that didn't already exist (which, among other things, is where the rating is stored), which removes the limitation of the COM wrapper I wrote for manipulating embedded rating values.

If anyone is interested in knowing what I changed in the XMP Toolkit SDK, let me know and I'll send you a patch file.  If anyone wants the COM wrapper object for reading and writing XMP packets in NEF files, drop me a line and I'll send it to you.

-Matt

[peterkrogh]

Matt,
You might be interested in this.
We've just tracked down an issue in Photoshop where a a blank XMP tag - IPTC Date Created - will stop Photoshop from parsing all metadata in a file.

Just a caution that stuff that should not cause problems sometimes does...
Peter

[matthewjheaney]

We've just tracked down an issue in Photoshop where a a blank XMP tag - IPTC Date Created - will stop Photoshop from parsing all metadata in a file.

Thanks for the heads-up.  I assume Photoshop is using some version of the XMP Toolkit, so that probably means there's a bug in there somewhere.  (There was probably a parse error, and the toolkit simply threw an exception.)

I did modify my copy of the toolkit to allow NEF files to be handled as TIFF files.  This worked, but there were a couple of issues.  When the NEF file didn't have an embedded XMP packet, the file handler would report that it did found an XMP packet (but with "unknown" offset and length).  I think that's because "getting XMP" in the XMP toolkit is interpreted more broadly than getting just the (raw) XMP packet; really it means "get me all of the metdata," but that's not what I want.  (Of course, I could be interpreting the return value incorrectly -- perhaps the intended mechanism is that if the value of the raw XMP returned has zero-length, then that's the toolkit's way of saying that there's now raw XMP.)

The other issue is that the TIFF file handler attempts to "reconcile" metadata in different parts of the file so it's all the same.  This is normally a good thing, but unfortunately I couldn't get the TIFF handler to just write the XMP packet I specified; it would write a /tiff:NativeDigest and an /exif:NativeDigest too.  This is harmless, but my goal was to attempt match the behavior of Nikon ViewNX.

So I ended up making a custom NEF file handler based on sources copied directly from the TIFF handler.  What the NEF file handler does differently from the TIFF file handler is to not reconcile any metadata.  It simply reads any XMP there, changes the xmp:Rating, and then writes it back out.  This appears to be working fine.

It's too bad Nikon and Adobe had to get into a pissing contest about NEF intellectual property, since the ultimate losers are us poor photographers.  Nikon wants to keep their NEF file format a secret, but then Adobe retaliated by deliberately crippling the XMP Toolkit SDK to not support NEF (or any other known camera raw file) files.  But to add an xmp:Rating to the NEF file (as ViewNX does), I can't use the NEF SDK because that API doesn't support file expansion (or indeed any other form of XMP manipluation).  So that means me, and every other developer of metadata-handling apps (including Adobe!), has to write our own code for TIFF file expansion.  This is really, really dumb.

It would be helpful if Nikon would make peace with Adobe and open the spec for NEF files, but control the standard the same way Adobe controls the TIFF and FLV standards (both of which are now fully open).  Then Adobe could officially incorporate NEF support into the XMP Toolkit SDK.  As it stands now, every tool author must make the same custom hacks to the XMP Toolkit.  This doesn't help Nikon or anyone else. 

Just look at all the headaches people have been having attempting to use EM to sync annotations back to NEF files.  The iView guys got it wrong, and now Microsoft has it wrong -- but what did Nikon expect if neither the XMP Toolkit SDK nor the NEF SDK supports XMP manipulation of NEF flies?  Without support from a standard API, then everyone is going to do it a little differently, and probably get it wrong anyway.

My advice to Nikon is to open the NEF file format, and then lobby Adobe to add NEF support to the XMP Toolkit SDK.    If nothing else, Nkon should customize the XMP Toolkit API to add NEF support and then bundle that with their NEF SDK.  This will allow developers (like me) to get it right when writing metadata authoring tools, which will enable users (like the OP) to have a seemless workflow experience.

To Adobe's credit they created the XMP standard, and provided a nice, easy-to-use API to manipulate XMP.  In principle this should be enough to create the infrastructure necessary to ensure all the metadata gets carried along correctly by all tools in the chain.  But in practice this depends on vendors being willing to allow Adobe's API to be used with their (proprietary) camera raw files -- but not all vendors are willing to do that.

Memo to Nikon: this is not helping you, or your customers.

-Matt

[matthewjheaney]

So I ended up making a custom NEF file handler based on sources copied directly from the TIFF handler.  What the NEF file handler does differently from the TIFF file handler is to not reconcile any metadata.  It simply reads any XMP there, changes the xmp:Rating, and then writes it back out.  This appears to be working fine.

I have uploaded the latest version of the NEF Help tool to my website:

http://home.earthlink.net/~matthewjheaney/nefhelp-1.0.1.0.zip

This allows you to read and write the xmp:Rating value (expanding the NEF file, if necessary). 

There are also methods for reading all of the metadata in the file, or just the embedded XMP packet.  Having the metadata in XML from allows you to use the DOM to parse it -- so this essentially duplicates the features available in exiftool.

The README.TXT file explains how to use the API from a script.

-Matt

[matthewjheaney]

I have uploaded the latest version of the NEF Help tool to my website:

I have just uploaded a minor update.  This adds a method to dump just the XMP packet to standard output:

http://home.earthlink.net/~matthewjheaney/damtools/nefhelp-1.0.2.0.zip

-Matt

[cdubea]

Matt,

Thank you for all your work.  Yes I got the files and will have to hack up an XM script to write the star ratings to the NEFs.

[matthewjheaney]

Thank you for all your work.  Yes I got the files and will have to hack up an XM script to write the star ratings to the NEFs.

OK, let me know how you make out.

As you've already found out, xMedia writes the classic IIM metadata to the image resource block when you sync annotations back to the file (that's why the ratings didn't get sync'd -- because the rating value is only stored in the XMP).  What I'd like to be able to do is sync the newer, IPTC core metadata to the XMP block -- and not write the image resource block (IIM metadata) at all.  For that reason, I'll have a new release soon that will allow you to sync all of the embedded XMP (not just the xmp:Rating value).

Eventually, I'd also like to provide a COM wrapper to the XMP Toolkit SDK, so you won't have to depend on what I happen to provide in NEFHelp and friends, or what xMedia happens to provide, or indeed what any DAM happens to provide.  Most DAM apps or browsers use some version of the XMP Toolkit to write metadata, so there's no real reason you shouldn't be able to too.

I have some other DAM tools that you might find useful that I'm preparing for upload to my website; I'll post again when they're ready.

-Matt

[matthewjheaney]

I have some other DAM tools that you might find useful that I'm preparing for upload to my website; I'll post again when they're ready.

I just posted v1.0.3 of the NEF Help tool.  There were no interface changes, but some error handling was improved.  Also, the zip file now includes a couple of Expression Media scripts that use the NEF Help object to read and write ratings from the image files.

The scripts use a progress dialog to monitor execution of the script, and to allow the user to cancel the script.  The progress dialog object is pretty much necessary when using Expression Media, because that application does not pump Windows messages during long-running scripts, which pretty much locks up the computer.  Version 1.0.1 of the progress dialog tool is available at my website.

http://home.earthlink.net/~matthewjheaney/damtools/

-Matt

keywords: XMP, NEF, rating, xmp:Rating, Expression Media

[matthewjheaney]

I just uploaded the latest version of my NEF Helper tool (v1.0.4) and Expression Media scripts package (v1.0.1), which now support synchronization of label values between Expression Media other apps such as Nikon CaptureNX2 and ViewNX.

To provide label support in the Expression Media script, a small configuration step is necessary to specify the mapping function from labels to integers; the README has the details.

http://home.earthlink.net/~matthewjheaney/damtools/index.html

-Matt

keywords: NEF, XMP, metadata, rating, xmp:Rating, xap:Rating, label, xmp:Label, xap:Label, synchronization

[matthewjheaney]

I have just released v1.1 of the NEF Helper tool.  It now supports reading and writing the photoshop:Urgency field.  This means you can now fully synchronize labels (and ratings) between Expression Media (or any scriptable DAM tool that runs on Windows) and Nikon tools (ViewNX, Capture NX2, etc).  It works for NEF files, JPG files, and for any other file type supported by the Adobe XMP Toolkit.

http://home.earthlink.net/~matthewjheaney/damtools/index.html


You can also use it to dump the XMP packet.  You can get the XMP data and print it in its serialized form, as raw XML:

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:xmp="http://ns.adobe.com/xap/1.0/">
         <xmp:Rating>4</xmp:Rating>
         <xmp:Label>mellow yellow</xmp:Label>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:MicrosoftPhoto="http://ns.microsoft.com/photo/1.0">
         <MicrosoftPhoto:Rating>75</MicrosoftPhoto:Rating>
      </rdf:Description>
      <rdf:Description rdf:about=""
            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
         <photoshop:Urgency>0</photoshop:Urgency>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>


You can also display the output of the DumpObject method in the XMP Toolkit, which has a more human-friendly format:

Dumping XMPMeta object ""  (0x0)

   xmp:  http://ns.adobe.com/xap/1.0/  (0x80000000 : schema)
      xmp:Rating = "4"
      xmp:Label = "mellow yellow"

   MicrosoftPhoto:  http://ns.microsoft.com/photo/1.0  (0x80000000 : schema)
      MicrosoftPhoto:Rating = "75"

   photoshop:  http://ns.adobe.com/photoshop/1.0/  (0x80000000 : schema)
      photoshop:Urgency = "0"


This release can also print a hex dump of the raw XMP packet, header and all, with each byte of the internal UTF-8 byte stream printed as an ANSI character.  This is useful for examining the encoding of Unicode characters that don't have a 7-bit representation (the copyright symbol is the canonical example):

0000: 3C 3F 78 70 61 63 6B 65  74 20 62 65 67 69 6E 3D  <?xpacke  t begin=
0010: 22 EF BB BF 22 20 69 64  3D 22 57 35 4D 30 4D 70  "" id  ="W5M0Mp
0020: 43 65 68 69 48 7A 72 65  53 7A 4E 54 63 7A 6B 63  CehiHzre  SzNTczkc
0030: 39 64 22 3F 3E 0A 3C 78  3A 78 6D 70 6D 65 74 61  9d"?>.<x  :xmpmeta
0040: 20 78 6D 6C 6E 73 3A 78  3D 22 61 64 6F 62 65 3A   xmlns:x  ="adobe:
0050: 6E 73 3A 6D 65 74 61 2F  22 20 78 3A 78 6D 70 74  ns:meta/  " x:xmpt
0060: 6B 3D 22 58 4D 50 20 43  6F 72 65 20 34 2E 34 2E  k="XMP C  ore 4.4.
0070: 30 22 3E 0A 20 3C 72 64  66 3A 52 44 46 20 78 6D  0">. <rd  f:RDF xm
0080: 6C 6E 73 3A 72 64 66 3D  22 68 74 74 70 3A 2F 2F  lns:rdf=  "http://
0090: 77 77 77 2E 77 33 2E 6F  72 67 2F 31 39 39 39 2F  www.w3.o  rg/1999/
00A0: 30 32 2F 32 32 2D 72 64  66 2D 73 79 6E 74 61 78  02/22-rd  f-syntax

Drop me a line if you have any comments or questions.

-Matt