Hierarchical Keywords: A summary of Findings

[Mike-D]

I finally had the chance to look into keywording last night.

Basically, I currently don't keyword anything.  But I want to.  And I want to in a way that does not make me dependent on the BR database, which I expect would tank the day after I finished keywording my sh*t.  So I went poking around, looking for a solution that would

    * allow me to store a master list of keywords in the hierarchy desired
    * be able to convert those keywords into the format needed for each program automatically
    * once a keyword is applied to an image in program A, that keyword should also show up applied to that image in program B. This is the toughie.
    * it would also be nice to be able to somehow export the keywords into the header of a finished image, so that they are at least transportable so that, in 4 years when you no longer use program X for keywording, all is not lost.

Playing around with this a bit more, I have discovered the following.

Lightroom 2 and Bridge CS4

    * LR and BR support hierarchical keywords
    * The hierarchical structure of assigned keywords can be exported when a RAW files is exported from LR, and will show up in BR. However the keywords exported along with a LR export are italicized in BR to indicate that they are transient and not part of the BR keyword set. They can be made Persistent by right-clicking the keyword and selecting "Make Persistent."
    * The keywords are actually stored in a special section of the JPEG header (which you can see on a Mac if you run more on a JPEG file that has keywords assigned) (if you don't know what that means, don't worry about it). The section is known as XMP Data, and the part in questions is called "<lr:hierarchicalSubject>" and the keywords are stored as a | delimited UTF-8 string, like this:

Code:

     
         <lr:hierarchicalSubject>
          <rdf:Bag>
           <rdf:li>Product|Album</rdf:li>
           <rdf:li>Venues|National Art Gallery|Cafe de Beaux Arts</rdf:li>
          </rdf:Bag>
         </lr:hierarchicalSubject>

     
      The "lr" part implies that this is an Adobe-ism that almost certainly will not translate to other non-Adobe products. This section can be manipulated using ExifTool.

    * Because of the above, the hierarchical structure survives:
      -a file name change outside of bridge;
      -a file copy outside of bridge;
      -a copy to a windows machine and back again (I was concerned that the keyword structure may have been kept in the data stream file on a Mac--not the case);
      -opening the file in PS and doing a "Save As" to a new file.
    * LR2 and BR CS4 export keywords in the same format: flat text file, using tabs to indicate the hierarchy.  This implies that a single source file can be maintained and imported as required into both apps.
    * When LR and BR assign keywords, they are also added to the IPTC keyword list; IPTC is structureless however, and so the hierarchy is lost.



PhotoMechanic 4.5

    * Any keywords assigned either by BR or by LR on an exported file are ignored by PM.
    * For the life of me I cannot find a way to display keywords assigned to an image in PM
    * The PM keywords are also stored in the IPTC header, but this time the hierarchy is lost.
    * Keywords assigned by PM show up in BR, but the hierarchy is lost. This is almost certainly due to the fact that PM stores keywords in the IPTC "Keyword" section only. This is also why keywords assigned in PM show up in BR under "Other Keywords."
    * The exported keyword list file from PM is the same as that of BR and LR.
    * I made a request to Kirk at Camera Bits about hierarchical keywords, and after we spoke today he is strongly considering support for the <lr:hierarchicalSubject> in the upcoming 4.6 release, so good news on that front.


I don't use any other products, so I have not tested anything else.

Keyword files
In general, the keyword files are based on a loose standard that is structured like this:

Code:

[Mike's Keywords]
[2009-01-02]
Cakes
<tab>{Cake}
Florists
<tab>{Florist}
Venues
<tab>{Venue}
<tab>National Art Gallery
<tab><tab>Cafe de Beaux Arts
<tab>Rockcliffe Pavillion
<tab><tab>{Rockcliffe Park}
Product
<tab>Album
<tab><tab>Magazine
<tab><tab>Matted
<tab>Canvas

Items in [square braces] are ignored and treated as comments
Items in {curly braces} are synonyms for the containing keyword
<tab> is the tab whitespace character, not the actual symbols for the string <tab>

So going forward, it looks like maintaining a synchronized keyword list is doable, since BR LR and PM can all import the same type of file.

Sources
http://www.adobe.com/products/xmp/
http://www.lightroomforums.net/archive/index.php?t-3422.html
http://kb.adobe.com/selfservice/viewContent.do?externalId=kb402660&sliceId=1
http://www.adobe.com/devnet/xmp/
http://it.toolbox.com/blogs/webdev-for-socialmedia/xmp-extensible-metadata-platform-27301
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html

[johnbeardy]

I think you can copy the whole LR keywords structure to Bridge and vice versa. In LR, it's Metadata > Export Keywords while in Bridge there's a little menu on the Keywords panel with Import/Export options. So you could maintain a persistent list in Bridge.

John

[ianw]

The "lr" part implies that this is an Adobe-ism that almost certainly will not translate to other non-Adobe products.

Not true!  The "lr" part is called a namespace.  It is a way of hardening the meaning of the data in this XML extract.  Somewhere at the top of the document will be a namespace declaration including xmlns:lr="http://www.adobe.com/...".  I don't have Lightroom so can't fill in all the gaps.

The power of the name space is that it allows you to combine data definitions from different sources.  For example your extract shows another namespace called "rdf".  This is the 'resource description framework', a published standard on how to describe data.  You will see another common namespace called "dc" which stands for "Dublin Core" (after a conference in Dublin, Ohio and not Dublin, Eire).  The whole of this data is wrapped in a namespace called "xmpmeta" which is Adobe's.  You may see other namespaces including "tiff", "exif" or"crs" (camera raw settings) if you examine closely.

A problem if you take data from different sources is that a data definition name can be reused.  Lightroom uses a data item name of "hierarchicalSubject" to store these details.  What if Expression Media used the same name, but then maybe used a different delimiter?  How would each application cope.  The answer is to use a namespace.  Lightroom uses "lr", Expression Media uses a namespace of "expressionmedia".  Now both applications can use a data item called "hierarchicalSubject" safely when preceeded by the appropriate namespace.  A better example could be a data item called "table".  Is this a table on a web page or a table, with chairs?  The anwswer is either or neither - it doesn't matter as long as namespaces are used.

As long as an application is namespace aware then it could use this data.  There's nothing to stop Lightroom reading Expression Media data from an image or vice versa.  Bridge is obviously aware of this Lightroom namespace and vice versa if they can handle each others hierarchical keywords.  Of course does Microsoft want to be Lightroom aware?  Possibly, but then they might not want the reverse scenario!  While being Exif aware is not an issue it might not be the same for a direct competitor.  The only solution here is an internationally agreed standard for meta-data that both applications use.

One downside of namespaces is that they can complicate things if not handled correctly.  I've not fully tested this but if you have an image that was annotated in iView Media Pro and then sync'd to the image then iView specific stuff is against a "mediapro" namespace.  If you further annotate this in Expression Media and sync again you get all the data against the "expressionmedia" namespace, but with the iView stuff left there.  If there's no way of removing this, other than maybe syncing it out using iView then there are potentially serious privacy issues here.

While I'm fairly proficient in XML it is always the namespace issues that cause me problems, so hopefully I've not confused you too much!

Ian

[johnbeardy]

Thanks Ian, I was watching the footy on the telly when I replied and overlooked the Adobe-ism error. I explain this like cellphone codes - 07973 may be an Orange-ism, but Vodafone or Verizon can still call it.

EM writes both EM and iView namespaces. The problem is when you roundtrip because LR and Br both show keywords hierarchically, but write hierarchical and flat keywords for compatability. EM, naturally enough, reads and writes both, so it's pretty easy for LR/BR to end up with the same keywords in both the hierarchy and the base level. But that's not my problem - I'm a leveller

John

[Mike-D]

I made no error. Please note that I did not say that other programs cannot use it.  What I said was that it was an Adobe-ism -- which it is -- and that other programs almost certainly can't use it -- which they can't, until they implement support for it. That's not to say that they can't implement such support, simply that no other product that I am aware of other than Adobe products makes use of it.  I'm hopeful this will change shortly with PhotoMechanic 4.6

...Mike

[peterkrogh]

Mike,
Expression Media 2 supports HK.
(I'd guess idImager does as well, but have not checked).
There are a number of Adobe-defined XMP namespaces that are very well supported by other programs.
If PM does not now support it (I haven't checked) I'd expect it to be coming before long.  They are responsive to changes like this.
As you test, make sure that you have the PM preferences set right - there is a huge amount of flexibility in that program, but that can mean that not everything maps the way you expect it to.
Peter