iView MediaPro catalog running sloooow

[Mark Weidman]

Ian - I forgot to mention that the html file that was created is 74MB

Mark W.

[ianw]

Mark,

That's not the output expected from my XSL.  You should get an html page such as:

Sorted list of Keywords used in catalog 'Test'

• Arms
• Black
• Blue
• Green
• Legs
• Red
• Test1
• Test2
• White

[/color]
Your output looks like the XML file that iView produces before it does the XSL on it - or one where no XML transformation takes place.  You did hit the "+" button beside the XSL Transform prompt?  If you viewed your HTML in a text editor then it would look something like:

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<?iview mediapro exportversion="3.0" appversion="3.0" ?>

<CatalogType>
 <Catalog pathType = "DOS">Catalog</Catalog>
 <UserFieldList>
  <UserFieldDefinition>Classification</UserFieldDefinition>
  <UserFieldDefinition>Colour</UserFieldDefinition>
 </UserFieldList>
 <MediaItemList>
  <MediaItem>
   <AssetProperties>
    <Filename>iwt-20060516-0001.dng</Filename>
    <Filepath>W:\7 xml\iView Test\2006 05 16\iwt-20060516-0001.dng</Filepath>
    <UniqueID>1</UniqueID>
    <Label>1</Label>
    <Rating>0</Rating>
    <MediaType>DNG </MediaType>
    <FileSize unit="Bytes">9473182</FileSize>
    <Created>2007:01:03 22:48:15</Created>
    <Modified>2008:12:12 22:29:35</Modified>
    <Added>2007:02:01 23:01:17</Added>
    <Annotated>2007:02:01 22:56:00</Annotated>
   </AssetProperties>
   <AnnotationFields>
    <Location>Dover Docks</Location>
    <City>Dover</City>
    <State>Kent</State>
    <Country>United Kingdom</Country>
    <Caption>Leaving Dover Docks on a ferry bound for France</Caption>
    <Keyword>White</Keyword>
   </AnnotationFields>
  </MediaItem>

When viewed in a browser it wouldn't know what to do with the tags, as they are not HTML tags, so just displays the content.  It's best to try it on a small catalogue first, even a test one with just a handful of images.

I still don't have any "performance" suggestions though.  While my biggest catalogue hammers my PC it's still functional.

Ian

[Mark Weidman]

Ian,

I think I finally got it - the XML document is 74MB.  Following is a sample of the metadata from one image.  Not quite sure how I can use this document to tally the keywords for the catalog.  It would require quite a bit of manual counting through this huge document.  Is there an automated way to count the keywords, perhaps in Word or Excel (as you alluded to in a previous message)?

If you or Peter K. is interested in examining the problematic iView catalog, I would be happy to compensate you for it.

Mark Weidman


RAW_061 Yellowstone NP, WY:WEIDMAN_081002_C5R3093.CR2</Filepath>
            <UniqueID>18622</UniqueID>
            <Label>0</Label>
            <Rating>0</Rating>
            <MediaType>CR2 </MediaType>
            <FileSize unit="Bytes">24867785</FileSize>
            <Modified>2009:03:17 14:39:56</Modified>
            <Added>2009:06:16 18:29:37</Added>
            <Annotated>2009:03:17 14:24:28</Annotated>
         </AssetProperties>
         <AnnotationFields>
            <Headline>Bull Elk (Wapiti, Cervus canadensis) in a meadow along the Madison River, Yellowstone National Park, Wyoming, USA</Headline>
            <Product>Bull Elk (Wapiti, Cervus canadensis) in a meadow along the Madison River, Yellowstone National Park, Wyoming, USA</Product>
            <Author>H. Mark Weidman</Author>
            <AuthorTitle>Photographer</AuthorTitle>
            <CreatorURL>www.weidmanphoto.com</CreatorURL>
            <Copyright>© 2008 H. Mark Weidman All Rights Reserved</Copyright>
            <Rights>No usage rights granted without express written permission by the Photographer.</Rights>
            <Location>Yellowstone National Park, Wyoming, USA</Location>
            <State>WY</State>
            <Country>USA</Country>
            <Caption>Bull Elk (Wapiti, Cervus canadensis) in a meadow along the Madison River, Yellowstone National Park, Wyoming, USA</Caption>
            <Keyword>Yellowstone National Park</Keyword>
            <Keyword>Wyoming</Keyword>
            <Keyword>USA</Keyword>
            <Keyword>park</Keyword>
            <Keyword>nature</Keyword>
            <Keyword>landscape</Keyword>
            <Keyword>scenic</Keyword>
            <Keyword>natural</Keyword>
            <Keyword>west</Keyword>
            <Keyword>western</Keyword>
            <Keyword>Americana</Keyword>
            <Keyword>wild</Keyword>
            <Keyword>wilderness</Keyword>
            <Keyword>beauty</Keyword>
            <Keyword>autumn</Keyword>
            <Keyword>fallBull Elk</Keyword>
            <Keyword>Wapiti</Keyword>
            <Keyword>Cervus canadensis</Keyword>
            <Keyword>Madison River</Keyword>
            <Keyword>male</Keyword>
            <Keyword>rack</Keyword>
            <Keyword>antlers</Keyword>
            <Keyword>rut</Keyword>
            <Keyword>animal</Keyword>
            <Keyword>wildlife</Keyword>
            <Keyword>deer</Keyword>
            <Keyword>OK - Marginal</Keyword>
         </AnnotationFields>
         <UserFields>
            <UserField_4>Weidman 3466 Photo Collection</UserField_4>
         </UserFields>
      </MediaItem>
      <MediaItem>
         <AssetProperties>
            <Filename>WEIDMAN_081002_C5R3096.CR2</Filename>
            <Filepath>10 Master Image Files: Master TIFF image file archive:RAW_061 Yellowstone NP, WY:

[ianw]

Mark,

If that's the output you are getting then you haven't followed the instructions correctly.  What you are seeing is the output from the Make XML Data File menu option but without any XSL transformation.  I'll repeat the instructions here for clarity...

You need to copy the following code into a file on your computer.  As previously mentioned it needs to be a character perferct copy.  I've emailed you a copy of this as well.

Code:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" version="1.0" encoding="iso-8859-1" indent="yes"/>
<xsl:template match="/">
  <html>
   <head>
    <title>Keywords in Catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</title>
   </head>
   <body>
    <p>Sorted list of Keywords <b>used</b> in catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</p>
    <ul>
     <xsl:for-each select="CatalogType/MediaItemList/MediaItem/AnnotationFields/Keyword">
      <xsl:sort select="." order="ascending" data-type="text"/>
      <xsl:choose>
       <xsl:when test="position() &gt; 1 and not(. = ./preceding::node())">
        <li>
         <xsl:value-of select="."/>
        </li>
       </xsl:when>
       <xsl:when test="position() = 1">
        <li>
         <xsl:value-of select="."/>
        </li>
       </xsl:when>
      </xsl:choose>
     </xsl:for-each>
    </ul>
   </body>
  </html>
</xsl:template>
</xsl:stylesheet>

The code should be saved in a file with an extension of .xsl e.g. keywords2html.xsl.

Then open up a catalogue in iView.  Start with a small one just to speed up the process, even if you have to create a test catalogue with only a handful of keyworded images.  Make sure all the images are visible.  In iView you don't need to select the images as I think the Create XML option always works on the whole catalogue - xMedia 2 now works on the selected images, which is an improvement.

From the "Make" menu option select "XML Data File".  For this process the only tick box you need ticked is the second one down marked Annotations.  Never tick any of the Create Folders options unless you really know what you want as you'll get new folders and files created on you computer.  At the bottom of this dialogue box is a "+" button marked XSL transform.  Press the "+" button and a file selection dialog should come up.  Navigate to the XSL file as saved above and select it and hit the Open button.  It should show the selected file name next to the "+" button now, although on a PC this is in a shade of grey that hardly shows up against the background!

Now hit the "Make" button.  This will prompt for the file name to save the output to.  Enter the name you want, also adding the .html extension.  If you don't put an extension it will default to .xml.  If you're doing this on a small catalogue it should only take a second or two, but a large catalogue can take a minute or two - hence testing this on a small one first.  It will first create a temporary XML file of your catalogue in the same folder you want to save the output.  On a PC this file is marked as "hidden" so you won't always see it.  Once this file is create it calls an XSLT transformer that it built in to iView.  This uses the XSL file above to transform the XML into the required output.  Once finished open the new HTML file in your browser and you should see a bulleted list of keywords as used by the images in your catalogue.

Please try to create the HTML file again.  From the sample XML the details for a single image said you had 27 keywords.  If your other images are as well annotated then this could add up to thousands of unique keywords.  Anyway report back later!

Ian

[Mark Weidman]

Ian

Thank you very much for the detailed message, I appreciate it.  It is 5AM MT and I am scrambling to leave for a three day photo shoot.  I will review your instructions this weekend upon my return and will follow through and reply on the forum.

Thanks again!

Mark W.

[Mark Weidman]

Ian

I am back in my studio and have spent about the last hour trying to follow your instructions extremely carefully.  The instructions are concise and easy to follow, but I am getting the same error message as before, and the html file is more like a Word document with all of the metadata run together.  I have used both the xsl file you sent and one I created (in MS Word).  I am doing this on a sample/test iView catalog.  When I follow the instructions and click "Make" it seems to go through the process but at the very end I get the error message (Failed to complete xsl transformation!  Make sure XSL file is valid. -8960).

Bye-the-way, the test catalog only has 2400 items in it.  But I have the same problem with it that began this quest.  When I switch from the Info mode to Organize it ties up the computer for maybe 10 seconds, but is not nearly as slow as my Catalog with over 30k items in it. 

I am at a loss for any other options.  I will see if I can someone else to follow your instructions to double check I am not repeating some simple error.

One specific question.  When I created a document into which I paste the code you provided, I am using MS Word, then saving the document with a suffix of .xsl    The document you emailed me is a Browser (Safari in my case) document.  Regardless, neither of them seems to do the trick.  Sorry to be such a pain!

Mark Weidman

[ianw]

Mark,

Don't use Word to create the file as that will contain a lot more text than the code supplied.  If you cut and paste the code it should be into an empty plain text file, which is then saved as "name.xsl".  If you were on Windows I would suggest using Notepad.

You should be able to open the XML file in a web browser and it will show it as structured text.  If you get any errors in a browser about it being badly formed or something similar then there's something wrong with the XSL and iView or any other program will fail - the error message you are getting sound about right for this.

If you still have a problem then you could email me a zipped up XML export of your test catalogue and I can try it here.  However I don't think it's a problem with the catalogue - although the full catalogue may be too big for it to work.  I think it's probably user error

Ian

[Mark Weidman]

Ian, AOK, I used TextEdit on the Mac and created a file with the code.  At first TextEdit would not let me save the file with xsl as the suffix, but when I changed it on my Desktop the Icon indicated a Safari - Browser document.

Here is the content of that file:

w:defaultTabStop w:val="720"/><?xml version="1.0" encoding="utf-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><xsl:output method="html" version="1.0" encoding="iso-8859-1" indent="yes"/><xsl:template match="/">  <html>   <head>    <title>Keywords in Catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</title>   </head>   <body>    <p>Sorted list of Keywords <b>used</b> in catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</p>    <ul>     <xsl:for-each select="CatalogType/MediaItemList/MediaItem/AnnotationFields/Keyword">      <xsl:sort select="." order="ascending" data-type="text"/>      <xsl:choose>       <xsl:when test="position() &gt; 1 and not(. = ./preceding::node())">        <li>         <xsl:value-of select="."/>        </li>       </xsl:when>       <xsl:when test="position() = 1">        <li>         <xsl:value-of select="."/>        </li>       </xsl:when>      </xsl:choose>     </xsl:for-each>    </ul>   </body>  </html></xsl:template></xsl:stylesheet>

When I followed your instructions in a test iView catalog, it produced a file named "Catalog-4.xml" but I still got the error message.  Here is the content of that xml file - per your previous suggestion I removed my contact info from this sample:

DRV_135:WEIDMAN_090609_C5R9098_MSTR.tif</Filepath>
            <UniqueID>2</UniqueID>
            <Label>0</Label>
            <Rating>1</Rating>
            <MediaType>TIFF</MediaType>
            <FileSize unit="Bytes">75645100</FileSize>
            <Created>2009:06:14 13:55:49</Created>
            <Modified>2009:06:20 13:54:40</Modified>
            <Added>2009:06:24 17:30:35</Added>
            <Annotated>2009:06:16 18:14:39</Annotated>
         </AssetProperties>
         <AnnotationFields>
            <Headline>The yellow blossoms of Thermopsis montana (Golden Banner) line RD 12 near Kebler Pass (9980&apos;), West Elk Mountains, Ruby Range, Gunnison National Forest, Colorado, USA</Headline>
            <Product>The yellow blossoms of Thermopsis montana (Golden Banner) line RD 12 near Kebler Pass (9980&apos;), West Elk Mountains, Ruby Range, Gunnison National Forest, Colorado, USA</Product>
            <Author>H. Mark Weidman</Author>
            <AuthorTitle>Photographer</AuthorTitle>
            <CreatorAddress> Drive</CreatorAddress>
            <CreatorCity></CreatorCity>
            <CreatorState></CreatorState>
            <CreatorPostcode></CreatorPostcode>
            <CreatorCountry>USA</CreatorCountry>
            <CreatorPhone></CreatorPhone>
            <CreatorEmail></CreatorEmail>
            <CreatorURL></CreatorURL>
            <Credit>H. Mark Weidman</Credit>
            <Copyright>© 2009 H. Mark Weidman, All Rights Reserved</Copyright>
            <Rights>No usage rights granted without express written permission by photographer.</Rights>
            <URL>www.weidmanphoto.com</URL>
            <Location>Kebler Pass, Gunnison National Forest</Location>
            <State>CO</State>
            <Country>USA</Country>
            <Caption>The yellow blossoms of Thermopsis montana (Golden Banner) line RD 12 near Kebler Pass (9980&apos;), West Elk Mountains, Ruby Range, Gunnison National Forest, Colorado, USA. Golden Banner is also known as &quot;False Lupine&quot; because the pea-shaped flowers of Golden Banner and Lupine are similar.  The genus name even recognizes this similarity: &quot;thermos&quot; is Greek for &quot;Lupine&quot; and &quot;opsis&quot; is Greek for &quot;similar&quot;.</Caption>
            <Keyword>Colorado</Keyword>
            <Keyword>USA</Keyword>
            <Keyword>west</Keyword>
            <Keyword>western</Keyword>
            <Keyword>color</Keyword>
            <Keyword>country</Keyword>
            <Keyword>destination</Keyword>
            <Keyword>direct</Keyword>
            <Keyword>drive</Keyword>
            <Keyword>fast</Keyword>
            <Keyword>freedom</Keyword>
            <Keyword>freeway</Keyword>
            <Keyword>landscape</Keyword>
            <Keyword>motion</Keyword>
            <Keyword>movement</Keyword>
            <Keyword>nature</Keyword>
            <Keyword>outdoor</Keyword>
            <Keyword>outdoors</Keyword>
            <Keyword>path</Keyword>
            <Keyword>perspective</Keyword>
            <Keyword>road</Keyword>
            <Keyword>rural</Keyword>
            <Keyword>scenery</Keyword>
            <Keyword>scenic</Keyword>
            <Keyword>speed</Keyword>
            <Keyword>speedway</Keyword>
            <Keyword>transport</Keyword>
            <Keyword>transportation</Keyword>
            <Keyword>travel</Keyword>
            <Keyword>view</Keyword>
            <Keyword>way</Keyword>
            <Keyword>lane</Keyword>
            <Keyword>route</Keyword>
            <Keyword>venture</Keyword>
            <Keyword>transit</Keyword>
            <Keyword>dirt</Keyword>
            <Keyword>gravel</Keyword>
            <Keyword>rough</Keyword>
            <Keyword>wildflowers</Keyword>
            <Keyword>flowers</Keyword>
            <Keyword>blossom</Keyword>
            <Keyword>petal</Keyword>
            <Keyword>colorful</Keyword>
            <Keyword>remote</Keyword>
            <Keyword>Lupin</Keyword>
            <Keyword>Lupine</Keyword>
            <Keyword>sunflowers</Keyword>
            <Keyword>Kebler Pass</Keyword>
            <Keyword>West Elk Mountains</Keyword>
            <Keyword>Ruby Range</Keyword>
            <Keyword>Gunnison Natioinal Forest</Keyword>
            <Keyword>green</Keyword>
            <Keyword>yellow</Keyword>
            <Keyword>blue</Keyword>
            <Keyword>purple</Keyword>
            <Keyword>springtime</Keyword>
            <Keyword>summer</Keyword>
            <Keyword>Thermopsis montana</Keyword>
            <Keyword>Golden Banner</Keyword>
            <Keyword>Better</Keyword>
         </AnnotationFields>
         <UserFields>
            <UserField_1>Alamy</UserField_1>
         </UserFields>
      </MediaItem>
      <MediaItem>
         <AssetProperties>

[Mark Weidman]

Ian

I did some math, extrapolating on the average number of keywords in the test catalog.  Based on those parameters I probably have well over a million keywords in the catalog.  Would that in itself be enough to cause the slowdown?  I never realized there was any kind of "limit" on keywords.

Thanks

Mark

[peterkrogh]

Mark,
Not really a million keywords, but rather keyword entries (number of keywords times the images that have each particular keyword) right?
This could certainly make it slow. 

How many images in the catalog?   

This certainly looks like the culprit.

I see that you duplicate the city, state and location name in both the location fields and the keywords.  I wonder if it makes sense to delete the entry in Keywords, and maybe that would help.

Peter

[Mark Weidman]

Hi Peter

Yes, I figured roughly an average of 30 unique keywords per image X 31,000± images = 930,000 keywords.  It could be more or less by 100k.

Yes, I do duplicate the city state & location name.  Since many of these images end up at several different stock agents I assumed that was the way to go, in terms of metadata, to increase the chances of photo researchers "finding" the images.  I could go through and delete those entries in the Keywords field, but, wow, what a task that would be.

Mark

[peterkrogh]

So the behavior you see is that the catalog runs at acceptable speed, except when the keywords panel is open?
Does it settle down after opening, or stay sluggish whenever the panel is open?

I'm trying to figure out the best approach, but am not sure how to proceed.  There are several options.
Peter

[Mark Weidman]

Yes, it works fine except when I switch to the Organize/Keywords panel.  It does "settle down" just a bit after awhile, but is still pretty slow to respond to any actions.

Mark W.

[ianw]

Mark,

You still don't have a valid XSL file, so any transformation will always fail, leaving behind just the XML version of your catalogue.  What you have pasted contains characters not in my code - specifically the w:defaultTabStop w:val="720"/> at the beginning.  I suspect there are more characters than this too.  This is the problem of using a word-processor for editing in that they add their own details to the file which don't get displayed in the document i.e. the formatting instructions.  The above code is MS Word's way of storing the position of tabs in the document.  You should be able to use the XSL document that I emailed you, as long as you don't edit it, which may 'corrupt' it.

If you have a million keywords in your catalogue then this probably is the cause of your performance issues.  The only limits that I know of in iView are the number of images that can be catalogued (128,000) or the physical size of the catalogue (1.6GB?), which ever comes first - plus a limit of 16 Custom Fields per catalogue, but none of these are the issue here.  Storing all this information in memory when your catalogue is open probably gets a bit much when you have that number of keywords plus other populated annotation fields.

While a million keyword instances shouldn't be a issue it's all the indexing behind the scenes which is the overhead i.e. if you select a specific keyword iView instantly knows which images are associated with it i.e. it has an internal index of images against each keyword.  I guess that iView loads this information into memory for 'quicker' use when you first swap to the Organise panel.  It may be that this really does take a minute or so to do.  Don't forget that in the Organise panel it indexes images by a lot of different data items.  I wonder if it is smart enough to not do this for fields that are not displayed in the organise panel?  Building indexes for 30,000 images against 30+ data items, where many of the items have unique and long values is no small task.  For example each of your images has country / state / city / location details.  If these are only held against an image in the catalogue then to show them in the Place Finder in the Organise Panel iView has to scan 30,000 images and build the data tree.  Repeat this for labels, ratings, title, category, event, status, creator,...  This adds up to several million index entries.

I notice that you have a lot of 'duplicate' keywords e.g. 'west' & 'western', 'color' & 'colorful', 'transport' & 'transportation', 'flowers' & 'wildflowers', 'outdoor' & 'outdoors', 'speed' & 'speedway'  I'm not a keywords expert - this is one of my next big tasks.  However is there any value in having both singular and plural versions of a keyword?  The only time I can see this having value is if the spelling is more different than just adding a letter 's' e.g. 'canary' & 'canaries'.

As Peter has pointed out you have location information duplicated into the keywords.  Does this add value?

I'm not sure of a way around this, as I suspect that Expression Media 2 works the same way under the hood.  In the long term maybe Expression Media 3, if / when this arrives, will solve this sort of problem.  It needs to address the catalogue size limit, so if it does then it must be capable of handling large numbers of keywords etc.  I'm guessing that to do this will mean that the internal structure of the catalogue will be quite different, even to the point that existing catalogues will need to be converted to a new format.  If a desktop computer can handle an Access database with millions of rows, or an SQL database with a similar number of entries, then there's no reason why a future version of Media can't do the same.

Ian

[Mark Weidman]

Hi Ian

Thanks again for the detailed input, I appreciate it.

Well, I finally got a list of Keywords from a small sample catalog.  I had to first open your xsl file in Word and save it in a Text Only Format.  That seemed to do the trick.  But, when I tried the exact same procedure with my Master Catalog it seemed to go through the process OK, but at the very end I got the same error message.

As an FYI, when I open your xls file in Safari, I get:    Keywords in Catalog '' Sorted list of Keywords used in catalog ''

When I open your XSL file in MS Word, I get the code:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
 <xsl:output method="html" version="1.0" encoding="iso-8859-1" indent="yes"/>
 <xsl:template match="/">
  <html>
   <head>
    <title>Keywords in Catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</title>
   </head>
   <body>
    <p>Sorted list of Keywords <b>used</b> in catalog &apos;<xsl:value-of select="CatalogType/Catalog"/>&apos;</p>
    <ul>
     <xsl:for-each select="CatalogType/MediaItemList/MediaItem/AnnotationFields/Keyword">
      <xsl:sort select="." order="ascending" data-type="text"/>
      <xsl:choose>
       <xsl:when test="position() &gt; 1 and not(. = ./preceding::node())">
        <li>
         <xsl:value-of select="."/>
        </li>
       </xsl:when>
       <xsl:when test="position() = 1">
        <li>
         <xsl:value-of select="."/>
        </li>
       </xsl:when>
      </xsl:choose>
     </xsl:for-each>
    </ul>
   </body>
  </html>
 </xsl:template>
</xsl:stylesheet>

Regardless of all this, it seems that the huge slowdown problem is related to too many keywords.  Hopefully the next update of EM will resolve this problem.  In the meantime I will try to avoid using the Organize/Keywords Panel.

Mark