iView MediaPro catalog running sloooow

[Mark Weidman]

I have an iView MediaPro catalog that is 409 MB in size, there are 31,361 items (all image files).  The catalog works great when I am in the "Info" interface.  But, when I switch to the "Organize" mode it slows down significantly.  It can take 30 to 60 seconds to respond to a command or selection.  What is the reason for this, and, is there anything I can do to improve the response time?

Mark Weidman

[peterkrogh]

Mark,
Does the problem show equally when the image drives are mounted as if they are nt (assuming you can dismount the drives).?

You might try rebuilding the catalog - drag all images to a new catalog, or try to make a new catalog, and Import>Import from Catalog

Peter

[Mark Weidman]

I generally only work when the image drives are mounted, and, yes, have been mounted during the course of this problem.  I will try your suggestion of rebuilding the catalog.

Do you suggest I stick with iView MediaPro, or, upgrade to Expressions Media 2?

Thanks

Mark

[Mark Weidman]

Peter

I created a new catalog and imported add the records into it, but the probem still remains.  The hard drive with the images is in an external enclosure.  The catalog itself is on my workstation's internal hard drive.  Is there a limit to the number of keywords allowed?  I have no idea how many keywords are in the catalog, but there are thousands.  The response, when in the "Organize" mode, is so slow that it is not even practical to use it.

Mark W.

[peterkrogh]

Mark,
There can be a problem if there are too many subfolders (I think that's more likely than too many keywords).  Since iView uses system resources to constantly monitor which images might have been moved, a large number of subfolders can slow the program down as it constantly checks.

Does the problem persist if the drive with the image files is disconnected?

And, yes, you might want to try Expression Media. Download the trial, and open a copy of your catalog in it. See if the problem goes away.
Peter

[Mark Weidman]

Hi Peter

I don't think there are too many (if any) subfolders of keywords.  The problem persists when I am working in the "Organize" mode of ONLY the "Keywords".  For example, if I go into the "People" folder, where there are far fewer items than in "Keywords" the program works fine.  I did download a copy of Expressions Media and the problem is the same.  I also tried (both IMP and EM) disconnecting the hard drive with the image files and the problem persists.

I will be leaving shortly for a several day trip but will watch for any further suggestions you may have.

Thanks

Mark W.

[peterkrogh]

Mark,
So the problem only happens when the Keywords panel is open?
Peter

[Mark Weidman]

The slow down is noticeable when I switch from the Info panel to the Organize panel, but, it seems to be at it's worst (i.e. slowest) when the Keywords panel is open.

Mark

[peterkrogh]

How many keywords you got in there?
If some of them are usage based terms (like "portfolio") maybe you could use catalog sets instead - this would split up the information into two places (which can also aid in searching the terms).
Peter

[Mark Weidman]

I don't know how many keywords are in the catalog.  Is there an easy way to automatically "count" them?  I never worried about creating "too many" keywords, but maybe that is the problem.

Mark W.

[ianw]

Mark W,

This post - http://thedambook.com/smf/index.php?topic=3014.0#msg16991 - will produce an html page showing the keywords used in your catalogue.  "Used" means any that have assets associated with it.  The instructions on how to do this are in another post that's referenced in it.

Note that due to the size of your catalogue it may not work.  Even if it does it may take several minutes to do.  This is not a bug with iView or Expression Media as such.  The XSL engine that is embedded within it can't cope with large XML files, but it's not clear when a file becomes too large.

You could of course just count the keywords manually!

Ian

[Mark Weidman]

Thanks Ian, I will give it a try and report back with the keyword tally.

Mark

[Mark Weidman]

Ian, I have carefully followed the instructions on the link you provided.  I keep getting an error message "Failed to complete xsl transformation!  Make sure XSL file is valid [-8960].  Any suggestions?

As an aside, I found about 1,000 catalog items that were not on mounted volumes.  I cleared these.  But the major slow down, when I switch to the Organize Mode from the Info Mode, is still a problem.  For example, I created a new, empty catalog and imported all 30k items from the problem catalog.  When I switched from the Info to the Organize mode, it takes well over a minute before I can move to the next function/step.

Peter - what exactly did you mean when you wrote the catalog may have too many subfolders (rather than too many keywords causing the slowdown)?

Mark W.

[ianw]

Mark,

I have just tried some XSL on my largest catalogue - currently 88,022 DNG files.  I was hoping to "break" it to see what error message I got.  However it worked for me so it wasn't a size problem!  However this catalogue contains no keywords, but a lot of catalogue sets - at least 4 entries per image.

I'm not sure why you are getting that error message.  It would suggest to me that it doesn't like the XSL file you've created.  It has to be a character perfect copy of what was entered in that thread.  A single missing character could render it invalid.  I'm not sure how you could test this on a MAC.  I know that if you open an XSL file in Internet Explorer or Firefox it will give an error if it's invalid - a parsing error.  Maybe this also works on a MAC?  If the file is valid then maybe your XML version of the catalogue really is too big!  Providing you only ticked the Annotations box in the "Include Sections" of the XML data file options, which keeps the XML size down, then I don't know of a way around this.

Most XML tools struggle with large files and it appears to be the phyiscal size of the file rather than the amount of content.  They have to read the whole file and create indexes to the various content in order to do the XSL, and the requirements for this grow exponentially with the size of the XML.  Usually they run out of memory when doing this and fail gracefully.  Older tools used to kill the PC!  The XML created by iView / xMedia is rather verbose, so makes things worse, and you have little control on what fields you want to include - just the 3 tickboxes to select/deselect chunks of data for each image.  At work we have to deal with an XML file that contains details for all the naughty and/or famous people in the world and this is currently over 600MB in size and growing.  We have to read it in record by record as if a plain text file and extract the details we wanted - very painful and slow!  The XML for my catalogue was around 50MB in size - iView creates, at least on Windows, a tmp_xml.noindex file in the output directory, although this is a hidden file so you won't always see it and it's only temporary.

Another way of seeing the keywords would be to make a text data file.  However this will give you a line per image and you would have to sort out the duplicates yourself.  This is easy in Excel, but you may not have this for your MAC.

I would say that for my largest catalogue it does kill the CPU for a long while, although I can swop between Info and Organise panels quickly. If I move to a different program, i.e. lose focus, it does seem to recover after a while.  There doesn't appear to be any disk usage, either on the drive my catalogue is or the drive where the images are.  I know that it does scan the directories within the catalogue looking for new files - the directory icon changes to indicate something new.  It does this even if you have not turned on folder watching.  I have almost 300 folders where my images are held.  I think this is what Peter is alluding to.  If you use "buckets" then iView / xMedia has less folders to look at.

Hope this helps.

Ian

[Mark Weidman]

Ian

The catalog has about 30,000 items (image files) and is 410MB in size.  The iView catalog resides on my G5 internal hard drive.  The image files are on an external hard drive in an enclosure.  I do use the "Bucket" system of organization; there are about 250 Buckets, with perhaps another 50 within those Buckets.  I think I finally got the XML file exported, but the format does not help in terms of counting keywords.  Here is a sample line of metadata extracted for just one image (note that the keywords are not broken out in any fashion):

WEIDMAN_081001_C5R2817.tif 20001 0 1 TIFF 128833430 2008:10:13 11:53:40 2009:03:17 14:36:31 2009:06:07 20:24:30 2009:03:17 14:17:31 Early morning view of fly fisherman, Madison River, Yellowstone National Park, Wyoming, USA Early morning view of fly fisherman, Madison River, Yellowstone National Park, Wyoming, USA H. Mark Weidman Photographer 28 Hanover Drive West Chester PA 19382 USA 610-388-9818, 610-368-8478 mark@weidmanphoto.com www.weidmanphoto.com © 2008 H. Mark Weidman All Rights Reserved No usage rights granted without express written permission by the Photographer. Yellowstone National Park, Wyoming, USA WY USA Early morning view of fly fisherman, Madison River, Yellowstone National Park, Wyoming, USA Yellowstone National Park Wyoming USA park nature landscape scenic natural west western Americana wild wilderness beauty autumn fall fly fisherman fishing sport challenge solitude man male cast rod reel vacation fun river Madison River BEST Released Alamy Weidman 3466 Photo Collection

Not sure if this additional information will help you provide further suggestions, but thanks again for your input.

Mark Weidman