Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
Limitation on folder depth or number of files?
The DAM Forum
Welcome, Guest. Please login or register.
October 21, 2020, 08:49:37 PM

Login with username, password and session length
Search:     Advanced search
28033 Posts in 5147 Topics by 2903 Members
Latest Member: kbroch
* Home Help Search Login Register
+  The DAM Forum
|-+  Software Discussions
| |-+  ImageIngester and ImageVerifier
| | |-+  Limitation on folder depth or number of files?
« previous next »
Pages: [1] Print
Author Topic: Limitation on folder depth or number of files?  (Read 6714 times)
sdwike
Newbie
*
Posts: 24


View Profile
« on: July 08, 2006, 08:01:38 PM »

Is there any known limitation on the depth of folders or number of files processed in a session?

If I point ImageIngester at a folder (Windows) that has 100 sub-folders, and a cumulative total of over 2500 files, will it run to completion, or does it use RAM in such a way that it would likely run out of RAM?

What about an entire hard drive full of images?  Coule it thoeretically process many tens of thousands of images in a session?

I am just trying to define my stratgey for re-naming and adding metadata to an entire collection.

Thanks,

Steve
« Last Edit: July 08, 2006, 08:25:54 PM by sdwike » Logged
Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« Reply #1 on: July 08, 2006, 08:39:43 PM »

Steve--

I have tested ImageIngester with thousands of files, but not to that depth of folders. However, there is no built-in limit, and it takes the memory it needs, so with today's systems, I'm pretty sure it can handle any depth.

However, be careful when you point ImageIngester at an arbitrary folder. It ingests everrything it finds, not just images. It doesn't know the difference between an iView catalog, a web page, and an MOV. (It does know what raw files and JPEGs are, however.) So, it makes sense to point it a hierarchy of images, but not at a whole disk full of all your stuff.

There's another problem: If you're running DNG Converter, ingestion stops at the first error, and, given enough raw image files, the probability that one of them is invalid becomes high enough to worry about. Also, you can't turn off validation of JPEGs, and an error in one of them also stops ingestion.

I'm planning a set of features to make scanning of entire disks more practical, but it's not something I'm working on just yet.

However, if you have deeply nested folders of images, and you think they are all valid, then the current beta release of ImageIngester can indeed plow through the whole lot.

(This new freedom to ingest arbitrary folders was designed for two situations: (1) When the camera captures directly to disk, and (2) when cards are copied to a portable drive in the field, and then ingested from that drive using ImageIngester back in the shop. Actually, even the older versions could handle these situations if the structure followed the DCF standard, but the flexibility of the newest beta versions is sometimes required.)

--Marc
Logged

sdwike
Newbie
*
Posts: 24


View Profile
« Reply #2 on: July 08, 2006, 08:56:05 PM »

Thanks for the feedback Marc.

In my case, the folders are all full of just image files, but they include .CRW, .CR2, .JPG, .TIF and .PSD file types.  What will II do when it encounters PSD?
I also wouldn't be using the DNG converter at the same time, as this first pass is really about normalizing naming conventions and metadata.  Baby steps.

Thanks again.  I have the new beta version and was tinkering with it when the question came to mind about how big a bite this tool was meant to take at a sitting.
I think I'll set up a test tomorrow and see how it behaves on a batch of about 2500 raw files that are in their own directory tree.

Thanks again.
Logged
Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« Reply #3 on: July 08, 2006, 09:42:44 PM »

Steve--

What ImageIngester will do with a PSD is ingest it... copy it, back it up, rename it, etc. ImageIngester never looks INTO any file, so the PSD will remain a PSD. The same is true of any other file it encounters. However, it knows enough not to send non-raws to DNG Converter, and it knows enough not to use JPEG validation on anything that doesn't end with ".JPG". Other than these, it doesn't care what the extension is.

Ultimately, there will be a version of ImageIngester that allows you to specify the extensions to be ingested. Also, if I can figure out how to do it efficiently, there will be a version that can keep going after an error.

--Marc
Logged

sdwike
Newbie
*
Posts: 24


View Profile
« Reply #4 on: July 09, 2006, 05:30:43 PM »

I tried this on a large directory structure of raw files it worked flawlessly.

I ran an "ingestion" of 4700 files (.CRW, .XMP) from a flashtrax USB external storage unit, into file folders on my firewire external RAID array.  It took about 45 minutes to completely finish, with no errors at all.

Wish list items:

1.  A way to tell it to skip certain file types ( in case you don't want original JPG's, or THM's, etc).
2.  A selection to save the original file name to a metadata field.
3.  More flexibility in how output directories are named/structured.
4. A switch to disable creation of the "backup" directory.  (maybe I missed this, but I couldn't find it?)
All in all, this worked very well.  Thanks for a very useful tool.
Where do we make contributions?

If one uses the DNG converter in the process, do you end up with a third directory of files (*.DNG), or does it then not make the other primary and backup directories of raw files?
Logged
Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« Reply #5 on: July 10, 2006, 11:00:51 PM »

Steve--

Glad to know this worked for you... of course, it was designed that way, but designers are still surprised when things work as they're supposed to! ;-)

If you convert to DNG, the primary folder contains DNGs, and the backup contains the original raws. "Backup" is a misnomer... the names should really be "pre" and "post". For non-raws, the backup and primary folders contain the same files. My original concept was that ImageIngester would have a built-in capability to save original raws, and that that would also serve as a temporary backup until the real backup kicked in. But, because I named it "backup," everyone assumed it was a backup, and that's why my original design (e.g., wacky folder names) didn't work out.

I am working on a new version that incorporates 1, 3, and 4 on your list. To do 2 involves playing with the internals of the sidecar, and I haven't decided whether to do that yet.

As for contributing: I am going to have an ImageIngesterPro sometime this year (day job permitting), and that will have a license fee. ImageIngester will continue to be free, and will get some additional features, but the new fancy stuff will go mostly into the Pro version. I will probably do what Adobe does with Lightroom and make the ImageIngesterPro beta free, so as to maximize feedback.

I haven't set the price for ImageIngesterPro. I'm leaning towards $2 million per copy. At that price, I doubt if I'll sell more than 3 or 4. ;-)

--Marc
Logged

sdwike
Newbie
*
Posts: 24


View Profile
« Reply #6 on: July 11, 2006, 02:50:17 PM »

Well, that seems a fair price . . . Put me down for a couple  Cheesy

Realistically, as it evolves, I can see it being worth a fair market price, like any other software tool.

One anomaly I ran across, was due to my use of your tool to do something it wasn't meant for:

I had a directory structure under a folder called "incoming" full of original raw files in sub-directories by date captured.

I had Image Ingester rename/move/add metadata to these files, placing them in my new directory area.

Since some of the files in each folder had at some time been opened in ACR, there were some xmp files scattered in the folders.  
Since these xmp files had dates corresponding to the last ACR access of that raw file, their dates didn't match the dates of the crw files.

So, as Image Ingester was churning through these directories, dutifully renaming and copying them to my new year and date folder structure, and adding metadata from my PS template, these xmp files ended up being tossed into folders under the date the xmp's were created.  

In many cases these happened to be dates where no other files were, so the whole folder was just xmp files.  

It was a bit of a mess until I caught on, and stopped processing long enough to delete all old xmp's from my old raw directories.

I realize that if I were converting to DNG at the same time, it would have been ok, since it would have processed the xmp into the DNG first, then done the move/rename.

It's just something I hadn't anticipated and adjusted for, but will going forward.

Obviously, the tool was designed for a specific workflow and process, and once all my old stuff is moved/re-named/converted, and I am on track, the usage as intended looks to be very smooth indeed, going from card to file, to DNG with metadata attached in one operation.

Thanks again, and be sure to put that $2M on my tab . . .

« Last Edit: July 11, 2006, 02:53:15 PM by sdwike » Logged
Pages: [1] Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!