ImageVerifier - new image verifying application (free beta for Mac)

[Marc Rochkind]

ImageVerifier (IV for short) is a new application I've been working on that can process a hierarchy of folders looking for image files to verify. It can verify TIFFs, JPEGs, DNGs, and non-DNG raws.

IV is designed to process large numbers of images. Folder hierarchies with 100,000 images or more should be no problem. In one test run, IV ran for 14 hours.

All verification is built-in; for DNGs IV doesn't use Adobe DNG Converter, as ImageIngester does, but instead uses Adobe's DNG SDK directly. JPEGs, TIFFs, and non-DNG raws are verified using built-in libraries as well.

For all image files, verification is performed by reading the actual image data, decompressing as necessary. This can find most errors, but I'm still working on coming up with some statistical data on how effective the methods are for various invalid files. Please share your own experiences with me.

The list of extensions for raw files is the same as II's, except that, unlike II, there's a hidden way to change the extensions. Let me know if you need this. (You'll need this if your raw files end in TIF.) I may add this option to the user interface in a later beta version.

The real work is done by subprocesses, so IV can take advantage of multiple CPU (or multiple core) computers. If you have 4 CPUs, it should be capable of fully loading all 4 at once.

For each verification run, called a job, you can choose the folders, whether to process subfolders or just the top level, what kinds of images to process (TIFF, JPEG, DNG, and/or non-DNG raw), the maximum number of errors to report, and whether to store the results in a built-in database.

You can save the settings in a named job, which acts something like II's Preference Sets.

There's a built-in scheduler that allows you to schedule jobs to be run once at a specified time; daily at a specified time on specified days (e.g., Tuesdays and Saturdays at 2am); and monthly on a specified day and time (e.g., the 3rd of every month at 5am).

The scheduler uses the "cron" facility built into OS X. IV doesn't have to be running for a scheduled job to run, nor does it keep its own daemon process running.

The download link is http://ImageIngester.com/ImageIngester/download/ImageVerifier-1.02.01B1.dmg. (It's a Universal Binary.)

The free betas expire at some point (this one on 30-April-2007). The released version will sell for somewhere between $25 and $40, and probably be packaged with ImageIngesterPro at a reduced price. There won't be a free version of IV as there is for II.

I plan to start work on the Windows version in a week or so, and I hope to have a Windows beta completed in one-to-three months.

As always, feedback is much appreciated, especially as to IV's effectiveness at finding invalid image files.

--Marc

[peterkrogh]

Marc,
Thanks for making this: I look forward to giving it a whirl.
Will this work on DVDs?
Peter

[Marc Rochkind]

Should work on anything mountable, Peter.

--Marc

[Tom]

Marc,
I have not found a place for request feature so will put it here for now.
Currently you have a root folder preconversion backup (raw) and post conversion backup (DNG).
Would it be possible to have a DNG written to the root folder for those of us that are considering using DNG from the start?
There maybe good reasons not to do this but let me know your thoughts especially if we import the DNG with full jpegs into LR.
Thanks,
Tom

[Marc Rochkind]

Tom--

I also responded privately, and I may not be understanding your question.

If you convert to DNG in IIP, you have:
1. Pre-conversion backup: original raw,
2. Primary folder: DNG,
3. Post-conversion backup: DNG.

The primary folder is normally what you would import into LR. (That's what I do for all but one of my cameras.)

If you don't convert, all 3 are the same, but you still import the primary folder into LR. (That's what I do for my Leica M8, which uses DNG for its raw.)

Can you elaborate on what you've asked that I've failed to answer?

--Marc

[BobSmith]

thanks Marc!  IV should prove to be a very useful little application.  

A few comments:  In my studio I use network home directories.  Home directories are on an OSX Server, not the local workstation.  Running IV gives an error that the database is locked.  It continues to run and report as expected but the tools>results window is not operational.  I'm assuming this is probably an issue with the network home, but it may have something else to do with the particular workstation I was on (Intel iMac... 10.4.8 )

I'd eventually like to see the addition of PSD support if possible.  Also, maybe make the progess window a bit smaller.  It stays on top of all other applications/windows and conveys relatively little info for its rather large size.  This is an issue when trying to run IV in the background on a busy Powerbook.

Bob Smith

[BobSmith]

And while I'm thinking about it...

Years ago I use to use an obscure little utility (I can't recall the name) that ran only on OS-9 and worked only on CDs.  It would report on how many times it had to read a particular segment on a CD in order to accurately extract the data.  It was a great way of indicating that you had a burner that was failing or problematic media.  A disk might verify during the burn process just fine... but only because the reader was having to read each segment multiple times before it could extract the data versus a good disk that's easily read on the first pass.  It gave a great indication of how good your disks really were.  And could show a failing disk before it reached the point of being unreadable.

I've never seen such a utility for OS-X or DVDs. As flaky DVDs can be at times, it's sorely needed. If anyone knows of such a thing I'd love to hear about it.  I mention it here as you Marc, seem to be the type of programmer that might be inclined to make such a utility happen.  It might even be incorporated into the feature set of a utility like IV

Bob Smith

[Marc Rochkind]

Bob--

Thanks for the feedback! When you got the message about the database being locked, is there any chance that another app (e.g., an SQLite Database Browser session) had it opened? Were two instances of IV running?

I may be able to do PSDs once I look into it.

Were you actually able to use your PowerBook with IV pinning the CPU meter? ;-)

--Marc

[Marc Rochkind]

Bob (again)--

Low-level checkers are outside of my area, but such a thing would indeed be useful. I've also heard of utilities that check hard drives in a similar way. I think Windows has more of this sort of thing than OS X does.

--Marc

[peterkrogh]

Marc,
I'm running it on some DVDs. No errors so far, but there's one coming up soon that I know has some corrupt files on it: I'll let you know if it finds them.

Can IV let us know what type of files are getting skipped?
Petert

[Marc Rochkind]

Peter--

Files are skipped only because of their extension. I suppose I could compile a count by extension of what's skipped.

--Marc

[BobSmith]

I've noticed that all of the skipped files I've encountered are xmp sidecars when verifying Canon CR2 files.

Mark, I'll try IV again on the iMac where I had the database locked error.  I don't know of anything else running on that machine that would have used SQlite and I know I didn't have more than one instance running.  Yes IV worked just fine on my Powerbook.  I was using it while reading this forum and others... doing email, web browsing and such.  IV is no speed demon but it seems to work well so far.  I need to feed it some known corrupt files and see what happens.  Thanks for all of your efforts!!

Bob Smith

[peterkrogh]

Marc,
IV found some known-corrupted NEF files this evening on a DVD - glad to see that.

I think the count of skipped file extensions would be very handy.

On a side note, part of my storage verification is to run disk utilities on the drive first.  Any chance we could get IV to launch a session with the user's favorite disk/directory/file verification before running? (Disk Warrior/TechTool Pro for me).

The other thing that would really be cool would be if there would be some way to compare it against a known saved list of files on the drive (like, say, against an iView catalog for the drive/disk).  This would provide an error-check against accidental user deletion of files.

I'm sure you could figure out a way to do this...

I think it's shaping up to be a great and necessary utility.
Peter

[Marc Rochkind]

Peter--

I think a comparison against another reference, such as iView catalog, would be very difficult to make usable. I could see numerous false error reports, wasting much time to track down, only to find that the error was inconsequential.

But, my plan is to provide a report to easily compare results on the same folder, since all runs are in the database. This would allow you to easily check that the number was the same as before. It doesn't help if the first run was also missing an image, however. A simple extension would allow a comparison of two arbitrary folders, and then one could be a run of IV against, say, the images in an iView catalog, assuming they were in the same tree (not the catalog itself).

The nice thing about these reports is that they are just reports against the database--not anything that requires something new in the basic IV operation. This is why the database is there. IIP also has a database, but it isn't nearly as useful, since there's not much there other than ingestion info.

Let me know if you try IV against any of your large rotating archive volumes.

--Marc

[peterkrogh]

Marc,
So you are saying that IV keeps a database, and further runs of the app will flag if the number of files changes?  This would be quite valuable.

Any chance that you could assign a subfolder on one drive to equal a folder on other media (I'm thinking about comparing buckets between drives/DVD).

So Bucket Raw_007 should be the same on each drive, flagging me if images on one instance are missing from another.

This would also create the need for the user to be able to update the database record if a deletion was intentional, and the user no longer wanted to be flagged in the event of an error.

As to iView functionality, it could first run a "Find Missing Items" check, then launch IV from iView as a helper app.  This would accomplish the tasks of checking for missing items, followed by an integrity check.

And how about having IV invoke DiskWarrior, TechTool, etc?

This could become essential software very quickly.

Peter