Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
Print Page - ImageVerifier - new image verifying application (free beta for Mac)

The DAM Forum

Software Discussions => ImageIngester and ImageVerifier => Topic started by: Marc Rochkind on March 08, 2007, 02:29:50 PM



Title: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 08, 2007, 02:29:50 PM
ImageVerifier (IV for short) is a new application I've been working on that can process a hierarchy of folders looking for image files to verify. It can verify TIFFs, JPEGs, DNGs, and non-DNG raws.

IV is designed to process large numbers of images. Folder hierarchies with 100,000 images or more should be no problem. In one test run, IV ran for 14 hours.

All verification is built-in; for DNGs IV doesn't use Adobe DNG Converter, as ImageIngester does, but instead uses Adobe's DNG SDK directly. JPEGs, TIFFs, and non-DNG raws are verified using built-in libraries as well.

For all image files, verification is performed by reading the actual image data, decompressing as necessary. This can find most errors, but I'm still working on coming up with some statistical data on how effective the methods are for various invalid files. Please share your own experiences with me.

The list of extensions for raw files is the same as II's, except that, unlike II, there's a hidden way to change the extensions. Let me know if you need this. (You'll need this if your raw files end in TIF.) I may add this option to the user interface in a later beta version.

The real work is done by subprocesses, so IV can take advantage of multiple CPU (or multiple core) computers. If you have 4 CPUs, it should be capable of fully loading all 4 at once.

For each verification run, called a job, you can choose the folders, whether to process subfolders or just the top level, what kinds of images to process (TIFF, JPEG, DNG, and/or non-DNG raw), the maximum number of errors to report, and whether to store the results in a built-in database.

You can save the settings in a named job, which acts something like II's Preference Sets.

There's a built-in scheduler that allows you to schedule jobs to be run once at a specified time; daily at a specified time on specified days (e.g., Tuesdays and Saturdays at 2am); and monthly on a specified day and time (e.g., the 3rd of every month at 5am).

The scheduler uses the "cron" facility built into OS X. IV doesn't have to be running for a scheduled job to run, nor does it keep its own daemon process running.

The download link is http://ImageIngester.com/ImageIngester/download/ImageVerifier-1.02.01B1.dmg. (It's a Universal Binary.)

The free betas expire at some point (this one on 30-April-2007). The released version will sell for somewhere between $25 and $40, and probably be packaged with ImageIngesterPro at a reduced price. There won't be a free version of IV as there is for II.

I plan to start work on the Windows version in a week or so, and I hope to have a Windows beta completed in one-to-three months.

As always, feedback is much appreciated, especially as to IV's effectiveness at finding invalid image files.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: peterkrogh on March 08, 2007, 03:05:44 PM
Marc,
Thanks for making this: I look forward to giving it a whirl.
Will this work on DVDs?
Peter


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 08, 2007, 03:26:18 PM
Should work on anything mountable, Peter.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Tom on March 08, 2007, 07:52:57 PM
Marc,
I have not found a place for request feature so will put it here for now.
Currently you have a root folder preconversion backup (raw) and post conversion backup (DNG).
Would it be possible to have a DNG written to the root folder for those of us that are considering using DNG from the start?
There maybe good reasons not to do this but let me know your thoughts especially if we import the DNG with full jpegs into LR.
Thanks,
Tom


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 08, 2007, 09:43:22 PM
Tom--

I also responded privately, and I may not be understanding your question.

If you convert to DNG in IIP, you have:
1. Pre-conversion backup: original raw,
2. Primary folder: DNG,
3. Post-conversion backup: DNG.

The primary folder is normally what you would import into LR. (That's what I do for all but one of my cameras.)

If you don't convert, all 3 are the same, but you still import the primary folder into LR. (That's what I do for my Leica M8, which uses DNG for its raw.)

Can you elaborate on what you've asked that I've failed to answer?

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: BobSmith on March 09, 2007, 03:22:57 AM
thanks Marc!  IV should prove to be a very useful little application.  

A few comments:  In my studio I use network home directories.  Home directories are on an OSX Server, not the local workstation.  Running IV gives an error that the database is locked.  It continues to run and report as expected but the tools>results window is not operational.  I'm assuming this is probably an issue with the network home, but it may have something else to do with the particular workstation I was on (Intel iMac... 10.4.8 )

I'd eventually like to see the addition of PSD support if possible.  Also, maybe make the progess window a bit smaller.  It stays on top of all other applications/windows and conveys relatively little info for its rather large size.  This is an issue when trying to run IV in the background on a busy Powerbook.

Bob Smith


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: BobSmith on March 09, 2007, 03:39:18 AM
And while I'm thinking about it...

Years ago I use to use an obscure little utility (I can't recall the name) that ran only on OS-9 and worked only on CDs.  It would report on how many times it had to read a particular segment on a CD in order to accurately extract the data.  It was a great way of indicating that you had a burner that was failing or problematic media.  A disk might verify during the burn process just fine... but only because the reader was having to read each segment multiple times before it could extract the data versus a good disk that's easily read on the first pass.  It gave a great indication of how good your disks really were.  And could show a failing disk before it reached the point of being unreadable.

I've never seen such a utility for OS-X or DVDs. As flaky DVDs can be at times, it's sorely needed. If anyone knows of such a thing I'd love to hear about it.  I mention it here as you Marc, seem to be the type of programmer that might be inclined to make such a utility happen.  It might even be incorporated into the feature set of a utility like IV

Bob Smith


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 09, 2007, 10:04:53 AM
Bob--

Thanks for the feedback! When you got the message about the database being locked, is there any chance that another app (e.g., an SQLite Database Browser session) had it opened? Were two instances of IV running?

I may be able to do PSDs once I look into it.

Were you actually able to use your PowerBook with IV pinning the CPU meter? ;-)

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 09, 2007, 10:07:15 AM
Bob (again)--

Low-level checkers are outside of my area, but such a thing would indeed be useful. I've also heard of utilities that check hard drives in a similar way. I think Windows has more of this sort of thing than OS X does.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: peterkrogh on March 09, 2007, 12:55:47 PM
Marc,
I'm running it on some DVDs. No errors so far, but there's one coming up soon that I know has some corrupt files on it: I'll let you know if it finds them.

Can IV let us know what type of files are getting skipped?
Petert



Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 09, 2007, 02:59:53 PM
Peter--

Files are skipped only because of their extension. I suppose I could compile a count by extension of what's skipped.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: BobSmith on March 09, 2007, 05:48:47 PM
I've noticed that all of the skipped files I've encountered are xmp sidecars when verifying Canon CR2 files.

Mark, I'll try IV again on the iMac where I had the database locked error.  I don't know of anything else running on that machine that would have used SQlite and I know I didn't have more than one instance running.  Yes IV worked just fine on my Powerbook.  I was using it while reading this forum and others... doing email, web browsing and such.  IV is no speed demon but it seems to work well so far.  I need to feed it some known corrupt files and see what happens.  Thanks for all of your efforts!!

Bob Smith


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: peterkrogh on March 09, 2007, 07:00:23 PM
Marc,
IV found some known-corrupted NEF files this evening on a DVD - glad to see that.

I think the count of skipped file extensions would be very handy.

On a side note, part of my storage verification is to run disk utilities on the drive first.  Any chance we could get IV to launch a session with the user's favorite disk/directory/file verification before running? (Disk Warrior/TechTool Pro for me).

The other thing that would really be cool would be if there would be some way to compare it against a known saved list of files on the drive (like, say, against an iView catalog for the drive/disk).  This would provide an error-check against accidental user deletion of files.

I'm sure you could figure out a way to do this...

I think it's shaping up to be a great and necessary utility.
Peter



Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 09, 2007, 10:12:04 PM
Peter--

I think a comparison against another reference, such as iView catalog, would be very difficult to make usable. I could see numerous false error reports, wasting much time to track down, only to find that the error was inconsequential.

But, my plan is to provide a report to easily compare results on the same folder, since all runs are in the database. This would allow you to easily check that the number was the same as before. It doesn't help if the first run was also missing an image, however. A simple extension would allow a comparison of two arbitrary folders, and then one could be a run of IV against, say, the images in an iView catalog, assuming they were in the same tree (not the catalog itself).

The nice thing about these reports is that they are just reports against the database--not anything that requires something new in the basic IV operation. This is why the database is there. IIP also has a database, but it isn't nearly as useful, since there's not much there other than ingestion info.

Let me know if you try IV against any of your large rotating archive volumes.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: peterkrogh on March 10, 2007, 10:35:21 AM
Marc,
So you are saying that IV keeps a database, and further runs of the app will flag if the number of files changes?  This would be quite valuable.

Any chance that you could assign a subfolder on one drive to equal a folder on other media (I'm thinking about comparing buckets between drives/DVD).

So Bucket Raw_007 should be the same on each drive, flagging me if images on one instance are missing from another.

This would also create the need for the user to be able to update the database record if a deletion was intentional, and the user no longer wanted to be flagged in the event of an error.

As to iView functionality, it could first run a "Find Missing Items" check, then launch IV from iView as a helper app.  This would accomplish the tasks of checking for missing items, followed by an integrity check.

And how about having IV invoke DiskWarrior, TechTool, etc?

This could become essential software very quickly.

Peter



Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: Marc Rochkind on March 10, 2007, 03:15:58 PM
Peter--

There's a database in the version you have. Choose Tools-Results and you'll see it.

Comparing two drives is very doable... I'll see what it takes to put that in. I think maybe there should be a panel to assign folders to mirror sets, to indicate that they are supposed to be the same. Then IV could check that they have the same files, in addition to checking that the files were valid. That sounds like a very natural extension IV's capabilities, well within its scope.

Changing the database gets involved in some usability issues. It needs to be clear whether database records what happened from the run, or what the result should be. Maybe I need two separate tables. This will require some careful thought.

Putting any apps the user desires on a menu would be easy enough. But invoking a tool just because there were invalid images seems like a stretch... I wouldn't want the user to think that a particular result implied that a certain problem was the cause.

--Marc


Title: Re: ImageVerifier - new image verifying application (free beta for Mac)
Post by: peterkrogh on March 10, 2007, 05:19:03 PM
>There's a database in the version you have. Choose Tools-Results and you'll see it.

already been looking at it. Very helpful, even in the early form.

> think maybe there should be a panel to assign folders to mirror sets, to indicate that they are supposed to be the same

Yes, please.

>putting any apps the user desires on a menu would be easy enough. But invoking a tool just because there were invalid images seems like a stretch...

Actually, I was thinking of slaving IV off of iView, rather than the other way around.  So I could have a catalog of my drive, and I would have iView check for missing items, and then access IV to run on the Drive/DVD/Bucket.  This would require putting some hooks in IV so that it could be controlled by scripts, right?  Is this hard to do?

And the other idea would be to have IV launch a disk utility before it runs its integrity check.  So I would point IV at a drive, it would launch DiskWarrior and perform the check.  Once that was done, and DW determined that there is no directory damage needing repair, IV would run and make sure that files were valid.  I think this is the proper order for file validation, right?

If it's too much work to get IV to launch a disk utility, then perhaps this could also be accomplished by a checklist of sorts in the DB. This could be accompanied by user prompts if necessary.

This also points up the need to have a printed version of the DB list (I would also *really* like this in IIP.) Although I am generally in opposition to requiring printouts to manage digital work, file validation seems to be a place where it would be handy and appropriate.

Peter