RE: Transferring Large Volume of Images, Integrity Check

[Emile]

I'm planning to move about 185 GB of image files from one Seagate to another.

When I download images from a CF card to the HDD of my main computer, I'm able to do an integrity check right away, by opening the images in a browser and working on them.  It's only afte I've completed the edit, and backed up the images that I wipe the cards.

I've never moved a volume this large, and I'm concerned about discovering corrupt files only after I've zeroed out the data on the older Seagate.  Yes, I've backed up the images to DVD, too, but I'd like to avoid having to recover the images from DVDs.

Is there a good way to do an integrity check on the images after they've been transferred without having to go through the whole volume in an image browser?

Suggestions, anyone?

Thanks in advance.

-Emile

[David C. Buchan]

Hi Emile,

I did a quick search on the web for "md5 checksum batch generator" and came up with http://www.brothersoft.com/file_disk_management/miscellaneous/ef_checksum_manager_28413.html

I've not used this software so all the usual caveats apply. Looks like will create a checksum for each file or directory which you can then regenerate and match after you've copied your files. The description on the linked page says it all.

If it works, post here as I'm sure this would be a useful tool for testing the integrity of backups to CD/DVD as well.

Regards,

David

[Marc Rochkind]

You can try ImageIngesterPro. It wasn't built specifically for this, but it does have the capability of handling lots of images and doing a verification of JPEGs and raws. The raw verification involves converting with DNG Converter and throwing away the result, so it's going to be slow for great numbers of images. Neither verification is absolutely foolproof, of course.

A better approach is to run a copying utility that can also then do a verification check. This doesn't really tell you whether the images are OK, only that the copy is the same as the original. But, that's probably all you want.

--Marc

--Marc

[Dawnne Gee]

FWIW, I used the method Marc describes to move roughly 0.75Tb of stored RAW photographs from three hard drives to a new disc array. As he notes, verification takes a long time, but it did work just fine. Also as he notes, verifying a good copy of a bad file doesn't tell us much. But if you know before-hand that you have no bad files, then verifying a good copy becomes meaningful.

[Marc Rochkind]

Dawnne (and others)--

The chief problem with a large re-ingestion is if you're running DNG Converter for verification, as it stops on the first error. Without running DNG Converter, it should be OK. JPEG verification works well for a large re-ingestion, as it will report as many errors as it finds.

For a future release, I'm considering making raw verification a built-in feature, to both speed it up and make it more robust when it finds an error. However, this will not be until Version 3.

--Marc

[Dawnne Gee]

Heh....guess I was lucky, then. I only had three "bad" images in the archive, and I took care of them before I used IIP to copy them over to the new server. Thanks for all the work on II/IIP you do Marc. It was truly a heartache saver during this migration!

[peterkrogh]

Marc,
What can one use for an independent JPEG integrity check?
Peter

[Marc Rochkind]

Peter--

Independent of what? ImageIngester?

--Marc

[peterkrogh]

Yes, independent of ImageIngestr.  Or can I point ImageIngester to a folder full of images and just let it run.  I have not tried that.

i was assuming that ImageIngester is using some other application (as it uses the DNG converter) to perform the JPEG integrity check, but maybe you just built some kind of checksum?

I'm interested in these kinds of data validation tools.
Peter

[Marc Rochkind]

Peter--

For JPEGs, the integrity check is internal to ImageIngester. I simply read the JPEG image data in and decompress it, using a JPEG-processing library, and see if there are any errors. However, I don't do anything with the in-memory image--I just discard it and go on to the next JPEG.

I plan in the future to do DNG and TIFF verification the same way, but that won't be for a while.

You can certainly use ImageIngester that way, although you will get a temporary copy, which you can then discard.

--Marc

[peterkrogh]

Well, it seems we can start work on ImageVerifier once ImageIngester delivers...
Peter

[Marc Rochkind]

Yup... that's exactly my thinking. Might be a built-in feature, though. I've also been thinking about doing something about the administration of buckets.

--Marc