The DAM Forum
Welcome, Guest. Please login or register.
September 19, 2014, 12:51:39 PM

Login with username, password and session length
Search:     Advanced search
28018 Posts in 5138 Topics by 2910 Members
Latest Member: kbroch
* Home Help Search Login Register
+  The DAM Forum
|-+  DAM Stuff
| |-+  Data Validation
| | |-+  Great Windows Utility For Verifying Archive Photos
« previous next »
Pages: [1] 2 Print
Author Topic: Great Windows Utility For Verifying Archive Photos  (Read 15610 times)
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« on: July 17, 2009, 04:57:20 PM »

I recently came across a very nice file verification utility.  Here's a brief description derived from the website:

    * A file integrity verification tool:
          o Use it to make sure files copied to CD-ROM are bit-perfect copies,
          o Use it to make sure backups copied from one drive to another are just right,
          o Use it to make sure files haven’t been changed or damaged over time.
    * Multi-threaded, so your extra CPU cores get used when scanning multiple files and work gets done faster.
    * Happy with Unicode file names
    * Supports multiple checksum routines (hashes), like MD5, SHA1, CRC32, RIPEMD and others.
    * Supports recursive directory scanning.
    * Supports Very Big Files — If it’s on your hard drive, ExactFile can handle it.
    * Does everything popular file summer utilities do, like fsum, md5sum, etc, but better!
    * Compatible with popular file checksum digest formats.
    * For Windows 2000, XP, and Vista.
    * GUI and command line versions.
    * FREE.

It's extremely easy to use.  You just point to the highest level folder you want to create hash codes for and it creates a file containing all of them in the folder you selected.  Then when you want to verify that the images are still good, you just point to that folder again and push a button.  It's also quite fast.  I just ran a little test using MD5 128 bit (the internet standard for file integrity checking) and it only took 5 minutes to verify 2000 RAW images on my 3 GHz Dual Core.  So 20,000 images could be verified in less than an hour. 

Here's a link to the ExactFile website.

Please consider a donation. (I don't know the author and have no financial interest in the product's success).

Dan
Logged
danaltick
Hero Member
*****
Posts: 1616

evaa-xdtb@spamex.com danaltick
View Profile WWW Email
« Reply #1 on: July 17, 2009, 08:32:13 PM »

Dan,

Sounds promising.  Even in Beta it looks like it's worth comparing to SyncBack.  Thanks for the info.  Have you tired it with file backups?

Dan
Logged

WindowsXP, ImageIngester Pro, RapidFixer, IVMP 3, ACR4, Photoshop CS4, Controlled Keyword Catalog, Canon EOS50D
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« Reply #2 on: July 18, 2009, 06:45:53 AM »

Hi Dan. I haven't tried it with backups.  And just to be clear, ExactFile doesn't do file transfers.  I use (and like) SyncBackSE for validated file transfer too and use Acronis for validated full disk image backups of my OS/Application partition.  I'm using ExactFile in the Archive part of my workflow.

On the beta part, I'm not at all concerned.  This software isn't modifying my images or metadata, it's just writing a text file to the location I select.  The code is fairly simple and the comments, below the descriptive text, Here gave me confidence.

Dan
Logged
danaltick
Hero Member
*****
Posts: 1616

evaa-xdtb@spamex.com danaltick
View Profile WWW Email
« Reply #3 on: July 18, 2009, 09:47:37 AM »

Dan,

Sorry, meant to say file transfers.  How are you using it in the Archive part of your workflow.  For me that's also file transfers; unless maybe you are talking about the DVD/Blu-Ray burning.

Dan
Logged

WindowsXP, ImageIngester Pro, RapidFixer, IVMP 3, ACR4, Photoshop CS4, Controlled Keyword Catalog, Canon EOS50D
danaltick
Hero Member
*****
Posts: 1616

evaa-xdtb@spamex.com danaltick
View Profile WWW Email
« Reply #4 on: July 20, 2009, 10:54:32 AM »

Dan,

Just read through the last paragraph of your first post.  I see now that you are using it for post file verification of your archive.  I've been holding out in hopes of using the DNG converter or IIP to use the new DNG 1.2 embedded hash codes for quickly verify just the unchanging Raw data without having to create an extra copy of the DNG.  Hopefully we'll see something like this soon.

Dan
Logged

WindowsXP, ImageIngester Pro, RapidFixer, IVMP 3, ACR4, Photoshop CS4, Controlled Keyword Catalog, Canon EOS50D
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« Reply #5 on: July 20, 2009, 04:25:21 PM »

Yes, for the verified transfer part of the workflow, I still us SyncBackSE. 

Once a month, I run an ExactFile Check against the archive to verify that nothing has changed.  Then I add the new batch of photos and rerun ExactFile to create a new hash code file for everything.  I'm using the tool to verify that there haven't been any unexpected changes to my archive, included any pushed metadata in the individual files.  I think this will work well for me as I only add photos to my archive, I don't change them (changed files get a new version code in their file name).

It's slightly more complicated than I've described because I'm currently using two archives and I alternate which one gets updated.  Eventually, I'll probably go to 3, with the third copy being updated less frequently.  If my collect grows faster than technology gets faster, I might eventually have a hash file for each year.  In that way, fewer hash codes would need to be recalculated. But I'd want to run the verification of the entire archive periodically.

I'm betting Adobe won't expand the focus of their free DNG converter to validate an entire archive.  That would be a lot more scope creep than I'd want.  But they might well add a single image verification option to the Converter.

Dan
Logged
danaltick
Hero Member
*****
Posts: 1616

evaa-xdtb@spamex.com danaltick
View Profile WWW Email
« Reply #6 on: July 22, 2009, 08:38:35 PM »

Dan,

Sounds good.  Not really sure though why you're alternating your archive backups as opposed to using the 3-2-1 method outlined in the book.

Dan
Logged

WindowsXP, ImageIngester Pro, RapidFixer, IVMP 3, ACR4, Photoshop CS4, Controlled Keyword Catalog, Canon EOS50D
peterkrogh
Administrator
Hero Member
*****
Posts: 5682


View Profile Email
« Reply #7 on: July 22, 2009, 09:12:24 PM »

Dan,
Sounds promising. It sounds like it will deal with more filetypes than ImageVerifier.
One thing I'd like is the ability to drop the hashes on a disk so that they could be used to validate, even on a different machine that has never seen the disk before.  I see it like this:

Hash the original data on hard drive
Put the hashes in the folder of stuff to burn
Burn the disk.

Could you do that with ExactFile?
Peter
Logged
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« Reply #8 on: July 23, 2009, 10:38:47 AM »

Dan,

You're right.  After thinking about it a bit, I don't need to alternate my Archive disk updates.  Backup rotation is standard IT practice to enable recovery from things like accidental erasure or replacing an older file with a corrupt copy. But I only add to my Archive disks, so I can simplify my process a bit by removing Archive disk alternation - thanks!

I don't depend on my images for my livelyhood, so I can tolerate more risk than Peter can.  I've adapted his 3-2-1 rule and other parts of his process to better fit my needs.  I aggressively prune my images before I archive them, but I do keep 3 copies, including one off-site one.  And I've elected to go without optical media which saves time and reduces complexity (e.g. the archive and main photo disks are organized identically - no buckets).  I have move risk of losing images, but I find the incremental cost to fix this is higher than the value of the potential benefits - for me.  Of course, I'm not going to have to explain to the bride why there are no pictures of the kiss.  Wink

Dan
Logged
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« Reply #9 on: July 23, 2009, 10:40:15 AM »

Peter,

The tool wasn't developed specifically for photographers.  It's a general purpose tool that works for any file type (e.g. audio and video too).  All it's doing is computing the hash code for each file and saving the code along with the file's path in a sequential text file.  So the bucket-burning scenario you ask about works perfectly with one significant disclaimer - Windows only!

Dan
Logged
peterkrogh
Administrator
Hero Member
*****
Posts: 5682


View Profile Email
« Reply #10 on: July 23, 2009, 12:43:22 PM »

Dan, Thanks.
Can you easily have it drop a copy of "the hash codes for this folder of images" into that folder?
Peter
Logged
Dan Zemke
Jr. Member
**
Posts: 51


View Profile
« Reply #11 on: July 23, 2009, 01:02:49 PM »

Peter - yes.

Logged
peterkrogh
Administrator
Hero Member
*****
Posts: 5682


View Profile Email
« Reply #12 on: July 23, 2009, 04:39:53 PM »

Cool - looks like I'll need to check this out before too long.
Thanks, Dan.
Peter
Logged
danaltick
Hero Member
*****
Posts: 1616

evaa-xdtb@spamex.com danaltick
View Profile WWW Email
« Reply #13 on: July 23, 2009, 04:52:32 PM »

Peter,

This prompts ths question: Do you feel it is important to validate the metadata in addition to the raw data?  Still a little unclear on thait.  Thanks.

Dan
Logged

WindowsXP, ImageIngester Pro, RapidFixer, IVMP 3, ACR4, Photoshop CS4, Controlled Keyword Catalog, Canon EOS50D
peterkrogh
Administrator
Hero Member
*****
Posts: 5682


View Profile Email
« Reply #14 on: July 23, 2009, 06:48:27 PM »

Dan,
I'm less concerned about metadata, exactly, than the ability to validate other file types.  Also, as I was recording some instruction recently, I wondered about the idea of embedding the hashes on the disk itself. It's kind of the macro version of the DNG validation hash. 

What it's mostly about is self-contained validation codes for  write-once media, no matter the content.
Peter
Logged
Pages: [1] 2 Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!