I have been using ExactFile in the last few months and like it very much.  It works very nicely and does what I want - it gives me confidence that data on disk is precisely the data I put there and what I expect it to be.  Every single bit.

In principle, it's quite simple.  It calculates a Hash (MD5 or your choice of many others) for each file in a directory structure (including all sub-directories, if you choose) and saves the calculated hash together with the filespec on a single line of a simple text file (which they call a Digest).  The result is a text file, the Digest, (easily read in Notepad) that lists the files with the hashes, one line per file.  You can choose where to save this Digest. 

Then, at any time in the future, you can run ExactFile again against the Digest and it will again create hashes for each of the files listed in the Digest and also compare them to the hashes that were produced in the original run.  If everything matches, every bit is in the right place and ExactFile will report this.  If there are any discrepancies or missing files, ExactFile will alert you to which was the failing (or missing) file.  (If you've added a file to the directory somewhere and it isn't listed in the Digest, ExactFile will not spot this, so it isn't a complete mirror-checker.  However a simple "properties" check in Windows to check number of files and space-on-disk can confirm complete mirroring.)

MD5 is a good all purpose hash method, though it might mathematically be a touch less than 1,000%.  Other hash methods  may be more reliable or less so and may take more or less CPU to calculate.  For my purposes in photography, MD5 is more than satisfactory.  (I am not a mathematician by training.)

Something I haven't done but which is available with ExactFile is the ability to also create an applet to store with the Digest, say on optical media.  This way, you can distribute (or save) burned DVDs and let someone in another place (or time) run the applet to confirm that the files are all valid, even without installing the application again.  Neat.

I have used ExactFile a number of times to confirm the integrity of archive files after transfer from disk to disk and before deleting the originals.  I have used it on hundreds of GB of data - and it works just fine even when run in a single run with thousands of image files in a deep directory structure and with a hundred or so GB of data.  In my experience, it will on rare occasion flag a false negative error when there are sub-directory names in a foreign language (right-to-left text), claiming a file to be missing; a quick check of the Digest will show a one-char spelling error and that the file is indeed present.  (You can also create a single-file hash to check things manually on a single file.)  If you're using English-language file names (and character-set) throughout, I wouldn't expect this problem to come up.

When creating the Digest, be careful to UNCHECK the "Include Full Paths in Output" so that you can compare the source files to the target files stored in ANOTHER location.  Otherwise, when checking files, it will check the originals, only.  If that's what you want to do (e.g. on DVD), fine.  For file transfers, though, you want this UNCHECKED.

Yes, the copy I downloaded a few months ago seems to be Beta but it works very nicely.  The confidence it inspires in file transfers (or storage) is valuable.

Jake

Dan,

I want to try out ExactFile; it sounds like it does (did?) some of the things I need at the moment.
(I want to confirm the integrity of a copy of a photo archive of 250 - 300 GB before deleting the old archive.  It seems that ExactFile can do this.)

Do you have any updated advice on this since you last wrote about it?
Any warnings or suggestions?
Alternatives?

It sounds simple enough.

Thanks.

Jake

Thanks for your insight, Roger.  Perhaps Disk Fragmentation isn't really that critical an issue today in most applications.


Everything I've seen recently on Windows XP says that the OS does NOT verify during copies.  The (admittedly) old DOS Copy command has a /Verify switch which does not actually verify the copy at all.  See: http://support.microsoft.com/kb/126457  I'd be a bit surprised if the Defragment Copy action was that different from other Copy action (e.g. of files or folders from within Windows Explorer).  If Windows would easily Copy with Verify, we wouldn't have that much need for third party Image Verification applications. 


TheDAMBook (2nd ed) calls transfer corruption, i.e. damage resulting from moving data from one place to another, one of the most common causes of file corruption. If copies are a source of trouble, then defragmentation could be, as well.  If not, not.  All the same, corruption during dormancy is a good reason to periodically check data, even if it hasn't been accessed or hasn't been on-line, so you make a good point.

This brings me back to my original problem of verifying a copy of an entire photo archive before deleting the original.  I'm still looking for a simple (freeware?) solution for that one.

Jake

Incidentally, the way some large IT shops used to handle disk defragmentation was to copy an entire disk onto something else (e.g. tape) and then load the entire tape (or series of tapes) back onto a clean, empty disk.  When done correctly, the directory and file structures are copied cleanly onto tape in an organized manner and then load back onto the disk without fragmentation.  (In principle, it should be possible today to do this directly from disk to disk.)  Of course, it's nice that some Operating Systems can Copy and Verify with Read after Write, an option that Windows doesn't seem to offer.    

My first tests with a Samsung Story Station external disk show that the included Samsung backup software produced a very fragmented backup copy, even onto a clean, empty, new disk.  I don't know why they did it like that; it shouldn't have been that hard to write the software correctly. 

Jake

After reading in The DAM Book about the need for file verification when copying files, I started wondering about the dangers of defragmentation.  I have to verify a copy of my entire photo archive (topic for another posting) and was thinking about errors creeping in while copying files and other ways that files can become damaged.

Hard disks typically become fragmented over time, as files are created and deleted.  Both individual (and especially large) files as well as complete folder structures containing many files can become fragmented.  Disk fragmentation can lead to lower performance as disk heads have to travel further to access files, possibly adding to wear and tear, too.  Operating Systems typically include a disk defragmenter utility to clean up the disk, to a greater or lesser some extent.

However, it seems that this could be a problem.  The process of Defragmenting involves finding files that are fragmented, finding an appropriate place elsewhere on the disk to fit that file, copying the file to the new location and then deleting the original file.  (This is called, not surprisingly, moving the file.    )  If there really are issues with non-Error-Correcting RAM, media degradation, connector problems, bus problems, etc. that cause you to lose sleep over simple copies of files, then it seems to me that defragmentation should be a worry, too.

Worse yet, with defragmenters, you end up with just a single copy of your original file and don't have the original to compare it to.  If moving the file induced an error, you have lost data - forever.

In practice, millions of computer disks have been defragmented and I haven't heard reports before of this being a danger.  On the other hand, lots of things don't work correctly on millions of computers and the techies often have no explanation...

I am wondering if anyone has an alternative opinion.  I don't think that Peter Krogh mentions defragmentation as a problem in The DAM Book.

Jake