Thousands of corrupted images

[Jonathan Lipkin]

I just found that I have several thousand corrupted images. Luckily, I have multiple copies archived on drives and DVDs and can recover them pretty easily. The question is, how to do this, how this happened, and how to avoid in the future.

I have v.2 of DAM book on order (lost my v1 copy, but it's probably dated..), so I hope to avoid this in the future.

Here's the story:

1. Using Mac os 10.6.3 on 2x1.8ghz mac pro, following workflow based on DAM book v1, more or less: Ingest using LR into folders, burn to DVD when the folder reaches 4.6g.
2. Last month, I decided to move my images from internal SATA drives to an eSATA RAID 5. I copied the files to external drives temporarily, removed the internals and used them to make the RAID. All seemed to go well- no copying errors.
3. Last week, LR told me that a few files in a collection were unreadable. Sure enough, they wouldn't open in PS either, nor bridge, Expression Media won't build a preview, etc. They appear to be the same file size as non-corrupt images.
4. Luckily the archived copies are fine, so I simply have to restore from the drives/
5. I tried using DNG converter to verify the images, but it doesn't recognize them as image files. In other words, when I point it at a folder containing the corrupt files, it gives me an error message stating that the folder contains no image files. So, if I point it at the enclosing folder, it will simply skip the corrupt files. There is no error stating that there was a problem converting the file.


So my questions are:
1. How do I verify that the files on my RAID are corrupted or intact? Bridge? Expression Media?
2. When I do find and replace the corrupt files, how do I ensure that my LR adjusments are copied into the DNG data?
3. I will await v2 of the dam book to see how to copy files and verify
4. Do I need to occaisionally verify images on the RAID? and how?

---------------------
Jonathan Lipkin
http://www.jlipkin.com

[Jonathan Lipkin]

To be more specific about verifying: I can't tell if an Expression media (I'm using v.1) catalog looks at actual image data or simply builds a preview through the thumbnail. Using bridge to go through 165 folders seems like it would take a long time, and Show Items from Subfolders doesn't seem to work.

Also, anyone having problems with EM? It seems to crash a lot and I can't find an upgrade path.

[Jonathan Lipkin]

PS, to add to the story, it looks like there are many many directories with corrupted images, and by that I mean that every file in the directory is corrupted, including jpg, dng and avi. About forty directories hold nothing but corrupt files. So it's not a bunch of scattered files, as far as I can tell, but as I've said, Im not sure how to truly verify images.

[peterkrogh]

Jonathan,
(Sorry for being away so long - it's been a crazy last few months)
So, were you able to recover all the files?
When you transfer, do you use a validated transfer?
I'm suspicious that one of the transfers went bad, or that the media is failing.

Please update us on the situation.
Peter

[Jonathan Lipkin]

Peter - Thanks for posting. In short, yes I was able to recover the images, mostly because I had a pretty rigorous backup and archiving system based on your first book. I simply copied from an archived HD. I think it was a transfer error, as I was not using a verified copy system. I now have a copy of your new book and Chronosync and will use that to copy files from now on.

Do you have any recommendations for verifying images? ImageVerifier seems to work pretty well but gives a lot of false positives (image is corrupt when it is not, and I suppose is better than a false negative) and given that my archive is 100k images, a 1% false positive rate still gives me a thousand images to check. Generating a hash is only good for archived images - writing data to the DNG changes the file. 

I think you mention using Expression Media - does it look at actual image data or only the thumbnail? If not, how would you recommend validating files that IV flags as possibly invalid? I think IV generates a text file log, so I could probably write a shell script that would open the files in a program like EM.

[peterkrogh]

Jonathan,
The DNG files are easy - no need for Image Verifier.  Just use the DNG converter as outlined in the movie on this page:

http://www.dpbestflow.org/data-validation/dng-validation

As to regular derivative files, the only way to validate for certain is to have a copy of the files that does not get updated.

Glad to hear that everything was recovered.  This could have been disaster.
Peter

[Mathew Farrell]

Jonathan, did you ever diagnose the culprit? I'm experiencing a very similar problem.

I can't verify yet, but it sure seems like once-fine files are becoming damaged in recent days. My big frustration is that I've just moved interstate, and backup HDDs and DVDs are still weeks away.

Mat

[Jonathan Lipkin]

Matthew --

I think that my problem was a transcription error - files got corrupted during the process of copying from one drive to another. I always use Chronosync's verified copy when copying files now. I'm not sure what happened in your case. I'm assuming that you've run disk utility and or disk warrior, although I'm not sure that would solve any of the problems you're having. I think the files themselves have become corrupt. Can you post some more details? What operating system, what type of storage medium, uncorrupted files are becoming corrupted, etc?

[Mathew Farrell]

Hi Jonathan.

Cheers for the thoughts, and I'm glad to hear you're back in action.

I started a fresh post on this: http://thedambook.com/smf/index.php?topic=5227.0 outlining the details. I'm starting to suspect something similar to you, as I did migrate files a short while back without verifying (similarly now using CCC). It is conceivable that I'm only now finding problems that have been there since then. There are still too many frustrating unknowns at the moment. It's a frustrating position as any testing or experimenting, as well as taking time away from actually making pictures, also means walking further down the road with an untrustworthy system.

The chaps at work are testing the suspect drive in a machine so we can read SMART data (unavailable to me through the firewire external enclosure). It mounts and works 'fine', but they weren't able to get it to even show up in Disk Warrior... Too early to read anything into this though.