Raw corruption issue

[jakobox]

Recently, I've been having some issues with the occasional corrupted RAW file.

The corruption manifests as large blocks/bands of color over an object.

I'd say that I see this issue once in about 2,000 images. I shoot with two cameras, and the issue has been coming up with both of them.

On the occasion where I've caught the issue before I re-formatted the cards, I was able to go back to the card and re-download the image and open it without a hitch.

So that leads me to believe that it is most likely the card reader. A quick visual inspection of the card reader shows that all the pins appear to be straight and I can see no visible problems. Or is it possible that it is the drive itself?

I have done some test runs with Image Verifier.. running it on a folder that includes one of the corrupted files, Image verifier does not find a problem with the file.

I am a bit stumped here... Can anyone help me troubleshoot, or give me any ideas what to look at? Thanks!

[peterkrogh]

Jake,
This one can be really tough.  When I had this happen to me, it turned out to be a drive enclosure. It took *months* to find.

Everything in the chain is suspect.

Can you outline the file path?

My system

Download through this card reader with this software
Saved to description of drive
Open with this software
...
Peter

[jakobox]

Thanks Peter.

I've had this problem while operating on 2 different file paths.

The first time this happened, I was ingesting from cards using Lightroom. LR was creating an automatic backup to a separate drive. The image was imported into LR, and not touched by any other software. The image and the automatic backup images were both corrupt (and in the same way).

Sometimes, for the sake of speed, I'll also download directly off the card (using Mac "image capture"), make backups from that, and do an initial edit in Photo Mechanic. In photo mechanic, the files do not show the corruption, but upon import into LR, they do show the corruption. The backup of the files (which have never been touched by LR, but I try to open with Photoshop or the Canon utility) also exhibit this issue.

A recent addition to my system has been an additional CF card reader (FW 800).. and for awhile I was completely convinced that importing the images simultaneously was somehow causing the issue. But then I realized that I saw this issue at least once before I got the new card reader (I compared the EXIF data to my b&h receipt  )

Also, don't know if this could be suspect, but I am using a Norco eSata enclosure. I've had it for a year+, and haven't had any other issues, but is that something I should be looking into?

I've removed the suspect USB card reader from the system, and I'm going to keep my eyes open to see if the problems recur.

Another couple questions in the meantime while I try to work this out: Is there any way to check folders of files to see if they might have suffered this sort of corruption (for example, that's what I was hoping that Image Verifier would do... is there any other software I should check out?). Also, thank goodness that this problem hasn't affected any truly critical files, but is there anything I could do to try to recover such a corrupted file in the future?

thanks!

Jake

[peterkrogh]

Jake,
Photomechanic is looking at the embedded preview, not the raw data, so it can show things fine even if the raw data is corrupted.

The uncorrupted JPEG preview means that the camera is capturing a good file.  It could be causing the error when writing to the card, but if you see this on more than one camera, that's unlikely.

If' you've swapped out the reader, and this happens on more than one card, I'd say the prime suspect is the drive.

Lots of information in the new book on data validation, which is the process you have in front of you right now.

Peter

[jakobox]

Thanks Peter.

Yes, what you say confirms my assumptions about the way Photo Mechanic works, and I wasn't surprised that even though the preview looked good there were problems with the file. Good to know that this helps to rule out the capture as being where the error occurred.

I ordered the new book from Amazon about a month ago (and read the first one quite some time ago, and found it to be very usefull).. Guess it's time to crack the new one open 

thanks!

Jake

[jakobox]

I've had a little more time with the problem, and some time to crack open the book 

I eliminated the original card reader, and discovered that I'm still experiencing this issue.. So that is to say I'm experiencing the issue with different cameras, card readers and cards.

I did a pretty through look through the chapters in the book about data validation. This is all very useful stuff, but as far as I can tell, it's more about identifying files with errors in them rather than diagnosing what caused them.

Since I am running into this issue at the rate of around 1 per 1500 ingested images, I'd really like to get to the bottom of it.

The book suggested running Disk Warrior. I did that, and it found the drive to be healthy.

You mentioned that you eventually were able to identify the problem as being from the drive enclosure (which I'm starting to suspect myself). Do you mind sharing how you worked that out in the end? Anything I can be doing to isolate where the issue is coming from?

thanks!

Jake

[peterkrogh]

Jake,
I found the problem by transferring files with a validated transfer (using Chronosync on the Mac). Every now and then, I would get an error in transfer when copying to a particular drive.  I would hit the "try again" button, and everything would copy fine. Eventually, I realized that the same thing that was causing the validated transfer to throw the error message was the thing causing the raw file corruption.  I removed the drive enclosure and all was fine.

To test, you might want to transfer a lot of data to the Norco drive and see if you get any "did not transfer properly" errors.  You will want to look at the program's preferences to make sure it does not automatically try again, as this will make the problem harder to spot.

If you are getting, say, one error in each 1500 files, and they are 20 MB each, then you would want to do 30 GB, on average, to show the error.  I'd set it up to do some large transfers overnight and see if the problem shows up.
Peter