Confirming file integrity - Mac OS 10.45

[Joe Reifer]

I recently upgraded from Mac OS 10.28 to Mac OS 10.45 and had some interesting discoveries about RAW file integrity.

I have previously been using a workflow for Canon CR2 and CRW files that only includes iView Media Pro and Photoshop.

I read the DAM Book, and realized that looking at thumbnails in iView (with prefereces set to use the embedded preview instead of building a new one) does not confirm the integrity of the actual file. Opening the folder of images in Bridge confirms the file integrity because it builds the preview from the original. A good way to identify any corrupted RAW files.

I am not converting to DNG yet, but have read the DNG converter is also a good way to confirm file integrity.

When I upgraded to OS 10.45, now the operating system can render previews of my RAW files. I recently embarked upon a project to make yet another backup of the files on my machine by moving them to a second external Firewire drive. When I went to copy the files to the external drive, not all of them would copy over. Apparently the Mac OS will error out if the files are corrupted. This happened both when I tried to just drag and drop files from my HD to the Firewire HD, and when using the SuperDuper! backup program.

Here's a couple of interesting discoveries. After identifying a few corrupted CR2 files, I tried running them through the DNG converter to see what would happen. The files converted to DNG with no error message, both from Adobe Camera RAW, and in the Adobe DNG converter.

OK - Are you still reading this? Sorry for being long winded. To review, here is where file integrity is confirmed in my current workflow:

1. Files transferred to computer, open the file folder and OS 10.45 will not show a preview for RAW files it cannot read.
2. Files copied to an external Firewire harddrive. Again, when moving the files, the OS checks integrity.
3. Files opened in Bridge, builds the preview from the original file, confirming integrity. I am not using Bridge until migrating to a DNG workflow - if I open my files in Bridge, and don't convert to DNG, I am saddled with XMP sidecars, which I don't like.
4. Instead, I'm considering using iView with preferences set to render its thumbnails from the original file instead of the embedded preview. This will confirm file integrity, but of course take longer to import.

*Note: Again, I was able to run corrupted RAW files through the DNG converter without any errors. These were files where the preview in Bridge showed up, but the colors were all screwed up (maybe the camera was turned off when the file was still writing?). I have not tried the DNG converter on files that did not show a preview at all.

If you think this email is long, you should've hung out at my house last Saturday when it took me 10 hours to figure all this stuff out.  

Cheers,

Joe

[peterkrogh]

Joe,
First, are you using the DNG converter 3.3?

Second, what are you seeing exactly?  Were the files ever opening normally by Bridge, or did this bad color come in sometime after the original download?

Can you adjust the files in Camera Raw to look normal?

Have you tried Disk Warrior on the drive?

iView will sometimes build from preview if the RAW data is corrupted, I believe.  If the file is a total loss, then iView shows an icon.

I am in the middle of a data loss problem (good thing for backups, which seem to be fine) which I will post about here shortly.

One thing I will say: Run Disk Warrior after every crash.
Peter

[Joe Reifer]

Oops - Thanks for moving this post to the Loss & Recovery area.

Yep - using the latest DNG converter - 3.3

Some files just appear to have weird color, others are just not appearing at all. Luckily, I'm not talking about a ton of files here. The strange colored preview or lack of preview appear to be consistent between the OS 10.45 finder, Bridge, and iView.

Because I was only using iView 2.64, Mac OS 10.28, and Photoshop for a long time, I never had anything in my workflow that confirmed file integrity unless I opened the file in Photoshop to edit. Now that I am experimenting with Bridge, the DNG converter, and especially since I'm on OS 10.45, I am all of the sudden identifying files I didn't know were corrupted.

I will check out Disk Warrior, but I don't think the HD is the problem. I also have DVD backups off-site, which is a relief.

Cheers,

Joe

[peterkrogh]

Joe,
I actually started this section in response to your post (and because I am in  a similar situation myself.)
Let's take one thing at a time.
You have files which the DNG converter will "convert" but the resulting DNG file will not open in Camera Raw?  If this is true, I'd probably want to take a look at that file.

Also, can you find any earlier versions of the file, in whatever form?

Do you know what applications have touched the file in the past?
Peter

[Joe Reifer]

For a couple of files where the color has gone alll weird (as shown in the thumbnail preview and in the file when opening in Camera RAW), the DNG converter will convert them just fine. I am calling these files "kinda corrupted." Or perhaps I have invented a new digital Polaroid transfer process? 

Haven't tried the DNG converter with files that have no thumbnail preview at all (these are definitely corrupted). These won't open at all, so I assume they won't convert to DNG. I am going to attempt to restore from backup DVD - but have been busy working way too much at my non-photographic day job. (Double    )

The only apps that have touched my files are: iView Media Pro (version 2.x and now 3.02), and Photoshop (7, CS, CS2), and in a very few cases Bridge.

My next task is to identify and isolate anything that's corrupted on a folder by folder basis.

[Joe Reifer]

I set my iView preferences to not use the embedded preview, and have been rebuilding my iView thumbnails bucket by bucket starting in 2003. There have been a few hiccups and a few more files with weird colors, but only 6-8 corrupted files so far.

Discoveries
 - iView 3.02 running under OS 10.45 using the Apple engine to build RAW files previews directly from the file works much faster than I thought. RAW thumbnails build faster than JPEGs!
 - I rebuilt a large number of JPEGs in iView (coupla thousand) and had about 200 where the preview didn't fully build, or just showed up as gray. I isolated these images and rebuilt them again in iView and they were fine. Next time I will rebuild JPEGs in smaller batches, and make sure not to scroll at all in the iView Thumbnail view while they are rebuilding.
 - Some of my file transfer hangups may have been due to old PSD files created in Photoshop 6 in OS 9 - I can't open these files in Photoshop CS2. These are film scans, and luckily my film is pretty organized, and we're not talking about a ton of files.

I hope someone finds this information useful. If not, at least it's therapeutic for me!

[jeremyrh]

I hope someone finds this information useful.

Yes - it is useful. In a way this website is half-devoted to the concept of assuring file integrity, so war stories are valuable and interesting.

It's also worrying, as, following a small catastrophe of my own [1], I added a step to my workflow to confirm file integrity by creating DNGs [2]. If that is not, in fact, diagnostic, then I'll have to think again!

[1] Apparently synching annotations from iVMP to NEF files resulted in some corruption - at least that's my guess right now.

[2] Following the LOCKSS principle - Lots Of Copies Keeps Stuff Safe

[peterkrogh]

It's also worrying, as, following a small catastrophe of my own [1], I added a step to my workflow to confirm file integrity by creating DNGs [2]. If that is not, in fact, diagnostic, then I'll have to think again!

[1] Apparently synching annotations from iVMP to NEF files resulted in some corruption - at least that's my guess right now.

Unlikely.  iView just uses the SDK from Nikon, so I'd be pretty surprised if this was the case.  Sounds to me like file system corruption or possibly a transfer problem. 

Joe, do you have any files that you could spare that either:
1. convert with no error message but have no image
2. Convert with no error message but have strange color

Peter

[Joe Reifer]

Update:

1 - Turning off Norton AntiVirus software when moving files solved a TON of problems. I was able to clone my hard drive to a new bigger hard drive, and also clone a Firewire drive to another Firewire drive without any errors. I used SuperDuper! and it worked great. When I attempted to copy or clone some of these files before I got errors which led me to believe I had more corrupted files than I really did. So turn off your anti-virus software when cloning drives!

2. I ran Disk Warrior on all of my drives. This solved a number problems including a pesky Firewire drive where I couldn't empty some files out the trash, and what I thought was a renaming problem but turned out to be a directory structure issue. So lesson 2 is: Disk Warrior is the best $100 I've spent in a long time.

3. RAW File corruption: Have you ever opened a RAW file in Camera RAW, and the colors look strange while it's building the preview? Somehow I had a few files where the preview didn't build all of the way and the colors looked weird. After step 1 & 2 above, these files seem to be building OK in Photoshop. I made DNGs of them which look fine, too.

Peter - I'm happy to send you an example of a RAW file that has the weird colors as it builds. I'm pretty convinced that Disk Warrior fixed whatever issue I was having though. What a relief. 

[peterkrogh]

Joe,
So the files that the DNG converter confirmed are in fact okay?
That first thing you are seeing is probably an embedded thumbnail/preview, rather than a CR-created rendering of the file. 
Peter

[jeremyrh]

It's also worrying, as, following a small catastrophe of my own [1], I added a step to my workflow to confirm file integrity by creating DNGs [2]. If that is not, in fact, diagnostic, then I'll have to think again!

[1] Apparently synching annotations from iVMP to NEF files resulted in some corruption - at least that's my guess right now.

Unlikely.  iView just uses the SDK from Nikon, so I'd be pretty surprised if this was the case.  Sounds to me like file system corruption or possibly a transfer problem.  

Peter

Unlikely, perhaps, but since it definitely wasn't a transfer problem (I opened and edited some of the files after transfer), and the file was recoverable by tinkering with the header, the alternatives are limited.

[peterkrogh]

What applications refused to open the file?  Nikon Capture? Photoshop?
Do you know what field you tinkered with to fix the problem?
Is it repeatable?
Peter

[jeremyrh]

What applications refused to open the file?  Nikon Capture? Photoshop?
Do you know what field you tinkered with to fix the problem?
Is it repeatable?
Peter

Hi Peter:
The gory details are at http://www.nikonians.org/dcforum/DCForumID36/15789.html , but basically nothing would open the image. The header was fixed by the kind intervention of another reader.
cheers,
Jeremy