Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
ImageVerifier now works effectively on non-DNG raws
The DAM Forum
Welcome, Guest. Please login or register.
December 05, 2020, 10:33:06 AM

Login with username, password and session length
Search:     Advanced search
28033 Posts in 5147 Topics by 2903 Members
Latest Member: kbroch
* Home Help Search Login Register
+  The DAM Forum
|-+  Software Discussions
| |-+  ImageIngester and ImageVerifier
| | |-+  ImageVerifier now works effectively on non-DNG raws
« previous next »
Pages: [1] Print
Author Topic: ImageVerifier now works effectively on non-DNG raws  (Read 3257 times)
Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« on: March 13, 2007, 12:46:44 PM »

I just uploaded beta 3, which now uses DNG Converter to verify non-DNG raws. On one test of 100 defective images produced by "punching" holes of 40,000 zero bytes at random points, ImageVerifier now reports 88% of them as invalid, which is exactly the same result produced by DNG Converter running standalone. The previous method, using the open-source dcraw, detected only 1%.

For DNGs, IV uses Adobe's DNG SDK (software development kit). It produced exactly the same results on a test of 100 defective DNGs as did DNG Converter, and runs twice as fast, so I'm keeping the internal method for DNGs.

Detection of invalid JPEGs was even better (98% or something like that). The very high compression of JPEGs makes even small errors detectable.

Detection of invalid TIFFs with an 800,000 byte hole punched in them was poor: Only 13%. However, 6MB TIFFs with 100,000, 200,000, 400,000, and 800,000 bytes lost from the end (i.e., truncated files) were all reported as invalid. Perhaps truncated files and files with their headers clobbered are the most commonly found bad files, and those are all detected.

(For some additional insight into the problem of detecting invalid files, imagine a file format that consists of a 4-byte width, a 4-byte length, and then width-times-height 3-byte RGB pixels. In this arrangement, as long as the width and length accurately describe the file, since any RGB combination is valid, no holes punched into the file would be detectable unless they affect the first 8 bytes. That's the worst case, and it's close to the TIFF case if no compression is used.)

--Marc
« Last Edit: March 13, 2007, 12:52:31 PM by Marc Rochkind » Logged

peterkrogh
Administrator
Hero Member
*****
Posts: 5682


View Profile Email
« Reply #1 on: March 13, 2007, 08:39:45 PM »

Marc,
Is the "hole" in the image data?  What do these look like when you open them?
Peter
Logged
Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« Reply #2 on: March 14, 2007, 09:54:12 AM »

Interesting question, Peter. For most of the files with "holes" in them (sequences of zero bytes written on top of what's there), they don't open, since the software I have to open them is very particular about errors.

A lot of the overwriting is into image data, since the holes are written at random points in the file.

I'll try to find one of those that DNG Converter didn't complain about and see what it looks like opened.

--Marc
Logged

Marc Rochkind
Hero Member
*****
Posts: 1136


View Profile WWW Email
« Reply #3 on: March 14, 2007, 12:43:50 PM »

Peter--

Here's an example of a bad image.

The holes were punched by writing 40,000 bytes of zeros directly into the image file at a random location. For 100 defective copies of a 10.5MB file, ImageVerifier detected 63 as invalid. The remaining 37 are bad files, but still apparently meet the DNG specification. I looked at a few of them in ACR and Lightroom. I couldn't see anything wrong with some of them, which means that the damage didn't affect anything important (missed any vital organs).

For the ones where I could see damage, it appeared as a rectangular defect:



Note that this image was one that IV called valid, and which ACR and Lightroom opened without a complaint. I would call it bad image data, but not damage that violated the specs for a DNG.

Also, everyone should realize that the damage I artificially inflicted is probably not typical of the the damage that actually occurs to images. I don't have any profile of what typically damaged images are like. It would be nice to build a collection of these mutants. Maybe anyone who has any could send them to me?

--Marc
Logged

Pages: [1] Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!