Your computer might not tell you when it fails

[Marc Rochkind]

I just finished spending most of a week tracking down a bug in a new version of ImageVerifier that turned out not to be a bug at all. It was a faulty memory module in my computer. In fact, it was ImageVerifier that first reported the problem and that allowed me to run experiments that eventually led to the solution. My Mac ran normally all the time and never complained. The whole story is on my blog (http://basepath.com/index-real.php?url=blogentry/2007-12-20.htm).

Some lessons I've drawn from this experience are in a separate blog article (http://basepath.com/index-real.php?url=blogentry/2007-12-20a.htm).

To summarize: Critical computer work, which for most people would include the work that earns them their living, ought to be done only on computers with ECC (error checking code) memory. This rules out all Macs other than MacPros, essentially all laptops, and most desktops. You should never ingest on any computer without ECC memory, and should never use a non-ECC computer for archiving, backup, or restoring. Processing an image is perfectly OK on a non-ECC computer, since if the image is screwed up you'll notice.

Dell and other vendors do make reasonably-priced desktops with ECC memory, but you do have to pay a bit more. A MacPro is a lot more if you don't really need a MacPro, but maybe not that much more if you really need the quad cores and the expansion capability.

--Marc

[peterkrogh]

MArc,
Thanks for the write-up.  It looks as though you've uncovered a pretty significant industry-wide problem here.

I's assuming that you ran TechTool and it did not spot the bad memory. Do you think there is a program out there that would spot the problem like IV does?

I'm warming to the idea that an expanded ImageVerifier should be used for all data transfers.  As I've been looking through my collection, and running data validation, I'm running into a surprising number of invisible problems. - ones that don't flag any other way. (Drive utilities don't always spot them, images can look fine in Bridge, Camera Raw, iView even though there are significant corruption problems with the files.

I'd really like to be able to use IV for data transfers.  Right now, I'm transferring with Chronosync, then running Toast Compare, and them running IV when time permits. I think the error checking on the Chronosync and Toast are pretty rudimentary (date and file size checking).  I'd like to set up IV to do the file transfer, and then do a hash compare once it's done.

Maybe we should get on the phone soon and talk about how IV could add additional features for general file transfers.

Peter

[danaltick]

Marc,

Some would argue that ECC really only pays dividends in mission-critical sytems where gigabytes of data are continuously flowing through the system and the data integrity is critical to operation or security.  I wouldn't consider a home business or DAM mission critical in that sense.  It is true that home PC's do much more today than just a few years ago, and ECC may prove viable in certain situations, but I would say at this point, that would still be the exception, not the rule.  I tend to put ECC vs. non-ECC in the same category as JBOD vs. RAID for DAM applications.  It is true that ECC can detect both hard and soft errors, but if your memory is faulty, it should be replaced.  If you purchase good quality memory and power supply for your business PC or Mac, it will be very very rare that you will have hard errors within the lifespan of the memory (usually 7 years or more).  Your PC or Mac will be replaced by then.  Soft errors are usually a result of cosmic radiation and happen very infrequently even for 24/7 operation with gigabytes of data continuously flowing.  If it is important to detect these types of errors and attempt to fix them, then I would consider this a mission critical application.  That is my current thinking, but this topic is certainly debatable.

Actually, what I would recommend; especially for DAM, is to periodically run memory diagnostic tests on your PC or Mac.

On a side note, if anyone is looking for a good file comparison/synchronization tool for the PC, this is the one http://www.donationcoder.com/Reviews/Archive/CompareTools/index.html.

Dan

[peterkrogh]

Dan,
I know I'm a little on the neurotic side about this, but once the data is called into question, (Marc's "wrong answer") you end up going down a path that ends with buying a film recorder and making transparencies of the best work...

I've been bitten by data copying errors a number of times recently.  If I were not testing data integrity as part of a study process, I would not be seeing it - the "wrong answer"  I'm getting is that the system reports the data to be fine.

I think this argues pretty strongly for a hash attached to each file that can be portable and indicate file integrity with certainty across multiple machines.  It would be even nicer if you could aim it straight at the parts of the file that should not be changing, like the mosaiced camera data, and/or the embedded proprietary raw file.
Peter

[danaltick]

Peter,

I think a hash attached to each file would be a good idea.  Is that something that would be added to the DNG spec?

My guess is though, the file copy problems you are experiencing are not a result of hard or soft memory errors, but rather a result of problems with media storage.

By the way, this is the tool I use for memory diagnostics and stress testing when overclocking http://en.wikipedia.org/wiki/Memtest86%2B.  It's very good, and free.

Dan

[Marc Rochkind]

Dan and Peter--

Just got onto this thread.

I don't know about memory diagnostics. Failure modes can vary, and with such a huge number of combinations and various patterns of access I'm not sure it's practical to check memory. Recall that my computer ran for months (perhaps even a year) with a serious problem that only very rarely caused anything noticeable to be wrong (other than the inconsistent IV results), and I don't know that even those things that were noticeable (e.g., occasional crash) were due to the bad memory, since computers crash for lots of reasons other than faulty hardware.

That said, certainly running memory diagnostics can't hurt, and might even reveal a problem.

I did run TechTools Deluxe, but its memory checker is very quick and not thorough.

Dan: Do you know of a serious memory checker for Macs?

As for having IV do the copies: That's not something I'm planning for right now. First I want to get the present version finished. I just put in the ability to verify PSDs, and brought the Windows version up to the level of the Mac, and I'm planning to release those two versions this week.

--Marc

[danaltick]

Marc,

You can give this a try http://www.memtestosx.org/.

Dan

[Richard Anderson]

Marc,

Would having more RAM increase the probability of RAM error? Would this be the memory equivalent of running raid 0?

Richard Anderson

[Marc Rochkind]

Richard--

RAID 0 takes the probability of failure of 1 drive and doubles it to give the probability of failure of either of two drives, which (for interleaving) means that the entire (doubled) drive has failed. However, it doesn't change the probability of a block failing.

Similarly, increasing the RAM, assuming that all RAM has the same probability of failing, doesn't change the probability that a particular bit or word has failed. It does double the probability that SOME bit or word failed somewhere.

To say it another way, if the probability of a 512MB computer failing somewhere (due to memory) is P, then increasing the memory to 4GB increases the probability of a failure somewhere to 8P.

--Marc

[Marc Rochkind]

Dan--

Thanks for the tip. I downloaded the app. As there is no trial apparently, I had to go ahead and pay full price. The good news is that full price is only $1.39.

It would be interesting for me to reinsert my bad module and run the tester, but I'm not going to, as this is my main system and I don't want to horse around with it unnecessarily.

--Marc