The DAM Forum
Welcome, Guest. Please login or register.
October 30, 2014, 08:10:11 AM

Login with username, password and session length
Search:     Advanced search
28020 Posts in 5140 Topics by 2910 Members
Latest Member: kbroch
* Home Help Search Login Register
+  The DAM Forum
|-+  DAM Stuff
| |-+  Software Discussions
| | |-+  Is Disk Defragmentation Dangerous to Your Data?
« previous next »
Pages: [1] Print
Author Topic: Is Disk Defragmentation Dangerous to Your Data?  (Read 12493 times)
Jake Livni
Newbie
*
Posts: 5


View Profile
« on: October 13, 2010, 08:05:02 AM »

After reading in The DAM Book about the need for file verification when copying files, I started wondering about the dangers of defragmentation.  I have to verify a copy of my entire photo archive (topic for another posting) and was thinking about errors creeping in while copying files and other ways that files can become damaged.

Hard disks typically become fragmented over time, as files are created and deleted.  Both individual (and especially large) files as well as complete folder structures containing many files can become fragmented.  Disk fragmentation can lead to lower performance as disk heads have to travel further to access files, possibly adding to wear and tear, too.  Operating Systems typically include a disk defragmenter utility to clean up the disk, to a greater or lesser some extent.

However, it seems that this could be a problem.  The process of Defragmenting involves finding files that are fragmented, finding an appropriate place elsewhere on the disk to fit that file, copying the file to the new location and then deleting the original file.  (This is called, not surprisingly, moving the file.   Smiley )  If there really are issues with non-Error-Correcting RAM, media degradation, connector problems, bus problems, etc. that cause you to lose sleep over simple copies of files, then it seems to me that defragmentation should be a worry, too.

Worse yet, with defragmenters, you end up with just a single copy of your original file and don't have the original to compare it to.  If moving the file induced an error, you have lost data - forever.

In practice, millions of computer disks have been defragmented and I haven't heard reports before of this being a danger.  On the other hand, lots of things don't work correctly on millions of computers and the techies often have no explanation...

I am wondering if anyone has an alternative opinion.  I don't think that Peter Krogh mentions defragmentation as a problem in The DAM Book.

Jake

Logged

Canon, Nikon Coolscan, Win XP
Jake Livni
Newbie
*
Posts: 5


View Profile
« Reply #1 on: October 13, 2010, 08:17:59 AM »

Incidentally, the way some large IT shops used to handle disk defragmentation was to copy an entire disk onto something else (e.g. tape) and then load the entire tape (or series of tapes) back onto a clean, empty disk.  When done correctly, the directory and file structures are copied cleanly onto tape in an organized manner and then load back onto the disk without fragmentation.  (In principle, it should be possible today to do this directly from disk to disk.)  Of course, it's nice that some Operating Systems can Copy and Verify with Read after Write, an option that Windows doesn't seem to offer.   Sad 

My first tests with a Samsung Story Station external disk show that the included Samsung backup software produced a very fragmented backup copy, even onto a clean, empty, new disk.  I don't know why they did it like that; it shouldn't have been that hard to write the software correctly.  Sad

Jake

Logged

Canon, Nikon Coolscan, Win XP
rogerhoward
Full Member
***
Posts: 103


View Profile Email
« Reply #2 on: October 13, 2010, 02:50:18 PM »

Jake - absolutely, a simple (and nicely redundant) method for defragging data files is to copy to an empty disk and back. Defragging happens at the OS level today, within limits - usually only files up to a certain size, but this effectively keeps the OS itself defragged (and with optimum placement of files on disk - not the same). Frankly, I don't think defragging today is worth the trouble in the vast majority of cases - it's long been mainly a placebo, except in edge cases where you need to wring every bit of I/O performance out of a disk, such as with huge streaming writes like uncompressed audio/video capture.

As for the reliability of the defrag, there is verification during writes, so I don't think defragging (by most modern tools) is particularly risky in that regard; most file corruption happens silently during file dormancy (where it's sitting quietly by itself, and bits are randomly flipped because of a bad sector), not during activity such as copies where there is verification done at the controller/driver level to insure bit-accurate copies (if there weren't, we'd be in deep trouble every day).

Like so many other bits of old conventional wisdom, I think the need to defrag (or frequently repair) filesystems is largely gone today; as often as not, these tasks cause as many problems as they may fix. I do regularly filesystem integrity checks, and keep good backups, but I don't proactively defrag or repair disks that don't need it.

- Roger Howard
Logged

-----

Roger Howard
Jake Livni
Newbie
*
Posts: 5


View Profile
« Reply #3 on: October 14, 2010, 05:54:53 AM »

Thanks for your insight, Roger.  Perhaps Disk Fragmentation isn't really that critical an issue today in most applications.

As for the reliability of the defrag, there is verification during writes,

Everything I've seen recently on Windows XP says that the OS does NOT verify during copies.  The (admittedly) old DOS Copy command has a /Verify switch which does not actually verify the copy at all.  See: http://support.microsoft.com/kb/126457  I'd be a bit surprised if the Defragment Copy action was that different from other Copy action (e.g. of files or folders from within Windows Explorer).  If Windows would easily Copy with Verify, we wouldn't have that much need for third party Image Verification applications. 

Quote
[...] most file corruption happens silently during file dormancy (where it's sitting quietly by itself, and bits are randomly flipped because of a bad sector), not during activity such as copies where there is verification done at the controller/driver level to insure bit-accurate copies (if there weren't, we'd be in deep trouble every day).

TheDAMBook (2nd ed) calls transfer corruption, i.e. damage resulting from moving data from one place to another, one of the most common causes of file corruption. If copies are a source of trouble, then defragmentation could be, as well.  If not, not.  All the same, corruption during dormancy is a good reason to periodically check data, even if it hasn't been accessed or hasn't been on-line, so you make a good point.

This brings me back to my original problem of verifying a copy of an entire photo archive before deleting the original.  I'm still looking for a simple (freeware?) solution for that one.

Jake
Logged

Canon, Nikon Coolscan, Win XP
rogerhoward
Full Member
***
Posts: 103


View Profile Email
« Reply #4 on: October 14, 2010, 11:11:58 AM »

I'm not an expert in disk controllers, but my understanding is the verification happens at the controller level - the OS hands data off to the drive controller to be written, and there's essentially a contract with the controller requiring perfect reads and writes. I'd say it'd be highly unusual for local copies to fail with silent corruption; I'd love to see a test demonstrating this happening. If such corruption is possible at this level, it's enormously unlikely given the massive amount of data reads/writes our machines perform every day without issue.

The only kind of copy I can imagine where the writes are significantly at risk of in-transit corruption is over certain network protocols, where the in-transit data may get corrupted without detection; the OS on the receiving end has no way of knowing this, and so it writes the data as-received from the network stack. FTP, for instance, is a historically common source of this kind of corruption, but with most network protocols its far less likely these days.

I'm skeptical that disk-to-disk copies are a significant source of data corruption but would love to see better data either way. As I said, I think the dormant corruption issues are far more prevalent. - files getting corrupted on disk due to uncorrected degrading of the disk/medium. These would be difficult to detect without either batch-based file comparisons (either to other copies, or to fingerprints like md5 digests) or filesystems which monitor and automatically correct on-disk corruption (ZFS, for instance).

- Roger Howard
Logged

-----

Roger Howard
stevod
Newbie
*
Posts: 7

Interior Designer


View Profile WWW
« Reply #5 on: July 26, 2011, 05:41:24 AM »

I thought the only need to defrag was to increase speed by reducing the number of separate reads per file.  But not all file systems need it anyway, and I have a feeling that the file system on my NAS is not prone to this anyway.

A note re SSDs though - these should never be defragmented due to the amount of wear on the drives which might reduce their lifespan.

S
Logged

stevod
Newbie
*
Posts: 7

Interior Designer


View Profile WWW
« Reply #6 on: July 26, 2011, 05:42:59 AM »

^which would certainly be dangerour to your data!

(Sorry to double post)

S
Logged

Pages: [1] Print 
« previous next »
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!