Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
Latest posts of: David C. Buchan
The DAM Forum
Welcome, Guest. Please login or register.
October 28, 2020, 10:54:49 AM

Login with username, password and session length
Search:     Advanced search
28033 Posts in 5147 Topics by 2903 Members
Latest Member: kbroch
* Home Help Search Login Register
  Show Posts
Pages: [1] 2 3 ... 5
1  DAM Stuff / Loss and Recovery / Scripts to verify media copied on Windows on: January 03, 2012, 11:23:33 PM
About a year ago I began playing around with MD5/SHA1 digests to check if files were being copied correctly between drives. Over many 10's of thousands of files I found this wasn't the case and there were always a few files that Windows said were ok, but which generated a different digest. For those unaware a digest is a signature string for a file. If even one bit is different in a copy the digest will not match.

I have developed two Windows Powershell scripts which I'm happy to share here on the proviso they are unsupported and you use at your own risk. I don't have time to publicly maintain them, nor even to document them very well, but as they are working for me I think others might like to know about them.

The basic process I use is:

1. Work with files on master copy drive.
2. Generate SHA1 digest for all files on a directory-by-directory basis. This results in a single checksum file per directory.
3. Sync from the master copy drive to a backup copy drive using Syncback
4. Compare the SHA1 digest for all files on the backup copy drive.
5. Recopy and repeat step 4 as necessary.

The script's use the command line version of ExactFile (free download) to generate the digests. Modify the script to point to it's install location. You will need to give your PC permission for the script to run under PowerShell using the Set-ExecutionPolicy RemoteSigned command. I figure if you care enough to run these scripts you'll be able to work it out. Note: By their very nature they will create (and sometimes) files on your drives - though only those relevant to the process.

In Windows, right-click and choose "Run with Powershell" or within Powershell using the . command as in ./[path]/hashcheck.ps1

Both scripts are smart and do their best to only recreate the checksum file if a directory is changed. Run hashcreate.ps1 to create the files and hash check.ps1 to check them. This second script creates two files in the directory you specify for checking. They are checksums-ok.txt and checksums-fail.txt. Across 1TB you don't want to be checking directories that have already passed muster. To re-check simply delete the checksums-ok.txt file. In checksums-fail.txt you'll find a list of all failed files. Once corrected re-run the check script and the whole directory will be rechecked.

## Creation script
  • Copy this code into a text editor and save as hashcreate.ps1
  • You can specify multiple directories when prompted by separating each with a space. Checksums will be created for the directory you specify and all below it
  • If run from the Powershell command line the -clean parameter will force a delete of all existing checksum.exf files
THIS CODE IS PROVIDED WITHOUT WARRANTY. USE AT YOUR OWN RISK.

Code:
param ([switch]$clean)

function runExf($path)
{   
    $timeStart = Get-Date
    $contents = Get-ChildItem -LiteralPath $path | where {!$_.PSIsContainer}
    if( ($contents | Measure-Object).count -gt 0 ) {
        [double]$sizeMB = ($contents | Measure-Object -property length -sum).sum / 1MB
        $s = "; Hashing {0:N0} files totalling {1:N3} MB" -f ($contents | Measure-Object).count, $sizeMB
        Write-Host $s
    }
    write-host "; @ $timestart"
    $exfOutput = "checksums.exf"
   
    ## Adjust the next lines to point to your install of exf.exe
    $exe = "path to\exf.exe"
    if ((test-path($exe)) -eq $False) {
        Write-Host -f white -b red "ExactFile cannot be found at $exe. Please check script."
        exit
    }
    $catchit = &$exe -sha1 -otf ("$exfOutput") -d "$path" *.*
    $catchit
    if( $catchit -match "; [0-9]+ files sucessfully hashed") {
        # Display the duration.
        $duration = (Get-Date)-$timeStart
        $s = "; {0:D2}:{1:D2}:{2:D2} at {3:N3} MB per minute`n" -f $duration.Hours, $duration.Minutes, $duration.Seconds, ($sizeMB/($duration.TotalSeconds/60))
        Write-Host $s
        Return
    }
    if( $catchit -notmatch "; [\*]{3} No Files Found") {
        # Something has gone wrong with the hashing. Exit.
        Write-Host -f white -b red "An error has occurred. Exiting. Please correct and re-run."
        Exit
    }
}

function hashDir($path, [switch]$isRoot)
{
    if( $isRoot ) {
        $statusPath = $path
    } else {
        $statusPath = $path.Replace($startDir,"")
    }
$files = Get-ChildItem -LiteralPath $path | where {!$_.PSIsContainer} | sort lastwritetime -descending | Select-object -first 1 | where {$_.name -eq "checksums.exf"}
if (-not $files) {
    # checksum.exf is missing from directory
Write-Host -f red "`n[??]" $statusPath
runExf($path)
} else {
if ($files.lastwritetime -lt $files.directory.lastwritetime ) {
#something has been modified the directory, even though checksumss.exf is the latest file
#could be a new file added, which has an older date
Write-Host -f red "`n[++]" $statusPath
runExf($path)
} else {
#checksums.exf is the latest file, and there are no directory modifications
#still have to check if lines in file matches lines in directory. Perhaps hashing was
#aborted due to halting or system crash
$linecount = (Get-Content -LiteralPath $files.fullname | where {$_ -match "^[a-f0-9]{40}\s\?SHA1"} | Measure-Object).count
$filecount = (Get-ChildItem -LiteralPath $path -force | where {!$_.PSIsContainer} | Measure-Object).count - 1
      if ($linecount -ne $filecount) {
Write-Host -f red "`[ne]" $statusPath
runExf($path)
} else {
        Write-Host -f green "[ok]" $statusPath
}
}
}
}

$startDir = read-host "Please enter a root (starting) directory"

if($startDir.length -eq 0) {
    $startDir = (pwd).path
    Write-Host "Defaulting root directory to $startDir"
}


$timeStart = get-date

if($clean) {
    Write-Host "Removing all existing checksums.exf files from directory and subdirectories"
    Get-ChildItem $startDir -include checksums.exf -Recurse | Remove-Item
    Exit
}

if(Test-Path -LiteralPath $startDir) {
    Write-Host "Checking root directory"
    hashDir -path $startDir -isRoot
    Write-Host "`nGathering sub-directories. Please wait."
    $subdirs = @(Get-ChildItem $startDir -Recurse | where {$_.PSIsContainer} | Sort fullname)
    Write-Host $subdirs.count "sub-directories found under $startDir`n"
        foreach ($subdir in $subdirs) {
        hashDir -path $subdir.fullname
    }

    $duration = (Get-Date)-$timeStart
    $status = "`n`nProcess complete; {0:D2}:{1:D2}:{2:D2} (duration)" -f $duration.Hours, $duration.Minutes, $duration.Seconds
    Write-Host $status} else {
    Write-Host -f red "'$startdir' does not exist. Nothing to do."
}

# The ReadKey functionality is only supported at the console (not is the ISE)

if (!$psISE)
{
Write-Host "Process complete"
    Write-Host -NoNewLine "Press any key to continue ..."
    $null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
    Write-Host ""
}

# Store for quick clean
# get-childitem -include checksums-ok.txt -recurse | remove-item
# get-childitem -include checksums-fail.txt -recurse | remove-item

## Checking script
  • Copy this code into a text editor and save as hashcheck.ps1
  • You can specify multiple directories when prompted by separating each with a space. Checksums will be checked for the directory you specify and all below it. Any failures will be in checksums-fail.txt

THIS CODE IS PROVIDED WITHOUT WARRANTY. USE AT YOUR OWN RISK.

Code:
function runExf($dir)
{
    $timeStart = Get-Date
    $contents = Get-ChildItem -LiteralPath $path | where {!$_.PSIsContainer} | where {$_.name -ne "checksums.exf"}
    if( ($contents | Measure-Object).count -gt 0 ) {
        [double]$sizeMB = ($contents | Measure-Object -property length -sum).sum / 1MB
        if( ($contents | Measure-Object).count -eq 1 ) {
            $s = "; Checking 1 file totalling {0:N3} MB" -f $sizeMB
        } else {
            $s = "; Checking {0:N0} files totalling {1:N3} MB" -f ($contents | Measure-Object).count, $sizeMB
        }
        Write-Host $s
        Write-Host $timeStart
    }
    ## Adjust the next lines to point to your install of exf.exe
    $exe = "path to\exf.exe"
    if ((test-path($exe)) -eq $False) {
        Write-Host -f white -b red "ExactFile cannot be found at $exe. Please check script."
        exit
    }
    $catchIt = &$exe -c (Join-Path -path $dir "checksums.exf")
    $catchit
    $duration = (Get-Date)-$timeStart
    $s = "; {0:D2}:{1:D2}:{2:D2} at {3:N3} MB per minute" -f $duration.Hours, $duration.Minutes, $duration.Seconds, ($sizeMB/($duration.TotalSeconds/60)) 
    Write-Host $s
}

function hashDir($path)
{
    # A [ or ] in the filename will trigger regular expression matching. We need to avoid that
    # Expand this pattern for any other characters which trip up your directories.
    $litpath = $path.replace("`[", "```[")
    $litpath = $litpath.replace("`]", "```]")

    if (Test-Path -path $litpath\checksums.exf) {
        Write-Host "`n[in]" $path
        $res = runExf( $path )
        if (($res -match "^No errors") -or ($res -match "^No files found")) {
            $path | out-file -filepath $outputOK -append               
            Write-Host -f green "[ok]" $path
        } else {
            ("[in] " + $path) | out-file -filepath $outputFAIL -append               
            $res | out-file -filepath $outputFAIL -append
            "" | out-file -filepath $outputFAIL -append
            foreach ($r in $res) {Write-Host -f red $r}
            Write-Host ""
        }     
} else {
        ("[??] " + $path) | out-file -filepath $outputFAIL -append               
        "" | out-file -filepath $outputFAIL -append
        Write-Host -f red "`n[??]" $path
}
}

function listFails() {
    if (Test-Path $outputFAIL) {
        Write-Host -f red "`nListing errors directory by directory`n"
        foreach($fail in (Get-Content $outputFAIL)) {
            Write-Host -f red $fail
        }
    }
}

$failsExist = $false

$startDirs = read-host "Please enter the root (starting) directories"

foreach ($startDir in $startDirs.split(" ")) {
    if($startDir.length -eq 0) {
        $startDir = (pwd).path
        Write-Host "Defaulting root directory to $startDir"
    }

    $outputOK = join-path -path $startdir "checksums-ok.txt"
    $outputFAIL = join-path -path $startdir "checksums-fail.txt"
    if( Test-Path $outputFAIL ) {
        Remove-Item $outputFAIL
    }

    $timeStart = get-date

    if(Test-Path -LiteralPath $startDir) {
        if( Test-Path $outputOK) {
            $okfiles = Get-Content $outputOK
        } else {
            $okfiles = $null
        }
        if(!($okfiles | where {$_ -eq $startDir})) {
            hashDir -path $startDir
        }
        Write-Host "`nGathering sub-directories. Please wait."
        $subdirs = @(Get-ChildItem $startDir -Recurse | where {$_.PSIsContainer} | Sort fullname)
        Write-Host $subdirs.length "sub-directories found under $startDir"
        foreach ($subdir in $subdirs) {
            if( $okfiles | where {$_ -eq $subdir.fullname })
            {
                Write-Host -f green "[ok]" $subdir.fullname
            } else {
                hashDir -path $subdir.fullname
            }
        }
        listFails

        $duration = (Get-Date)-$timeStart
        $status = "`n`nProcess complete; {0:D2}:{1:D2}:{2:D2} (duration)" -f $duration.Hours, $duration.Minutes, $duration.Seconds
        Write-Host $status
    } else {
        Write-Host -f red "'$startdir' does not exist. Nothing to do."
    }
}

# The ReadKey functionality is only supported at the console (not in the ISE)
if (!$psISE)
{
Write-Host "Process complete"
    Write-Host -NoNewLine "Press any key to continue ..."
    $null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
    Write-Host ""
}
2  DAM Stuff / Backup Strategies and Tools / Re: Recommendations for Hard Drive only Archive and Backup on: September 14, 2010, 04:29:02 PM
Hi Jordan,

It's been a long time since I've been on these forums but coincidentally you have exactly the same question I've got so let's see if we can work out a solution together.

For me it is also a case of time and money. For the sake of easy maths, 1TB takes approx 200 DVDs (no blu-ray for me as yet) at a cost of AUD$200, plus hours and hours of time. Though DVD backups have saved me once I can't justify the time and cost when I can purchase 2 x 1TB drives for the same amount, automate the process and have it run much faster.

I'm thinking of:

1.  2 x onsite copies, 1 x offsite as my main backups
2.  4 x offsite archive copies at 6 monthly intervals.

At the moment I use the free version of SyncBack on Windows which works fine. Genie Backup and Sync Tool weren't at all reliable if I actually checked what was copied.

For validation I've cut my own version of SHA1 digests. I'm using the commandline version of ExactCopy to create a hash digest for each directory. Validation steps are:

1. Create hash file for each directory
2. Sync to backup drive
3. Validate hashes

Sad but true, not all files copy bit for bit perfectly as the validation proves.

Now the problem with the validation tools such as ExactCopy is they can recurse directories but to be sure you have to recreate for each folder. That takes hours and should it fail you have to start again. I've created a couple of Windows Powershell scripts that get around this problem. ExactCopy is great because it uses each CPU core which makes it as fast as possible. I've wrapped scripts around that to only call it if needed. First time around is slow but then I only create the hash file for a directory if it is missing, is not the youngest file in the directory, or if the directory has been modified in some other way. Same for validation. If I've validated I create a small marker file otherwise I know I've done this dir and move on.

It's working well and I will get to posting the scripts on my blog soon - need to find the time to document. Because, with this system, you can backup and validate cheaply (time and $) I'll be moving away from DVD copies.

Cheers,

David
3  DAM Stuff / Backup Strategies and Tools / Re: Acronis True Image Home 2010 on: September 14, 2010, 04:17:38 PM
Acronis True Image Home 2011 is just out. Working ok on XP. Massive overhaul to the clumsy user interface is an improvement.
4  Software Discussions / Lightroom / Re: Both photos and videos in archive on: February 03, 2009, 09:21:07 PM
I've used IMatch for years to manage both together.
5  Software Discussions / Choosing Software/Other DAM Applications / DAM workflow in IMatch on: June 18, 2007, 03:09:55 PM
my IMatch describes how I brought what I learnt from this forum, Peter's book and iView to IMatch which which has always been my preferred cataloging software.

The article covers why I went back to IMatch after using iView, how I categorise and how I use buckets. I hope you find it useful, even if you use different software.

Regards,

David
6  General / GPS/ Geotagging / Re: Anybody geotagging? on: February 21, 2007, 04:32:25 PM
>I've found the 4 level IPTC fields (country, state/province/city and location) to be too limiting in number. To tie all photos to a single location can be too confining if the >location is relatively large (eg. shots taken around one of the world's disneylands).

David,
The Location is a Child of the City, State, or country (whichever is the appropriate next level up).  So Disneyland in Anaheim, California, USA is not the same as Disneyland in Baghdad, Iraq.

I have not had time to read this whole discussion, but there seems to be some valuable expertise in this group.  I will look forward to waht we come up with.
Peter

I too understood location to be child of a city - your wording helps me to explain. I'm talking about the child of the child. So whereas Disneyland is a location, Space Mountain is it's child. It may not be relevant from a IPTC point of view but for fairly "accurate" GPS encoding Space Mountain is a definate sub-location.

David
7  General / GPS/ Geotagging / Re: GPS Error/accuracy.. I got long winded here, your warned on: February 21, 2007, 03:20:53 AM
Not so much the long-windedness - the issue merits it - but the elusive style made me read the post mor[t]e than once.

Now, what's the actual contents? Are you saying, Terence, that non-military GPS is not accurate enough for our needs? Or are you merely plugging Leica's system? I am really interested, so don't see this as belligerent trolling, I just want to know what to argue.

I simply found it interesting to know how inaccurate non-military systems are. I'm not planning on purchasing a GPS myself as I don't think I'd be a high enough user to warrant even a basic system (having borrowed my father-in-law's and found it cumbersome to use).

David
8  General / GPS/ Geotagging / Re: GPS Error/accuracy.. I got long winded here, your warned on: February 20, 2007, 01:52:47 PM
GPS accuracy, think Sedan deVille, M1 Humvee..

Terence

Ah! Just like me taking a good photo. Extremely elusive.

This was a good introduction to the foibles of GPS Terence. And not too long winded.

David
9  General / GPS/ Geotagging / Re: Anybody geotagging? on: February 20, 2007, 02:51:59 AM
David

I've never heard of exiv2 but it seems to have a command line utility, that accepts arguments, so in theory should be scriptable from iView. I don't see geographical exif items in the spec, though. You could store the co-ordinates in a custom field(s) but you would have to be very consistent in data entry. possibly a job for the vocab editor?

John

I've successfully scripted it from iMatch before. One of the limitations of all these tools is that they tend to be confined to JPEG only, even if the EXIF spec appears elsewhere. Vocab editor may be a good idea. I hadn't thought of that. Still fairly manual though. It would work on a per-shoot basis. I'm recataloging 8000 images.

David
10  General / GPS/ Geotagging / Re: Anybody geotagging? on: February 20, 2007, 02:50:14 AM
I'm interested to know if anyone has a workable geotagging system running. I can tag my photos on flickr but that's once they are out of Iview. It would be nicer to store some metadata somewhere and read it into the file so that flickr/others can read it directly.

David,

Here are two ways to GEO-tag images, depending on whether you have a GPS-reciever or not:

1. Picasa and Google Earth.
Select the images in Picasa and choose Tools > Geotag > Geotag with Google Earth...
You will manually have to navigate to the location of each image. There is a "Tag all" option for images taken on the same location though.

2. Using a GPS-tracklog.
Make sure that the camera and GPS-clocks are synchronized. Log the position while taking the photos. Use some software to calculate the position based on the GPS track log and stamp the position in Exif. There are not too much software that is able do this, but RoboGeo and WWMX location stamper can. I personally prefer WWMX which is free.

As far as I know there is no way to use these techniques on RAW images. I hope this will be possible in the near future. I am, therefore, saving my track logs for later.

1. Picasa an Google Earth is not a scalable solution and by scalable I mean able to handle large volumes of images. If I come back to a location I'd rather have iView remember the coordinates for that location rather than have to go into Picasa and Google Earth again. At least for me I can't get it to work well.

2. A GPS is another piece of equipment and then there is still the need to populate the photo. Also, a GPS may put in a slightly different location per photo. Peter's concept of tag error above comes into play here.

Thanks for the ideas and for others they may work fine. For me at the moment they wont.

David
11  General / GPS/ Geotagging / Re: Anybody geotagging? on: February 20, 2007, 02:47:44 AM
Peter,

I can see lots of needs - mostly ones that interface with DAM software.I'd like to be able to link the IPTC location fields up to a Google (or other) database to suck GPS data back into the database record
I've found the 4 level IPTC fields (country, state/province/city and location) to be too limiting in number. To tie all photos to a single location can be too confining if the location is relatively large (eg. shots taken around one of the world's disneylands).

I think there needs to be some kind of margin of error tag that goes with this.  If I have photos shot in Boston, I don't want to have to choose an arbitrary place as the location, I'd like to choose a vector-drawn area. (not sure if the current spec could suport that.)
Hadn't thought of the approach of a tag error.

12  General / GPS/ Geotagging / Re: Anybody geotagging? on: February 19, 2007, 06:56:02 PM
Thanks for the links Terence.

One of the problems I see with products such as RoboGeo or even linking directly through flickr is they require me to remember the locations or re-set them each time. It would be nice if coordinates could be stored against a location in iView and use a script to put it into the jpegs (using exiv2) for example. Then I can export my jpegs for flickr, run a script over them to prepare the metadata and upload once. Flickr then does the rest of the work.

And I would do this without touching the original files.

David
13  General / GPS/ Geotagging / Anybody geotagging? on: February 18, 2007, 04:11:58 AM
Hi all,

I'm interested to know if anyone has a workable geotagging system running. I can tag my photos on flickr but that's once they are out of Iview. It would be nicer to store some metadata somewhere and read it into the file so that flickr/others can read it directly.

David
14  Software Discussions / iView MediaPro / Re: EXIF Data from DNG to JPEG on: February 10, 2007, 09:29:34 PM
I just discovered this problem myself. One issue I have with Make/Model not coming across is that flickr (www.flickr.com) uses it to enable access to EXIF information online. If there is no model, nothing shows.

David
15  DAM Stuff / Loss and Recovery / Re: Recovering from a RAID crash - which drive failed? on: February 07, 2007, 11:22:21 PM
Stephen,

Since RAID is designed to protect against hard drive failure, it's a wonder it is so difficult to solve.

David
Pages: [1] 2 3 ... 5
Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!