All posts by admin

Rebuild Lightroom Previews to save space, add speed and confirm integrity

I have a new article up on Petapixel that outlines the benefits of a full rebuild of Lightroom previews.

  • Your catalog may be significantly smaller. My two main catalogs decreased in size from 600GB to about 200GB.
  • You may see a speed increase in scrolling – I saw a marked increase.
  • You will be informed if Lightroom fails to rebuild any previews, which may be an indication of a corrupted file (It could also be a file that Lightroom can’t open). The process correctly identified a handful of known-corrupted files that I use for testing.
  • Read the entire article here.

Note that I have outlined a specific work order for this, which I suggest following. I’d love to hear about anyone’s results, either confirming or conflicting with the results I got.

Shelter-in-place sale extended to May 31

Well, we hoped that things would go more quickly and we’d be back out in the world by now. Some of us are still (mostly) sheltering in place. We’re doing a final extension of the 50% off shelter-in-place sale of all DAM Useful publications until May 31st. This applies to both electronic and paper publications when purchased from DAMuseful.com

While we don’t expect that things will be back to normal by June 1, we are hoping that we will all be able spend more time outside, while observing proper social distancing.

Note that we have not updated the copy on every link on the site, but if you buy from DAMuseful.com, you will get the discount. (Sorry, can’t afford to do this on Amazon,)

Stay safe everyone. See you on the other side.

Webinar this Friday May 8

Here is a link to the notes for the presentation.

Use your shelter-in-place time to work on your photo collection – shore up your storage and backups, organize your photos, and create something! I’ll be doing a webinar this Friday, May 8 at 2pm EST on Facebook Live. The program is being hosted by Ripple Effects Images. You can find it on their facebook page.

There should be something in here for all photographers and people managing photo collections. I’ll provide some clear, actionable advice for improving your digital photography management.

Ripple Effects Images is the brainchild of National Geographic photographer Annie Griffiths. The organization sends some of the best photographers and videographers out to report on the challenges facing women and children in the developing world. The images and videos are them made available for free to vetted nonprofit organizations. It’s a fabulous organization, doing great work.

I’ve been involved with Ripple for a while, and now as a member of their Board of Advisors. I’m thrilled to play a part in their important work. Here’s a video of Annie speaking to Adobe Max a few years ago, which should provide a little inspiration for our troubled times.

Also note that we have extended our shelter-in-place 50% off sale for all my books for another two weeks.

Chronosync Backup Tip: Chronoagent fixes reliability problems

I’ve been using ChronoSync for many years, and depend on it to manage backup from several machines to a central file server. It has worked quite reliably over the years, and I have only had to do some occasional maintenance. That changed over the weekend.

I’ve retired the old 2007 iMac that was my file server in favor of a 2015 model. This allows me more flexibility in drive enclosures, since it supports USB 3 and Thunderbolt.

The 2007 was still running 10.6, which is from 2009, but everything worked just fine. However the new computer (running 10.13) kept failing when I tried to synchronize. I tried everything I could think of – clean install of the OS, run a test sync on newly formatted backup drive, change permissions. And still the problems persisted.

The most common problem was ChronoSync failing during the sync, with a message that the target was not available (even when the target was clearly mounted), or the file just failing to sync.

I was seeing the “cannot locate target” warning in ChronoSync, even when the destination was mounted. After installing ChronoAgent, it shows up as an option in “Connect to:” and behaves just like the server did.

And then I remembered ChronoAgent. This is an option application in the ChronoSync suite which manages the connection from source to destination. After a quick installation, the problems have all seemed to disappear. I’m still testing all the machines that are pointed to the server, but it seems to have solved the problem.

Installing ChronoAgent was really easy, and makes the backup process much more trouble-free, particularly over a network.

I know several people who have complained about reliability issues with ChronoSync. When asked, my new reply will be to suggest setting up ChronoAgent.

Digital Media Objects – A definition

I use the term “object” a lot in The DAM Book 3.0 when I’m referring to digital media. It’s a pretty common word to hear in the software development, museum or enterprise IT community. While it’s frequently used, it’s not well defined. To add to the confusion, a digital object could refer to many types of things, like a password or a line of code. To help clarify the use of the term digital object, I propose the following definition for media objects: A digital media object is a file, set of files, or bitstream that can be rendered into tangible media.

Let’s take that apart.

  • A digital object is not necessarily a single file. Sometimes multiple files are needed to render the finished media. This could include an audio file and a video file. Or it could be a package of files. (An Apple Keynote document is a package of individual media files and a controller file.) The digital source could also be a bitstream, like a streaming video playing as an embedded object in a web page.
  • The essential quality of a digital media object is that it can be turned into actual media. This could be a photo, video, audio presentation or other thing you can see, hear or experience in a tangible way. This provides a useful separation from the more generalized concept of a digital object, which could be a chunk of software code, a password, or some other thing that does not ever need to be rendered to be useful.
  • A digital object could have a very complex internal structure (e.g., a DVD is a digital object that may be made up of hundreds of component files). Or it could be very simple, like a PNG image. It could be something that should be rendered in only one way, like a profile-tagged JPEG, or something that could be rendered in all kinds of ways, like a camera raw file.

Is a creative project a digital object?

An InDesign or Final Cut project fits the definition of a digital object because it can be rendered out in tangible form. But I suggest sticking with the designation of creative project to describe this sub-type of digital object.

These projects are really designed to create a freestanding output, which will serve as a playable distribution copy. From a workflow standpoint, it’s useful to retain this distinction between the project used for creation and the digital object output version.

Components

Digital objects will usually have at least three parts: the media payload, some metadata and one or more digital object identifiers (DOI).

  • The media payload is the actual image, video audio, etc. This is the thing or things that can be made tangible.
  • The metadata describes, at minimum, what the media type is and how to render it.  The metadata may also describe the subject matter, provenance or other property of the media.
  • The identifier might be a traditional file name (which is one reason it’s important to have unique file names). Sometimes the identifiers are computer-generated hexadecimal IDs that are not really human readable. Here’s file name for a Lightroom preview JPEG: FFE3EBE9-0A27-412D-81F1-138284A21AE1-a689c6aea923108279afcc6fab8775c0.lrprev
    Of course Digital Object Identifiers are often not filenames, but instead a metadata tag. Lightroom creates and preserves one called OriginalDocumentID, which is also a hexadecimal identifier ED2A2082136A7F551B2F07CB9F2C4712

Files

A file is similar to an object in that it has a name, a payload, and it conforms to a file format (such as TIFF). A file, however, must be a block of data, and not a bitstream or collection of files. So a file could be a digital media object, but a digital media object may be something other than a file.

Bitstream

As its name implies, a bitstream is a stream of digital data, rather than discrete file stored on disk. A bitstream may start life as a file, like a YouTube video, which has a beginning and an end. A bitstream could also be an ongoing stream of data, like a feed from a security camera – it might never be saved as a conventional discrete file.

This post is drawn from The DAM Book 3.0

Offer Extended! 50% off Shelter-in-place sale

PLEASE NOTE: To guaranteed you get the discount, please go to DAMUSEFUL.COM instead of using the BUY buttons on this site!

Well, it looks like it’s probably going to be a longer shelter-in-place period than any of us want. So we’ll have lots of opportunity to catch up on streaming series, Zoom meetings, cleaning out closets, and calling old friends. We can’t help you there.

You may also want to catch up on some reading, or organizing your photo collection. We can help you with this. We’re dropping the price of all our books to 50% of the cover price, something we’ve never done before. This applies to both printed books as well as digital downloads. (Discount codes don’t apply during this period.)

Our sale will run for the next two weeks, until April 10 until the end of April.

We also have a limited number of rail systems for copying 35mm slides and negatives. Since some of these parts come from China, so we’re not sure when we will be able to restock.

Thanks for reading and stay safe.

Quick advice on storage and backup

Editor’s Note: Since I’ve turned Facebook comments off, I’m experimenting with turning them on directly in the blog. Feel free to ask any questions you may have about this subject at the bottom of the page. No log-in required.

I’ve given this advice several times recently, so I figured I’d turn it into a quick blog post.  If you know someone who is struggling with decisions on how to store or backup media files, please share.

For most people, reliable storage and backup has gotten really cheap and easy to implement, even for people with lots of photos and video.

The trick is to make use of the newer high-capacity drives and affordable cloud backup services. So here’s what I end up telling most people who ask. Note that this advice is designed for people with a data set smaller than about 10TB who don’t need multi-user access.  And, this advice assumes that you have a media collection you want to store and protect, and that it exceeds the capacity of your computer’s internal drive.

General Advice
This holds true for most photographers and (since everyone is now a photographer) also for “regular people”.

  • Use modern big drives – 10-14 TB drives are newer, better designed, more reliable and much faster than older ones. They are also very affordable. If you are using multiple smaller drives, it’s time to replace with a single larger one.  Even if your archive is only a few terabytes, go for one of the large drives.
    Currently, I recommend G Technology drives. 10TB is the current sweet spot for prices ($300). You can bump up to 14TB for a total of $450. (I’ve linked to the USB3.1 version of the drive. You can also get these drives in Thunderbolt, but it won’t be any faster with conventional spinning disks.)
  • Use as few drives as possible – If you can easily get all your stuff on one single drive, do it. It’s much easier to backup and restore than an array of older drives. You don’t need to remember what is where.
  • Get a drive to make a “twin” onsite backup – Again, simpler is better, and big drives are your friend. This is what protects you against drive failure. Much easier to make, keep current, and restore from a local backup than a remote one.
  • You also need an offsite backup to protect against fire or theft – There are two main method to do this. You can get a second backup drive and keep it offsite. You can also use a cloud service to make a 3-2-1 compliant backup. I do both, but let’s handle them independently.
  • Backblaze cloud backup – Backblaze is a great cloud service that I depend on for my own work. The personal version of the service offers unlimited file backup for $60 a year. Uploaded files are encrypted for privacy. You can add external drives to the backup, and it happens automatically in the background. If you are a Photoshelter Pro customer, you can also use their service as your cloud backup. Unlimited storage is included with the Pro accounts.
  • Additional drive for offsite backup – An offsite backup drive provides excellent protection, and quick restoration in the event of a problem. Store it offsite – in general, someplace easy is better than someplace really secure. If you are worried about the data falling into the wrong hands, you can format the drive with encryption. Keeping an offsite drive updated can be difficult, and there are almost always gaps between what’s currently on your primary storage and what’s on the backup. This is why I currently favor Backblaze, especially for working files.
  • Avoid spanned drive devices (like RAID, multi-drive NAS, Drobo, etc.) unless:
    • You really need them for a particular reason. Most people don’t really need single storage volumes larger than 14TB.
    • You understand how to maintain them or have a good tech service to use. These are little computers running Linux, and in general require maintenance, monitoring and updates. I know many people who have experienced total failure of spanned disk devices.
  • If you outgrow the single-drive units, get an additional set – This is getting outside the scope of this post, but worth mentioning. If you can’t fit everything on one drive (say you have 20TB data), then I suggest getting an additional set of drives, rather than going to a RAID device. In most cases, it’s easier, cheaper and safer.

This advice is drawn from content published in The DAM Book 3.0. If you’ve got a larger or more complicated storage and backup job in front of you, you’ll find a lot more discussion over there.

Petapixel Article on Megapixels and Camera Scanning

I wrote an article that has just been published in Petapixel. In the article, I offer some insight on megapixels and camera scanning, particularly for 35mm film originals.

We are hitting a point of diminishing returns on pixel density where going to very high resolution sensors like the Sony A7 RIV (61 megapixels) may actually get you inferior quality to lower resolution (30-45 MP).

If you are putting a camera scanning system together, you should carefully evaluate critical sharpness and make sure that you can get consistent sharpness across the entire frame.

What I did not say specifically in the article is that use of a rail-based system becomes even more important as you go to higher megapixels since it allows easier alignment of the camera to the film I stayed away from any specific hardware mentions in the article.

If you are interested in rails, which offer precise alignment of the camera and film, I’ve just produced a new batch of the PS-4 version, which seems to sell out as fast as I can make them.

Looking for iView Replacements?

Last week, I had the pleasure of speaking to the National Geographic photographers group, The Photo Society. The topic of my talk was the death of iView (aka Expression Media, and Phase One Media Pro). This application has been a real workhorse for photographers, and especially for this group.

Phase One discontinued Media Pro about 18 months ago, but the real moment of truth is the release of Mac OS 10.15. It fully drops support for 32 bit applications so none of the iView/EM/MP versions can run at all under it. One day, you will upgrade your OS, and all the 32 bit apps will be dead. (Windows does not have the same problem right away, but it’s surely coming).

Life Support
If you want to keep Media Pro running, you’ll need to delay OS upgrade as long as possible. This means keeping old hardware running since Apple does not allow you to install an old OS on a new computer.  This is not a huge problem since older computers are inexpensive, plentiful, and powerful enough.  If you need to upgrade your primary machine (and you will, eventually), you’ll need to have another machine to use the software with.

If you expect to make sure you can run the software, you’ll want to make sure you have a bootable clone backup of the OS drive, in the event your computer stops working for some reason.

Migrating iView metadata
Most of the metadata you create in Media Pro is standard IPTC data which is easily readable by many other programs. In order to make that info readable, you typically only need to export the metadata back to the files.  (Select all files, choose Action Menu>Export Annotations>check all options and go.)

If you have Catalog Sets information, you will need to do some work to make it available to other applications. Note that one thing that will NOT come along is the sequencing of the Catalog Sets.

Migration with Bridge
You can convert Catalog Sets to Lightroom-compatible Hierarchical Keywords using Adobe Bridge along with a script. Install the script, and you can transfer the data with a simple menu command,

Here is a link to download the script.

Programs that read the Catalog Sets
Photo Supreme is an application that can read the catalog sets information directly from the files and convert it to Photo Supreme Portfolios. This requires the use of a script supplied directly by Photo Supreme.

Daminion is a Windows only program that looks almost exactly like Media Pro. It can read the Catalog Sets data natively, so no migration is needed.

A Big List
If you’d like to see a huge list of applications, check out list of applications shown at Impulse Adventure.

Vote for our 2020 SXSW proposal!

Anna Dickson and I have, once again, made a proposal for SXSW. This time it’s called the Machine Learning Bake-off. In this presentation we’ll do some real-world comparisons of Machine Learning services for analyzing photos. We will test services and present findings on the good, the bad and the ugly.

Here’s the proposal, including the link to vote.
And here’s the proposal info.

Machine Learning Bake-off

Is ML the solution for making sense of vast collections of images? In demo form, it looks amazing. But does it really provide actionable information for you, or does it junk up your tags with a lot of low value (and wrong) information? Time for a taste test! In this presentation, you’ll see the results of real world testing from leading services – Google, Amazon, Microsoft and Clarifai. Our test set includes a wide variety of images representing multiple industries and tagging challenges. We’ll show you where each ML shines, and where each misfires, and how the serviuces have evolved. Armed with our evidence and conclusions, you can decide if it’s delicious, or not yet ready to eat. As a bonus, we’ll show you how to easily run your own test on tens of thousands of images for under $200.

Takeaways
• Get a solid idea of the info that Machine Learning can currently add to image collections. Understand what it’s good and bad for.
• Get a handle on the differences between ML services and how each can help you. Get a better idea of how to evaluate your options.
• There is no substitute for some real-world testing on your own material – at scale – if you want to determine the value of a service. Here’s how.