Quick advice on storage and backup

Editor’s Note: Since I’ve turned Facebook comments off, I’m experimenting with turning them on directly in the blog. Feel free to ask any questions you may have about this subject at the bottom of the page. No log-in required.

I’ve given this advice several times recently, so I figured I’d turn it into a quick blog post.  If you know someone who is struggling with decisions on how to store or backup media files, please share.

For most people, reliable storage and backup has gotten really cheap and easy to implement, even for people with lots of photos and video.

The trick is to make use of the newer high-capacity drives and affordable cloud backup services. So here’s what I end up telling most people who ask. Note that this advice is designed for people with a data set smaller than about 10TB who don’t need multi-user access.  And, this advice assumes that you have a media collection you want to store and protect, and that it exceeds the capacity of your computer’s internal drive.

General Advice
This holds true for most photographers and (since everyone is now a photographer) also for “regular people”.

  • Use modern big drives – 10-14 TB drives are newer, better designed, more reliable and much faster than older ones. They are also very affordable. If you are using multiple smaller drives, it’s time to replace with a single larger one.  Even if your archive is only a few terabytes, go for one of the large drives.
    Currently, I recommend G Technology drives. 10TB is the current sweet spot for prices ($300). You can bump up to 14TB for a total of $450. (I’ve linked to the USB3.1 version of the drive. You can also get these drives in Thunderbolt, but it won’t be any faster with conventional spinning disks.)
  • Use as few drives as possible – If you can easily get all your stuff on one single drive, do it. It’s much easier to backup and restore than an array of older drives. You don’t need to remember what is where.
  • Get a drive to make a “twin” onsite backup – Again, simpler is better, and big drives are your friend. This is what protects you against drive failure. Much easier to make, keep current, and restore from a local backup than a remote one.
  • You also need an offsite backup to protect against fire or theft – There are two main method to do this. You can get a second backup drive and keep it offsite. You can also use a cloud service to make a 3-2-1 compliant backup. I do both, but let’s handle them independently.
  • Backblaze cloud backup – Backblaze is a great cloud service that I depend on for my own work. The personal version of the service offers unlimited file backup for $60 a year. Uploaded files are encrypted for privacy. You can add external drives to the backup, and it happens automatically in the background. If you are a Photoshelter Pro customer, you can also use their service as your cloud backup. Unlimited storage is included with the Pro accounts.
  • Additional drive for offsite backup – An offsite backup drive provides excellent protection, and quick restoration in the event of a problem. Store it offsite – in general, someplace easy is better than someplace really secure. If you are worried about the data falling into the wrong hands, you can format the drive with encryption. Keeping an offsite drive updated can be difficult, and there are almost always gaps between what’s currently on your primary storage and what’s on the backup. This is why I currently favor Backblaze, especially for working files.
  • Avoid spanned drive devices (like RAID, multi-drive NAS, Drobo, etc.) unless:
    • You really need them for a particular reason. Most people don’t really need single storage volumes larger than 14TB.
    • You understand how to maintain them or have a good tech service to use. These are little computers running Linux, and in general require maintenance, monitoring and updates. I know many people who have experienced total failure of spanned disk devices.
  • If you outgrow the single-drive units, get an additional set – This is getting outside the scope of this post, but worth mentioning. If you can’t fit everything on one drive (say you have 20TB data), then I suggest getting an additional set of drives, rather than going to a RAID device. In most cases, it’s easier, cheaper and safer.

This advice is drawn from content published in The DAM Book 3.0. If you’ve got a larger or more complicated storage and backup job in front of you, you’ll find a lot more discussion over there.

Petapixel Article on Megapixels and Camera Scanning

I wrote an article that has just been published in Petapixel. In the article, I offer some insight on megapixels and camera scanning, particularly for 35mm film originals.

We are hitting a point of diminishing returns on pixel density where going to very high resolution sensors like the Sony A7 RIV (61 megapixels) may actually get you inferior quality to lower resolution (30-45 MP).

If you are putting a camera scanning system together, you should carefully evaluate critical sharpness and make sure that you can get consistent sharpness across the entire frame.

What I did not say specifically in the article is that use of a rail-based system becomes even more important as you go to higher megapixels since it allows easier alignment of the camera to the film I stayed away from any specific hardware mentions in the article.

If you are interested in rails, which offer precise alignment of the camera and film, I’ve just produced a new batch of the PS-4 version, which seems to sell out as fast as I can make them.

Looking for iView Replacements?

Last week, I had the pleasure of speaking to the National Geographic photographers group, The Photo Society. The topic of my talk was the death of iView (aka Expression Media, and Phase One Media Pro). This application has been a real workhorse for photographers, and especially for this group.

Phase One discontinued Media Pro about 18 months ago, but the real moment of truth is the release of Mac OS 10.15. It fully drops support for 32 bit applications so none of the iView/EM/MP versions can run at all under it. One day, you will upgrade your OS, and all the 32 bit apps will be dead. (Windows does not have the same problem right away, but it’s surely coming).

Life Support
If you want to keep Media Pro running, you’ll need to delay OS upgrade as long as possible. This means keeping old hardware running since Apple does not allow you to install an old OS on a new computer.  This is not a huge problem since older computers are inexpensive, plentiful, and powerful enough.  If you need to upgrade your primary machine (and you will, eventually), you’ll need to have another machine to use the software with.

If you expect to make sure you can run the software, you’ll want to make sure you have a bootable clone backup of the OS drive, in the event your computer stops working for some reason.

Migrating iView metadata
Most of the metadata you create in Media Pro is standard IPTC data which is easily readable by many other programs. In order to make that info readable, you typically only need to export the metadata back to the files.  (Select all files, choose Action Menu>Export Annotations>check all options and go.)

If you have Catalog Sets information, you will need to do some work to make it available to other applications. Note that one thing that will NOT come along is the sequencing of the Catalog Sets.

Migration with Bridge
You can convert Catalog Sets to Lightroom-compatible Hierarchical Keywords using Adobe Bridge along with a script. Install the script, and you can transfer the data with a simple menu command,

Here is a link to download the script.

Programs that read the Catalog Sets
Photo Supreme is an application that can read the catalog sets information directly from the files and convert it to Photo Supreme Portfolios. This requires the use of a script supplied directly by Photo Supreme.

Daminion is a Windows only program that looks almost exactly like Media Pro. It can read the Catalog Sets data natively, so no migration is needed.

A Big List
If you’d like to see a huge list of applications, check out list of applications shown at Impulse Adventure.

Vote for our 2020 SXSW proposal!

Anna Dickson and I have, once again, made a proposal for SXSW. This time it’s called the Machine Learning Bake-off. In this presentation we’ll do some real-world comparisons of Machine Learning services for analyzing photos. We will test services and present findings on the good, the bad and the ugly.

Here’s the proposal, including the link to vote.
And here’s the proposal info.

Machine Learning Bake-off

Is ML the solution for making sense of vast collections of images? In demo form, it looks amazing. But does it really provide actionable information for you, or does it junk up your tags with a lot of low value (and wrong) information? Time for a taste test! In this presentation, you’ll see the results of real world testing from leading services – Google, Amazon, Microsoft and Clarifai. Our test set includes a wide variety of images representing multiple industries and tagging challenges. We’ll show you where each ML shines, and where each misfires, and how the serviuces have evolved. Armed with our evidence and conclusions, you can decide if it’s delicious, or not yet ready to eat. As a bonus, we’ll show you how to easily run your own test on tens of thousands of images for under $200.

Takeaways
• Get a solid idea of the info that Machine Learning can currently add to image collections. Understand what it’s good and bad for.
• Get a handle on the differences between ML services and how each can help you. Get a better idea of how to evaluate your options.
• There is no substitute for some real-world testing on your own material – at scale – if you want to determine the value of a service. Here’s how.

New Rail Systems for Camera Scanning Available

I’ve just finished a new batch of rail systems for camera scanning and I have a few additional ones available. These are among the very best tools for camera scanning slides and negatives with a DSLR. You can use just about any camera and macro lens. Faster, better, cheaper than a conventional scanner.
I have 5 rails and four light kits currently available. Here is the order page.

More info is in my book, Digitizing Your Photos.

The rails are $350/$375 depending on rail length. light kits are $200.

After making 50 or more of these, I’ve finally gotten it down to an assembly-line workflow, best done with at least five units at a time.


Drill press time! I’ve created a jig here so that I can properly place the access hole in the cover plate repeatably. 

My SXSW proposal – Small Photos, Big Data: A Connectivity Manifesto

It’s Panel Picker time again! Please take a moment and vote for my  session proposal for SXSW 2019. Once again, I’ve teamed up with Anna Dickson to explore the use of visual media and the data that is connected to it.

Small Photos, Big Data: A Connectivity Manifesto

On the mobile web, images serve a greater purpose than simple visual description. Rich media images are increasingly used to connect people, events, institutions, ideas, advocacy and commerce. As we move into a new era of visual communication, this trend is accelerating. While the use of connected images blossomed on social media services, it reaches far beyond walled gardens into API-based interchange on the open web. Machine learning and linked data are creating new methods to make connections, and the Data Transfer Project is opening up access to the underlying graph for portability and innovation. In this presentation, we will explore the current state of visual media connectivity, what it can do for you, how to enhance your own image connectivity, and how to avoid costly mistakes.

Adobe Max – A great conference for visual creators

I’ll be headed to Los Angeles in mid-October for Adobe Max, my third time there.  Over the last several years, the conference has grown like crazy, including the addition of a lot of photo-related programming. In each of the years I’ve attended the conference, I walked away with a much better understanding of the emerging media landscape.
Here is a highlight video from 2017. It gives you a peek at the type of content at Adobe Max.

There is a fascinating mix of programming at Max. There are breakout presentations, workshops, pre-conference multi-day workshops, and plenary sessions. The big plenary sessions are the ones that were most interesting to me, including inspirational talks from Annie Griffiths, and Jonathan Adler.If you are interested in where Artificial Intelligence and Machine Learning is going, Max provides a showcase for Adobe’s massive undertaking, Sensei. Sensei is purpose-built for the creative, marketing and communication industries, and it is poised to have far-ranging effects on the way visual media is created and deployed.

The Sneaks are a look at experimental development efforts, including products that are still on the drawing board. Always fun and popular, and hosted by a people like Nick Offerman or  Kumail Nanjiani.As you can see, there is a lot of the content available on free video channels. So why go? Like all good conferences, the value is frequently found in the personal connections you make rather than strictly in the programming. And in the best conferences, you open your mind with new programming at the same time you are making connections with new people.

There’s also a pretty good party at the end of the thing, usually including good live music, a ton of great food and drink, along with other fun and games.

Max is not cheap – list price is $1595, and the discounted price of $1295 is only available until July 31. I have still not cracked the code to get a presenter slot at Max, but this year I’m going as a TA. I’ll help out someone’s classes, learn, and meet new people. If you are looking for a hint of what the future of media will bring, I suggest you give Max a try.

Blockchain for creators?

There have been a flurry of companies producing white papers on the way that blockchain applications can help to solve the challenges of independent creators. These range from new distribution networks to services that claim to solve the attribution/ownership issues.

In response to a tweet by my friend Leora Kornfeld, I launched a little fusillade at her, explaining why I think that all of the proposals I have seen are worse than worthless – they actually provide negative benefit to independent creators. Here is that tweet storm in paragraph form.

________

I come at this as a person who makes the bulk of his income by selling intellectual property – photos, books and videos – print, DVD and download. None of the challenges I face are going to be helped by blockchain.

The main challenges are, in this order:
1. Making content worth buying
2. Developing, maintaining and expanding an audience
3. Find technical solutions to make the production and sale of materials possible and profitable

There is a database problem in this task set, primarily the audience relationship management. But this task is not going to be helped by using a public, immutable, distributed database. (GDRP anyone?)

We have a valuable long-term relationship with my readers. We need to keep records of what they bought, how to contact them, what their communication preferences are, whether we’ve done seminars with them, what problems they encounter, whether they are nice (almost all are).

This calls for Filemaker (or some other dedicated contact/customer management tool), not a blockchain.

Okay, so maybe blockchain is not going to help you sell, but can’t it help protect your stuff? Put it in the blockchain and it’s protected…somehow.

In the US at least, putting something in a blockchain gets you exactly zero protection. If you want to protect your stuff, you need to register it with the Copyright office. (A blockchain app might do the registration, but the blockchain part does not get you any benefit.)

And it’s basically impossible to use a blockchain “fingerprint” for any enforcement (unless it’s accompanied by a copyright registration – the real lever).

The “fingerprint” of any digital image or video will change each time it’s uploaded to a new service and recompressed. So the “immutable record” turns into “this might be the same photo/video”, but only if you have other matching software running separately from the blockchain.

So, in the end, what is left for the blockchain? Payment? How will that be better than Visa, Paypal or Venmo? These can all be converted to local currency nearly anywhere in the world. We sell directly to customers in dozens of countries. Shopify handles most of these transactions seamlessly for a 2% transaction fee.

But wait, it gets worse. The investment in blockchain vaporware takes money and focus away from real solutions.

This could include small claims copyright remedies, new distribution channels that have a chance of functioning, international agreements, new methods to monetize owned content, etc.

As a metadata and asset management nerd, I think in data structures. I’m always looking for new paradigms, new uses. But I just can’t find a good structural use for a blockchain.

Since you’re reading this, you might want to talk a look at my books. They deal with the intersection of digital technology and visual media. https:theDAMbook.com

But there is no blockchain in them…

DIGITAL ASSET MANAGEMENT FOR PHOTOGRAPHERS