Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3

Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/content/60/9972860/html/smf/Sources/Load.php(225) : runtime-created function on line 3
Latest posts of: NYBG
The DAM Forum
Welcome, Guest. Please login or register.
October 28, 2020, 10:39:26 AM

Login with username, password and session length
Search:     Advanced search
28033 Posts in 5147 Topics by 2903 Members
Latest Member: kbroch
* Home Help Search Login Register
  Show Posts
Pages: [1] 2
1  DAM Stuff / DNG / Wow...That took months on: December 02, 2011, 10:24:34 AM
Holy cow - I just read my last post from back in August... August!

I just finished (it's December!) converting about 125,000 TIFFs to DNG. The Tiffs started around 210 MB, the DNGs are about 90 MB. I starte with 15.8 TB of TIFFs and ended up with 6.8 TB of DNGs - reclaiming 9 TB of space on New York Botanical Garden's servers.

I installed Lightroom on seven networked computers and converted batches of 1200 images at a time on each machine. (While I would have preferred to have used the DNG converter, for some reason it did not work well over the network.)

 
2  DAM Stuff / DNG / Re: Why DNG on: August 11, 2011, 10:20:56 AM
I have a compelling reason to convert my TIFFs to DNGs: space (http://digitalphotorepro.blogspot.com/2011/08/global-plants-initiative-jstor.html).

I am facing a storage shortage right now. I have 15 terabytes of TIFFs sitting in an archive that are rarely, if ever, used. On top of that, they are duplicated offsite. So, by converting to DNG, I estimate that I can recover about 7.5 terabytes!

This is essential, because with additional imaging projects coming online, I figure I am going to need at least 5 terabytes of space through the end of 2011.

I have been reading to see if there are any downsides to converting TIFFs to DNG. So far, it seems that the only downside is the perception that TIFF is somehow more 'archival' a format. But, as a colleague of mine from NYU's library said, there is no archival format - preservation is an institutional commitment that requires ongoing human diligence (http://digitalphotorepro.blogspot.com/2011/04/digitization-special-interests-group.html).

3  DAM Stuff / Migration Issues / Re: Migrating 10 Terabytes on: August 09, 2011, 06:51:32 AM
Unfortunately the DNG converter encountered numerous errors due to lack of memory, which I guess is to be expected with that many files. So, I am back to exporting as DNG from Lightroom. A temporary set back, but I have a lot of time. Some of the specimens I am dealing with are hundreds of years old. No one is in that much of a hurry.
4  DAM Stuff / DNG / 15 days of converting on: August 04, 2011, 09:58:37 AM
Entering the final stages of data/format migration and archive restructuring (http://thedambook.com/smf/index.php?topic=5135.0).

I now have two server partitions with 183,531 RAW files (of various types). I am using the DNG converter to get them all into one format. At a current rate of about 500 conversions an hour, I estimate that this will take about 15 days!
5  DAM Stuff / Migration Issues / Re: Migrating 10 Terabytes on: August 03, 2011, 12:37:04 PM
My "unable to export" error may have been a memory issue: http://digitalphotorepro.blogspot.com/2011/07/thanks-to-lightroom-forums.html

I have finally rooted out all the TIFFs and all the duplicates (and saved TIFFs for which there are no duplicate RAW files). Now I am starting to convert the legacy RAW files to DNG. I was using Lightroom for this. I started with a test batch of 2,500. This also gave me some time to re-read the DAM Book.

In it, I was reminded of the free DNG Converter from Adobe. I am giving that a try right now. So far it is searching the directory on the archive and is up to finding 8,300 RAW files. There are over 100,000 RAW files in the directory I am searching. This also looks like it will take a while...
6  DAM Stuff / Migration Issues / Error: XMP parse threw an unknown exception on: July 29, 2011, 08:31:30 AM
I successfully converted the Kodak TIFF RAW files into DNG!

Now, I want to convert the remaining RAW files (NEF, CRW,DCS) to DNG. Unfortunately, I keep encountering the following error: "Unable to Export. An internal error occurred: XMP parse threw an unknown exception". (http://digitalphotorepro.blogspot.com/2011/07/adobe-lightroom-27-error-unable-to.html)

I have read that ditching old XMP files can help over come this. That is my next test. The only problem is, I want the data from the XMP files.
7  DAM Stuff / Migration Issues / RAW TIFF Kodak DCS 660 on: July 25, 2011, 08:45:03 AM
Ahh... it's starting to become clearer.

I see that all the weird "TIFF" files were shot with the Kodak DCS 660 back in 2001. A little searching and I found an old post on luminous-landscape.com and dpreview.com about the file format that the DCS 660 used. It was a proprietary "RAW TIFF".

So, Lightroom and Windows both see the .tif extension and think the file is a TIFF, but in all practicality it is a Raw file.

All the more reason I am going through this archive. I am going to hurry up and convert these files to DNG as soon as I have this directory on tape backup.

8  General / General Discussion / Re: Deleted on: July 22, 2011, 12:08:01 PM
There was quite a lot riding on the delete button Wink
9  DAM Stuff / Migration Issues / Re: Migrating 10 Terabytes on: July 22, 2011, 10:51:23 AM
Can not. Can not make heads nor tails out of this collection.  Wink
10  DAM Stuff / Migration Issues / Re: Migrating 10 Terabytes on: July 22, 2011, 10:50:20 AM
I can make heads nor tails out of this collection.

I cataloged it and Lightroom showed 66,726 TIFF files and 94,932 Raw files. I wanted to compare lists, so I ran a Windows search to generate a list of TIFFs. Windows found 106,616 TIFFs.

So, I filtered the results by TIFF and RAW in Lightroom and made collections of the two groups. TIFFs were straightforward. If they said TIFF, they were TIFFs. But then I was looking in the RAW collection. I saw various RAW extensions, but I also saw the .TIF extension in with the RAW files. How could this be?

I exported one of these RAW .TIFs as original, and I got a 156x104 pixel image (that is somehow 5MB) and an .xmp file. So I guess it is a RAW file, because if I export it as a TIFF I get a 3040x2008 pixel, 17 MB TIFF.

On further inspection, I see that it was shot with a Kodak DCS660C back in 2001. Now it's starting to make sense... All the more reason for .DNG files.

11  DAM Stuff / Migration Issues / 2.8 TB Found on: July 21, 2011, 01:17:37 PM
I just completed the TIFF backup described above and was about ready to have Rob in our IT department start backing up legacy files to tape when he says to me, "What do you want to do about these?"

Turns out, there is another folder (previously hidden for no clear reason) containing 164,000 files in 6,200 folders - another 2.8 terabytes! Guess I gotta start on this file now too.
12  DAM Stuff / Migration Issues / Migrating 10 Terabytes on: July 20, 2011, 07:27:51 AM
Thanks to that DAM Book and all t the great comments on this site, I am well into the process of modernizing the image archives at the New York Botanical Garden Herbarium (http://sciweb.nybg.org/science2/VirtualHerbarium.asp).

NYBG was ahead of just about every institution when it came to digitizing it's collection of over 7 million plant specimens. They started back in 1999 and have amassed over 15 terabytes of images. Of course, this also means that they have gone through many years of format changes, changes in architecture, and changes in imaging practice.

When I started 6 months ago, I found an archive with 830,415 files scattered away in some 214,000 folders - thousands of the folders were empty, many contained subfolders with images tucked in them.

Using Lightroom, I created catalogs of all the entire archive (it took several days to load) so I would at least be able to see where everything was.

My first task was to clear space, because we were running out quickly since we are creating over 10,000 new images a month. We had close to 200,000 Tiff files, made prior to DNG's existence, eating up close to 8 terabytes of space. Since we have all these images as RAW files, I decided to get these on to tape and take them offline (http://digitalphotorepro.blogspot.com/2011/06/archive-deleted.html).

Before I did this, I wanted to make sure I didn't throw out any images that we didn't have as a RAW file. But how to do this? I searched the archive for .DCR and .CRW files (yes, we have about 20,000 Kodak RAW files) and created a list using NirSoft's Sys Explorer (http://digitalphotorepro.blogspot.com/2011/05/nirsoft-sys-exporter.html) a very handy utility that makes a list out of the contents of a Window's search window. I then made a search of TIFF files and created a list.

Now i had two uneven lists of files. My colleague, Tony Kirchgessner, created an Access macro that compared the two lists and generated a new list of files that existed only as TIFF files. I took this list of TIFFs only and created a CSV file and search my Lightroom Catalogues (http://digitalphotorepro.blogspot.com/2011/05/searching-multiple-files-in-lightroom.html) and made a collection of what I call our archive TIFFs.

This may have been a round about way of doing things, but I created a backup of all the files that only existed as TIFFs on an external hard drive and I am now in the process of copying them all back to our server. This way, although I have taken most of the TIFFs offline on tape, I am keeping every picture we have ever taken on the server.

It's a process. There are still 60,000 TIFFs to be moved offline and deleted, but at least I am not deleting any TIFFs for which there is no other format.

I have also consolidated all those crazy folders. So now, instead of 214,000 folders, I have a much more manageable structure with \Archive RAW, \Archive DNG, and \Archive Tiff (there are some subfolders necessary during the transition). Once I am done shuffling TIFFs, I am going to start converting all the RAW files to DNG.

The big appeal of DNG to me is the ability to store metadata in the file. For our consumers (scientists) they really don't even care about the picture. In most cases, all they want is the metadata. One nifty trick someone showed me is that you can open a DNG using a text editor and you can read all the stored metadata (this also works for Jpeg and TIFF but not for proprietary RAW files).

So, more to follow as I move forward. I am going to keep reading the DAM Forum to learn as much as I can about importing metadata and working with DNG. Thanks for everyone's help.
13  General / General Discussion / Update on: June 30, 2011, 01:47:18 PM
119,570 TIFFs deleted - freeing up 5.1 terabytes from our servers.

The TIFFs have been safely tucked away on tape, sitting on a shelf.
14  General / Photo Blogs / Deleted TIFFs on: June 30, 2011, 01:23:53 PM
It is a task. I expect to have the structure of the archive a bit more manageable by the end of the year.

Since the New York Botanical Garden has been archiving before DNG came about, they used the best standards available at the time. They would shoot a RAW capture and archive it, process a production master TIFF file and archive it, and create JPEG derivatives for use in the online database.

The TIFFs, all 7.5 terabytes of them, have never, to my knowledge, been used for anything other than to archive. I have just recently started to archive only DNG.

The TIFFs have all been backed up on tape, and I am in the process of deleting them from the server. As I type this, another 73,597 TIFFs are being deleted.

My next step will be to convert the various RAW formats to DNG.

15  General / Photo Blogs / Digital Photo Repro on: June 30, 2011, 07:54:06 AM
I started a blog to keep track of my activities as the imaging manager for the New York Botanical Garden Herbarium.

www.digitalphotorepro.blogspot.com

NYBG has been digitizing plant specimens since 1999 and has an online image catalog of several hundred thousand specimens.

Lately we have ramped- up production  and I am finding the DAM Book extremely helpful as I try to modernize the archive. There are about 15,000 Kodak DCR files, 150,000 Canon CRW files, and 250,000 TIFF files scattered throughout over 100,000 folders. Some have metadata, some do not.

I am trying to corral all the files, take legacy TIFF duplicates offline, and convert all remaining RAW files to DNG.

I hope you enjoy it.
Pages: [1] 2
Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!