I've seen the topic touched on in a number of threads, of course, but I'd love to see people who have systems they like tell us about them.  I'm not just talking about controlled vocabulary issues, but broader ones.  I am talking about deciding what sorts of things go into keywords, what sorts of things are organized by catalog sets, and what things are organized both ways.  How do people go about this?

So far I'm zeroing in on a catalog set hierarchy that seems like it will work for me, but my keywording is pretty erratic.  I'm a (mostly) non-professional photographer who is really a musician and landscape painter, and not coincidentally I shoot a lot of musicians and landscapes.  My top level catalog sets include:

Art (subsets for artists, paintings, sculpture)
Events (subsets for weddings, parties, specific events)
Jobs (someday I may have more than one of these...)
Music (subsets broken down by genre and by region)
People (subsets for specific individuals and one for candids of unknown people)
Places (subsets broken down by location and by type - landscape, cityscape, night, architecture, etc)
Things (subsets for inanimate objects, plants, and animals, the latter also broken down into cats, dogs, etc)

Plus of course, various "utility" sets that are not subject matter specific and are not really relevant to this particular discussion.

My cataloging software has a searchable "Notes" field that is free text, and I use it for a brief description that applies to a whole batch of pictures ("hiking @ Mount Falcon", "honeymoon @ Venice").  For keywords, I am currently duplicating some but not all of the catalog set information.  For example, if a picture is in the Music/Genre/Jazz and Music/Region/Colorado catalog sets, it gets keywords music and jazz at least.  Rather than duplicating region information, I will often use a keyword indicating the specific venue.  Also a keyword for the specific musician(s) pictured.  But for other types of pictures, I'm a lot more haphazard.  Many of my pictures in the Places/Type/Landscape catalog set have no keywords at all.  Location information is contained in the Notes field.  I suppose I should at least have a keyword of "landscape".  If I were more industrious, I'd add keywords that more specifically listed the elements.  But I don't know, most landscapes have trees, grass, sky, etc - unless it's something really out of the ordinary ("sunset") it doesn't seem worth my time to note anything more specific.  If I painted from photographs more I might care ("gee, I feel like painting a landscape with a grouping of trees and a pond - wonder if I have anything like that?"), but I don't paint from photos, so I don't need to answer those types of questions.

As you can see, my "system" pretty haphazard, and I'd like to improve.  I'm not really concerned about others browsing my images using keywords, so I'm pretty lax about using them at all.  Mostly its either for information I don't want to risk losing in the event of database failure or migration, or for information useful to me but perhaps too fine-grained to be worth using catalog sets.  Like "train" as a keyword in a picture that is in the Places/Type/Cityscape and Places/Location/Cinque_Terre categories.  Pictures in this latter categories also have keywords of "Europe", "Italy" and "Cinque_Terre".  But the same picture taken here in Denver might just get the "train" keyword, and another taken after the train has passed might get no keywords at all.  I'm thinking I should probably take all the major category names ("cityscape", "animal", "wedding", etc) and push them into keywords also.  Do others do this regularly?

Anyhow, I'm just interested to see other people's overviews of their systems, to give me ideas.  And I'm assuming I'm not the only one curious about this...

That was more or less where I started in my thinking too.  But I found I prefered using virtual sets as it would let me see groups of (potentially related) files that had been processed into derivatives.  If you use Peter's suggested naming scheme for derivatives, then you've already got a way to see by looking at a file whether it is an original, master, or what, and you've got a way to look for a given version of a file if you know it exists.  You've even got a way to find all versions of a given file by simply doing a wildcard search on "MJS_060612_0101*".  So it started to look like there wasn't any additional benefit to adding keywords or anything else that was file-specific to each file.  And indeed, I don't *need* the virtual sets either.  But they do seem to provide added value.  Especially in these early stages while I'm still working out the system.  It makes it easy to find all derivatives and the associated originals so I can rearrange, reprocess, or retag them if decide I want to do things differently.


I'm not sure that you can't do the same with virtual sets, although that will depend on your cataloging software, I suppose.  But while it's nice to be able to do these thigns, I'm thinking the vast majority of the time, the basic searches I do will be of the form "show me all (family, sports, flower, whatever) photos, using the master if available, original if not".  And I don't want to have to include that same language into every search I do to get that result.


This is indeed what version control software I have used basically does, and I'll be glad when it becomes more standard in cataloging software.  I guess I've been assuming the "master" would always be the one I'd want to identify as "current", but of course, that need not always be the case.  However, it seems you'd have to always type "& current" on the end of all your searches in order to get this filtering.


I'm not suggesting making any more or fewer edits than you would have otherwise.  I'm simply suggesting that if master and print are one and the same, that it makes more sense to me to call that one thing "master", at least as far as the actual file name is concerned.  Then you can always use a tagging or virtual set scheme to also identify it as the "print".  Actually, this immediately suggests a way your idea could be rolled into mine: have another subset of "Derivatives" called "Current".  It would start out by containing all the originals in the entire catalog.  When you create master, you'd remove the original from "Derivatives/Current" and add the master there (as well as adding it to "Derivatives/Current").  If you then created, say, a crop of the image that you wanted to treat as the primary or "Current" version, then you would add that to "Derivates/Current" and remove the master from that virtual set (or not, I suppose - no reason there couldn't be two "current" versions if you had two different crops you liked, for example).

Note in particular I said, you wouldn't want to see the original unless you were specifically looking for it.  Creating a new derivative after the master seems like a good reason to go back to the original.  But really, if you save the master as JPEG at highest quality, you might be able to live with the one additional generation loss you get by making further derivatives from it rather than the original most of the time.  What you gain by doing this is not having to reproduce the work you put into creating the master in the first place.  And if the idea of the second generation loss bothers you too much, you can always save your master as TIFF or PNG - the latter being a format with lossless compression that is closer to JPEG than TIFF in size.  Only downside is that PNG doesn't support much in the way of metadata.  Personally, I'm thinking I can live with the JPEG generation loss if always save at highest quality.  I don't really expect to be saving derivatives other than masters very often.

See my reply in the "dumberer" thread.  I had read a few other threads on this topic, and asked a few questions myself, and am becoming convinced that there are some issues unique to folks like us that probably do have solutions, but they haven't really been worked out yet.  And I think this forum is a fine place for us to do so together.


My virtual set method would work for this too, although if there are all sorts of derivatives, just knowing that, say, a 4x6 version already existed wouldn't be *obvious* unless you looked at the whole bunch of these virtual sets underneath the "Derivatives" set and sorted on name - then you'd see all derivatives made for each file side-by-side.  But that's better than having to type in file names trying to see if there is an MJS_060612_0101_4x6 already your system.


Why not call this a master and skip the print-derivative?  I think the trick, again, is to have the master replace the original for most purposes - you don't want to even see the original without specifically looking for it.

Marc Sabatella

This also confounded me at first, and I have to say, the divergence still bothers me a little.  Mostly because I'm still shooting JPEG for now, so I can't do even simple exposure or color adjustments without creating a derivative.  So I may be leaning more heavily on derivatives than some do.  Although I don't feel the need to do even  adjustments very often, and not (just) because I'm not a "serious photographer".  It's worth noting that I'm also a semi-professional landscape painter, and I do virtually all my work in the field and only rarely revisit my paintings in the studio later.  And actually, my real profession is music - specifically, I am a jazz pianist.  Most of the music I create is improvised - this has a similar what-you-see-is-what-you-get, take-it-or-leave-it vibe.  So my attitude regarding my photos is in keeping with my attitudes about my other passions.  On the other hand, I'm quite convinced I'll be shooting mostly RAW at some point soon.  But for various reasons, I'd prefer to get a really good JPEG workflow figured out to use for now and keep re-evaluating when it makes sense to switch as time goes on.

Since I ended up going with ACDSee Pro (despite some very valid concerns raised by Peter), I have no built-in version tracking, but I'm experimenting with ways of tracking derivatives myself.  One idea I'm playing with is setting up nested virtual sets just for this purpose.  As I have it currently, I have a top-level set called Derivatives, and underneath that are sets for Original and Master.  Right now, that's the only kind of derivative I'm really interested in keeping - any others I use are usually created and deleted automatically for me (when, for example, I choose to send some as screen-sized email attachments).  As I'm doing my initial work with images assigning metadata and rating, if I see any I feel need post-processing, I toss them in my "Derivatives/Original" virtual set.  If I'm feeling industrious, at that time, I'll also make my adjustments and save them into a new file.  Just as one would in the "usual" DAM Book workflow, the new file goes into the derivative bucket, which is completely decoupled from the originals bucket.  But I'll also assign it to the "Derivatives/Master" virtual set.  At that time I can also copy the metadata and assign it to all virtual sets the original was in, and perhaps *de-assign* the original or even remove keywords from it so only one copy of the file shows up in future searches.

So far I have just one big set for the originals and another for the masters, but of course, I could subdivide these into virtual "buckets", which could correspond to the actual originals buckets, or be content-based.  Assuming I actually do follow through and make masters for all files in my Derivatives/Original set, then there will be a one-to-one correspondence between originals and masters, which I figure may help me keep track of them down the line.  I like the way this makes it easy to view just derivatives, just originals, or both side-by-side (by viewing both sets and sorting by name).  That way if I later upgrade my processing software - or, more to the point, if I upgrade my own processing skills :-) - I can easily browse these images and see which might be worth revisitng.

But I think the main idea here is that my master for most practical purposes replaces the original.  In fact, I would generally be treating my originals more like RAW files that have been converted to DNG but are still being archived just in case.

Anyhow, I'd appreciate any feedback on these ideas.  And feel free to remind me of what a mistake it is to be shooting JEPG or using ACDSee Pro; at some point, I'll write up a defense of these choices for myself.  But one thing that impressed me on reading Peter's book is that the ideas in it are *not* just valuable for the professional, but many of the specific techniques described are going to seem so daunting to many amateurs that they won't go there at all, or will make only token efforts that miss the point (eg, the guy who said he wasn't sure he needed a database...).  I'm interested in working out a "DAM for the rest of us" approach that is as true as possible to the ideals put forth by Peter.  And given that "the rest of us" is going to look a whole lot like the set of folks for whom shooting RAW is just out of the question, and hence will not be able to take advantage of ACR adjustments and DNG in doing the sort of basic post processing that even we like to do, we're going to need a way to deal with dealing with the derivatives we create.

The most obvious benefit as far as I am concerned are virtual categories.  This alone is so huge, it seems hardly worth mentioning other advantages, such as speed of search...

I'm looking to start a discussion on things to think about when trying to construct a DAM workflow based on shooting JPEG.  I'm mostly thinking aloud here, but I'm hoping those of you have have given the matter some thought will chime in.

As someone who has until now primarily shot JPEG and is just starting to adopt DAM practices, I'm at something of a crossroads.  I know the recommendation from most here will be to just go ahead and shoot RAW all the time, and I am certainly considering that.  Up until now, my rationale for shooting JPEG has been twofold.  One, it seemed more convenient to not have to make the conversion to JPEG or TIFF if I wanted to work with the file - the RAW processing applications I had been using did not allow you do anything much with the RAW file itself.  Two, given that Pentax RAW files are not compressed, working RAW really limits how much I can fit on a card and how fast my disk fills up.  The workflow described in "The DAM Book" seems like it largely addresses most of these concerns.  ACR and other RAW processing applications I am looking at seem like they'll let me do virtually everything I want to do without ever making a conversion to JPEG or TIFF, and the DNG's created from my RAW files are only barely bigger than the JPEG's.  Card capacity while shooting is still an issue, but bigger cards are getting cheaper, too.  So I can certainly see myself going to mostly RAW soon.  However, I am also trying to fully think through the implications of sticking with JPEG for the majority of my shooting, both to help me make sure I'm making thre right decision, and to give me something concrete to recommend to people as I spread the "gospel" of DAM around to folks who I know will nto consider RAW.

It seems that there are a number of issues that arise that affect the workflow in non-obvious ways, and as I work through these, I'm wondering if/how others are dealing with this.  Most importantly, the idea that you can make some basic adjustments to your camera originals without requiring new derivatives is a huge advantage of RAW over JPEG.  For me, I could see these adjustments would cover the vast majority of the cases that might normally cause me to create derivatives in the first place (eg, camera original JPEG is underexposed or shot with inappropriate white balance).  As it is, I probably only actually make such adjustments to my JPEG's less than 10% of the time, but if there were a way to make this work as well as the process described in "The DAM Book", I'd probably take advantage of this quite a bit more.  I note that some applications, like Picasa 2, actually store all edits to JPEG files in a database and/or sidecar file, leaving the original JPEG alone.  Most "serious" DAM applications, it seems, do not take this approach, and I'm sure Peter would be suspicious of an approach in which these adjustments were stored in a proprietary way, anyhow.  So if I wanted something similar to the DNG-based workflow described in the book, I can see a few options:

1) If the DNG converter could somehow take a JPEG file and create a DNG file from it, I could actually use virtually the same workflow as with RAW files.  The only real difference would be that I'd be applying my adjustments post-conversion rather than pre-conversion.  Of course, I realize the DNG created wouldn't *really* be raw data, but at least RAW processing programs would be able to work with the information.  Actually, it seems to me such a converter would be extremely cool.  Searching around, it looks like the Lightroom beta has this feature, but I'm not sure I want to bank on that right now.

2) Assuming I can't get #1, next best thing would be to convert my JPEG's to a lossless format that supports layers and use that as my "original", much like I might convert my RAW file to DNG and use that as my "original".  Then I could apply any necessary adjustments to a layer and preserve the original image data, but would not need to keep the actual original JPEG.  Unfortunately, the formats I know of that support layers are not particualrly compact; if I had to convert all my JPEGs to PSD or TIFF, I'd be losing much of the storage advantage of shooting JPEG in the first place.  On the other hand, just making the conversions for files I actually apply any adjustments to could make sense, if I could make this process reasonably painless.

3) If I can't easily get a single file that contains my original data as well as my adjustments, then I have to live with "master" derivatives for every file for which I want to correct exposure or color.  This would result in the need for far more derivatives than using DNG, but I could live with this *if* there were a straightforward/automatic/foolproof way of maintaining my derivatives and their associations with the originals.  For instance, I'd want only the master to appear by default in any virtual catalogs I create or show up in searches, but I'd want a simple way to find the original given the master, and to know right away that there is an original to be found.  I gather idImager has some sort of version control built in.  It also seems one could work out a scheme using filenaming, keywords, IPTC data, or database fields to track this sort of thing without explicit support from the cataloging software.  I suspect folks are already doing this to some extent right now in a RAW workflow, but one probably would want the process to work a bit more automatically if one were having to create derivatives any time one wanted to tweak exposure or color.

Anyhow, sorry for being so verbose.  I'm hoping that some of you have given these matters some thought already, and by explaining where I am coming from, perhaps get some good relevant discussion happening.

Thanks for confirming my suspicion.


That is indeed impressive, and it makes me feel better about the viability of using that preview for something other than thumbnail display.  I'm not sure it directly addresses my concern about differences in rendering between programs, except by suggesting I wouldn't actually *need* to ever render mosts file a second time.  Interesting.

Right.  Like I said, I'm sold on DNG in general.  Being able to save keyword and other standardized metadata in the file itself is indeed an important advantage I neglected to mention.  Here's where I have complete faith that even programs that don't *currently* support this will do so in the very near future, because it strikes me as a no-brainer.  Obviously a good thing to do, obviously technically feasible.


But that would be significant to me only if the format actually supports a *standardized* way of doing these things, so metadata written in one program can actually be understood and correctly intepreted by another.  Writing a preview seems straightforward, so I'm sure you're right that others will jump on board here - applications should be able to write previews directly into a DNG file whether or not there is a published spec on how to pass adjustment data into the Adobe DNG converter, and obviously they can read them.  I'm more concerned about recording the adjustments themselves, and in particular, about *communicating* those adjustments between applications.  If there is no standard for how this data is stored and interpreted, then storing this data in the DNG file seems no better to me than storing it in sidecar files or a database.

For example, it seems it would be *possible* to define a standard for how "curves" should behave - how many points should be settable, how their X&Y coordinates should be expressed in the XMP metadata, what shape the curve should take between set points, exactly how a given X,Y value on the curve should be interpreted in a given color space, etc.  But based on my previous experience participating in the development of technical standardizations, and based on the fact that I've never seen any mention of these issues in any discussion of DNG, it seems quite likely to me that such standardization does *not* yet exist.  Furthermore, it seems likely to me there would be some amount of resistance to such standardization from the various companies involved with RAW processing.  Everyone wants to be able to claim they do things "better" than the competition, and in the context of actually producing an image, a "better" way of interpreting the same metadata can only mean "different from the standard".  The usual result of these conflicts is for the standard to specify only behavior for some sort of "least common denominator", which in this case might mean, for example, only color temperature/tint, a single value for sharpness, brightness, and contrast, and for each application to implement other features (eg, curves, unsharp mask settings) in incompatible ways.  Now, I realize DNG is completely under control of Adobe - it isn't the result of the usual IEEE standardization processes.  But I still have this suspicion that this is not a solved problem.

This is why the overal tone of my initial post was one of skepticism.  Again, not skepticism as to the value of the format in general, but skepticism over the extent to which it really addresses all of the interoperability problems one might encounter in migrating between software packages.  Regarding adjustments, all I can tell for sure is that *if* you use ACR, then your adjustments will be preserved as you upgrade to new versions of ACR.  But that should be the case for *any* RAW processing software that allows saving this information at all, whether in sidecar files or in a database.  The situation appears a little more promising regarding the preview - like I said, even if other programs currently are not capable of writing such previews, it seems likely to me this capability will show up soon enough.  But I'm less concerned about the preview itself than the work that went into creating it.  And again my reason for concern isn't in deciding whether to use DNG or not - that's a done decision for me - but in deciding just what capabilities I really need from the software I am trying to choose.

I realize that DNG allows for embedding a previewing, and that's a great thing, but I'm concerned here about the adjustments used in creating that preview.  While in theory the format allows the settings for the various adjustments used to create the preview to be saved as XMP data within the file, I'm wondering how much standardization there is in this regard.  My perception is that every RAW processing program I have seen has slightly different controls, and indeed produces slightly different results for similar settings of similarly-named controls.  So I am not really confident any other given program would be correctly interpreting the exposure, sharpening, or other settings made by another program.  For example, if I initially use ACR to do my adjustments and then generate the DNG file, but later on I decide I want to work on the DNG file with RawShooter, is it reasonable to assume that the latter would render the file pretty much exactly the same way as ACR did given the settings stored in the DNG file?  If not, it seems to me that some of the advantage of the "openness" of the DNG format is not as great as it would first appear: all the work that goes into making initial adjustments would be lost (except for seeing a preview) if you changed software later.

On a related note, it isn't clear to me if the Adobe DNG converter listens to adjustments made by other programs, or if it only knows about ACR's adjustments.  Obviously, ACR must be passing metadata information through to the converter to embed in the file, perhaps through sidecar files.  Is this interface documented, so other programs can in theory do the same when they invoke the DNG converter?  That is, if I make my initial adjustments with SilkyPix instead of ACR, and then invoke the DNG converter from SilkyPix, is the converter going to be able to understand the adjustments made within SilkyPix (assuming SilkyPix actually tries to communicate them)?  If not, it seems even the embedded preview feature would only be useful if you do your initial adjustments with ACR.

Don't get me wrong, I'm pretty well sold on DNG in general, if for no other reason than they can be compressed whereas my Pentax RAW files are not, and of course just being assured that the files always will be readable at all is comforting.  I'm still trying to sort out the extent to which I really need to use Adobe products in order to realize the full benefit of the DAM principles.  Elsewhere, I had asked about all-in-one solutions like ACDSee Pro, and I appreciated the responses from Peter on that thread.  But even assuming I don't go that route, I'm not convinced Adobe Elements would be as strong a solution for me as using idImager, possibly combined with RawShooter or even the Pentax-supplied RAW software I already have.  If I have to use Adobe software in order to gain the supposed interoperability advantages of DNG/XMP, doesn't that seem a little contradictory?

I'm definitely considering that, although then of course, I still need a cataloging program on top of that.


That makes sense.  One thing I should also have mentioned is that I do relatively little RAW shooting - although I certainly recognize that could change.  With the bulk of my pictures shot in JPEG, and most of it not edited in any way after that, I'm less worried about losing data entry work, because I *can* write almost everything I do to the JPEG IPTC metadata if I choose (and this can be done automatically after the fact).  But I do shoot *some* RAW, and anticipate I'll be shooting more as opposed to less as time goes on.  What I'd be banking on is that ACDSee would upgrade the product to write to DNG at some point before I'd need to abandon the program.  A couple of things work in my favor here.  One, ACDSee has been around quite a while and probably will be for quite a while to come.  And since I'm not a professional photographer, I'm not going to oushing the limits as fast as some of you.  Meaning it may be a long time before I outgrow or otherwise need to abandon ACDSee, so they'll have plenty of time to add the functionality I'd need to make that easier.  But I do agree it is a concern I should take seriously.

Thanks for taking the time to answer, and giving me things to think about.

Marc Sabatella

The workflow outlined in "The DAM Book" involves at least four more-or-less separate applications - Bridge, Camera RAW, iView MediaPro or other cataloging software, and Photoshop.  A lot of the considerations given for choosing a cataloging program hinge on the fact that it has to communicate with the other applications.

Well, as someone who doesn't currently own Photoshop CS, and who doesn't particularly love any of the browsing, RAW converting, or photo editing programs I currently use, and who frankly is not a professional photographer and has somewhat less demanding needs, I'm not convinced an all-in-one solution wouldn't make sense.  The advantages as I see them are:

- communication between the aspects of the program would be pretty automatic (no worrying about whether the cataloging program sees the ACR conversion settings, no need to push ratings into keywords to pass info between Bridge & cataloging program, etc)
- in theory, such an application could be compact and fast
- in theory, such an application could be relatively cheap, if I don't need the full editing capabilities of Photoshop

I've been looking hard at ACDSee Pro, mostly because I started doing so before reading "The DAM Book".  I'm finding that while it does have limitations such as those described elsewhere on these forums, most of those limitations are really only problems if you are planning to use the program with the Adobe products (eg, not using embedded previews in DNG files).  So far, I'm not seeing much to convince me this wouldn't allow me to reap virtually all of the benefits of the DAM workflow as described in the book, although I certainly understand why some might be reluctant to throw all their eggs into one basket.

Anyhow, two questions come to mind.

First, aside from general issues with the whole idea of all-in-one solutions, and realizing that no all-in-one package will have editing capabilities as powerful as Photoshop, what other specific factors might cause people to *avoid* an all-in-one solution in favor of the Bridge-ACR-iView-Photoshop type of workflow?

Second, assuming one *is* looking for an all-in-one solution, what other programs seem worthy of consideration?  I'm getting very curious about idImager after seeing the comments in this forum, and will probably be checking out that trial version out soon.  It looks like its cataloging capabilities are better than ACDSee Pro's, and the version tracking capability in particular is most intriguing (I'm surprised there wasn't more discussion of tracking issues in "The DAM Book").  I also appreciate the developer's relatively "open" design approach.  It seems from what I can gather that the RAW processing and editing capabilities are probably not as strong as ACDSee Pro's, though, and it also seems likely to be rather more awkward to use.  I imagine that within a few years, both programs will have implemented the better features of the other, and ultimately it won't matter which I choose now.  But I'm all ears if people have suggestions along these lines.