Bucket system with Catalog Sets?

[drmrbrewer]

Hi Dan

If I'm understanding you correctly, it sounds like you are using virtual buckets (i.e. virtual sets) within your cataloging software to emulate Peter's physical bucket infrastructure.  

Well, yes and no.  I'm not actually doing this, but just exploring this as an option.  But yes, that's what I'm thinking.

You are then burning your DVD's directly from your cataloged virtual sets/buckets.

Yes.

If I'm correct here, on the surface this might seem more flexible because you're not limited by a physical infrastructure.  However, the problem I see with this is it ties you completely to your cataloging software to maintain your infrastructure as well as to burn your archive.  That's something I would be fearful of doing.  If you ever migrated to a new catalog that couldn't import your virtual bucket sets from your old catalog, it could be ground zero again.

That is a point, although MediaPro does enable you to sync catalog sets to the file, so in theory it should be possible to pick up those sets in the new software and use them in an analogous way with whatever interface it provides.  But yes, I can see that having your buckets independent of the software is more robust.  Especially since at the moment I think catalog sets are only synced to certain types of file (e.g. NOT to NEFs)... but I would hope naively that this will improve; iView seem to do the best they can to make it easy to get data out in as many ways as possible.

But then again, Peter advocates liberal use of Catalog Sets in his book to add value to your catalog by creating image groupings, and you would have exactly the same issues in respect of these when migrating to new cataloging software.  I bet you'd think twice about jumping ship to new software if you knew you'd lose all those.

Also, I wouldn't be reliant on the cataloging software to do the burning; I'd hope there is burning software out there that can accept dragged-and-dropped files from my MediaPro catalog sets to tell it what files to burn and where they are, with full hierarchy maintained.  I haven't looked into this.  Perhaps my current software can, I'm not sure.  But it would seem an obvious interface to have.  

Personally I don't really see any advantage to maintaining your bucketed infrastructure virtually because you still have to keep track of the bucket size and burn the DVD's just the same.

I can see advantages in:

(a) setting them up in the first place (just run a script);

(b) migrating them to a new media size (just run the script again with a larger number plugged in);

(c) allowing you to maintain a physical folder structure that means something to you, keeping related files together physically (although maybe in different virtual buckets), thereby allowing you easily to navigate to find files related to a particular shoot without relying on your cataloging software to get there; and

(d) that you're not tied to the limitations of a rapidly aging media technology.  I mean, 4 Gigs is really quite small nowadays, and the thought of splitting up my entire hard drive into tiny 4 Gig chunks doesn't appeal, especially knowing that I'll have to re-group everything in a couple of years' time for the new media format.  Doing it virtually -- well that's different.  To me it would be like sticking to eight-character filenames just to maintain compatibility with an old technology (OK that analogy is pushing it a little, I know).


Having said that, I really can see there are advantages in having your files split into physical folders that correspond exactly to the DVDs.  So I'm torn.

Thanks,

Mike

[danaltick]

Mike,

(a) setting them up in the first place (just run a script);

You can do the same thing with physical buckets using O/S scripts.

(b) migrating them to a new media size (just run the script again with a larger number plugged in);

Just encapsulate the smaller physical buckets into larger physical buckets as Peter outlines in the book...very easy.

(c) allowing you to maintain a physical folder structure that means something to you, keeping related files together physically (although maybe in different virtual buckets), thereby allowing you easily to navigate to find files related to a particular shoot without relying on your cataloging software to get there; and

Keeping all of my files in one big physical folder does not mean anything to me.  I think it's much better to use job folders within the buckets as Peter outlines in the book.  I just don't see that keeping your derivatives co-located with your originals is that important.  In my archive, I actually use an internal bucket structure that keeps all my derivatives together and separate from my originals.  I would rather depend on my catalog to keep my originals co-located with my derivatives (If I care to do that).  To me that's better than depending on my catalog for everything.

(d) that you're not tied to the limitations of a rapidly aging media technology.  I mean, 4 Gigs is really quite small nowadays, and the thought of splitting up my entire hard drive into tiny 4 Gig chunks doesn't appeal, especially knowing that I'll have to re-group everything in a couple of years' time for the new media format.  Doing it virtually -- well that's different.  To me it would be like sticking to eight-character filenames just to maintain compatibility with an old technology (OK that analogy is pushing it a little, I know).

See answer (b) above.

In summary, we're back to virtual buckets just not really buying you anything except sole dependency on your catalog.  And yes, even if you could copy your virtual buckets to a physical folder before burning them, that still means you have to copy your buckets every time you want to burn...a major inconvenience; especially once we go to Blu Ray.  Also, what about your backup software.  Current backup software, like Retrospect and GBM do not know anything about virtual buckets.  And you certainly don't want to be mirroring your entire physical archive.  I could just see Peter attempting to do that.  It would probably bring his whole operation to a crawl.  Ideally you only want to mirror the current bucket.  Hope that helps.

Dan

[drmrbrewer]

Hi Dan

And yes, even if you could copy your virtual buckets to a physical folder before burning them, that still means you have to copy your buckets every time you want to burn...a major inconvenience; especially once we go to Blu Ray.  

I would be dragging and dropping from a MediaPro bucket set onto my DVD writing software.  This doesn't physically copy any files, it just transfers the paths of the media items into the DVD software.  Real quick.  It's a simple way of just letting your DVD software know what files to write, without having to have them all in the same physical folder.  Nice and easy... Ctrl-a in MediaPro to select all media items in the bucket set, drag and drop onto the DVD software window, press Go, and that's it.

Also, what about your backup software.  Current backup software, like Retrospect and GBM do not know anything about virtual buckets.

I wouldn't use my backup software to backup to DVD.  It wouldn't need to know about virtual buckets.  I would use my backup software to do incremental backups of my hard drive, e.g. overnight, to an external hard drive and/or an external ftp site I have set up.  It would by design only backup those images that have changed recently, wherever they might be in my physical data structure.  It doesn't need a bucket folder structure to tell it what has changed recently and what hasn't.

And you certainly don't want to be mirroring your entire physical archive.  I could just see Peter attempting to do that.  It would probably bring his whole operation to a crawl.   Ideally you only want to mirror the current bucket.  

Now put yourself in the position of someone other than yourself or Peter, like a keen amateur who doesn't really need to go to the extreme of mirroring certain parts of his/her disk to multiple drives, who can adequately protect his/her valuable images with incremental backups (and the occasional full backup) to multiple locations including to a set of DVDs.  (Peter's book is, after all, also aimed at such people.)  Would you still consider that a virtual bucket set system has no merits; such a person would find nothing in any of the advantages I gave previously?  I really don't understand your point of such a system making me more dependent on my cataloging software.  I'd really be no more dependent than I already am; I already make heavy use of catalog sets and would only realistically ever be able to leave MediaPro behind if I could take my catalog sets with me.  Besides, you can always script an exit strategy if there isn't one provided for you.

Mike

[danaltick]

Mike,

You're right, there is nothing to prevent you from using virtual buckets in your cataloging software and just dumping all you images into a single folder on your hard drive.  Just keep in mind, this does deviate from Peter's definition of DAM; which relies on physical buckets for unlimited growth capability, with some degree of physical infrastructure.  If you feel you will never have an archive that grows big enough to concern yourself with that, and as long as you aren't concerned with incremental backups of your entire archive, then by all means go ahead and use virtual buckets.  Be sure to let us know how it goes.

Best Regards,
Dan

[peterkrogh]

>Just keep in mind, this does deviate from Peter's definition of DAM

Just to be nit-picky, this is not my definition of DAM, just the way I am currently implementing it, working with the limitations of current technology.

I fully expect that we will see a day before too long where a single storage device will be able to store all the pictures you shoot in your lifetime.  At that point, the buckets will be much less necessary.
Peter

[drmrbrewer]

Hi Dan

You're right, there is nothing to prevent you from using virtual buckets in your cataloging software and just dumping all you images into a single folder on your hard drive.

But a key point of my argument in favour of using virtual buckets (my advantage (c) above) is that it allows you to retain whatever physical folder structure you like.  I wouldn't ever consider just dumping all my images into a single folder and subdividing only using virtual buckets.  My folders are divided according to Year and then Date of Shoot, as set out early on in this thread.  This structure means something to me, and it's easy to navigate.  All related images (not all images, just all related images) are in one place, not scattered around the place.

If you feel you will never have an archive that grows big enough to concern yourself with that, and as long as you aren't concerned with incremental backups of your entire archive, then by all means go ahead and use virtual buckets.

My feeling is that drive capacity and performance for the same money is increasing at least as fast as my image collection, so I'm unlikely ever to run into problems in this respect.  I expect I'll always be able to house my entire collection on a single drive.  And an incremental backup really doesn't take that long, and certainly no longer than 10 hours overnight.  It's more like about 2 minutes (maximum) at the moment for an incremental backup, and I can't see it going beyond 10 hours in my lifetime.

Be sure to let us know how it goes.

I will... all I have to do now is wait for iView to get their scripting engine sorted, because proper access to catalog sets via JS / VBS is currently broken.

I appreciate you sharing your views.  Many thanks.

Mike

[danaltick]

Mike,

This has actually turned into an interesting topic that I had never really considered until you made me aware of it.  When I first read the thread, I had to read it a couple of times to make sure I understood your objective.  My hats off to you for thinking outside the box and recognizing this approach.  It did make me stop and think.

I want to follow up with two questions though.  Peter, the first one is to you:

I fully expect that we will see a day before too long where a single storage device will be able to store all the pictures you shoot in your lifetime.  At that point, the buckets will be much less necessary.

I can certainly see this happening for hard drives, but I'm not so sure I see it happening for write-once media...any comments?

And Mike:

I'm actually a computer engineer and programmer studying to become a photographer.  One thing I've learned through all my years of programming is that unless creating a virtual layer actually offers you a degree of freedom that you wouldn't otherwise have via the physical layer, then the price you pay is usually performance and complexity due to an extra level of indirection.  That's really where my question still remains.  I can't say that I'm convinced at this point that an extra virtual layer is worth it.  All I see it really doing is creating a duplication of effort, once in the physical and again in the virtual; assuming that you are maintaining structure in your physical, which you say you are.

Dan

[peterkrogh]

Peter, the first one is to you:

I fully expect that we will see a day before too long where a single storage device will be able to store all the pictures you shoot in your lifetime.  At that point, the buckets will be much less necessary.

I can certainly see this happening for hard drives, but I'm not so sure I see it happening for write-once media...any comments?

Dan,
I have no insider information on this, just observation of a steady geometric progression in storage capacity.  I'm assuming that there are lots of smart people working on new storage devices.  Eventually it will be here.  And although it might not be write-once, I suspect that locking will be an available feature.

Peter

[AlanDunne]

Mike,

I have been following this thread and even contributed a thought or two along the way. I am beginning to see how you are relate the buckets idea to catelog sets, to create "virtual buckets". I commend you for bring new ideas to this problem of DAM. I tend to agree with Dan, however, that I cannot yet see that the problem you are trying to solve via virtual buckets cannot be solved by other means. For situations like this I would use a basic cost/benefit approach. Is the cost of the extra complexity (virtual layer) worth the benefit (theoretical more flexible buckets).

For me personally, with the tools available today, your virtual bucket idea does not pass the cost/benefit test. This is novel thinking, however, and I will see if this approach merits re-consideration in the future.

Great topic ... Al

[danaltick]

Peter,

Well, I don't see that happening anytime too soon.  I figure we will have to get a few generations beyond Blu-Ray first, assuming Moore's Law continues to apply.

And one other thing we should probably factor in here too.  Digital cameras are continuing to progress as well, with more and more megapixels creating bigger and bigger Raw files.  I suspect this will prolong it as well.

Dan

[drmrbrewer]

Hi Dan, Al, Peter,

Dan wrote:

I'm actually a computer engineer and programmer studying to become a photographer.  One thing I've learned through all my years of programming is that unless creating a virtual layer actually offers you a degree of freedom that you wouldn't otherwise have via the physical layer, then the price you pay is usually performance and complexity due to an extra level of indirection.  That's really where my question still remains.  I can't say that I'm convinced at this point that an extra virtual layer is worth it.  All I see it really doing is creating a duplication of effort, once in the physical and again in the virtual; assuming that you are maintaining structure in your physical, which you say you are.

I would say that having the virtual layer does offer a significant degree of freedom over doing it at the physical layer instead.  It allows you the freedom to have whatever physical structure you like, unconstrained by whatever the current limit of technology is.

What's more, the process of splitting with virtual buckets is also completely reversible; in fact because it's virtual, there really is nothing to reverse.  Your physical structure remains intact, allowing you to move things around and re-organise in the future without any constraints.  With physical buckets you're stuck with what you decide at the time you decide to do it.  With virtual buckets you have the freedom to change.  

It's as if using physical buckets obeys the law of entropy, while using virtual buckets does not.  With physical buckets you divide everything up into little bits.  It's like dropping that cup on the floor and watching it shatter into pieces.  New items then go into different bits to the previous bits even if they are related to previous items.  With a 4 gig limit even a new batch of images will end up in different bits.

Re duplication of effort, and extra complexity.  There really is no effort in the physical layer.  It's all done by my import software.  I've decided on a convention and the import software does everything for me, putting the images in the right place with the right names.  There is effort in the virtual layer, but no more so than with physical buckets -- in fact perhaps even less so because I would hope to be able to script away a lot of the effort, for example by getting a script to find the most recent bucket and to place newly-imported items into that bucket up to the limit, creating a new bucket if it starts to overflow -- most of these sort of tasks would be automated.

Mike

[danaltick]

Mike,

I think we might be clouding the picture by introducing automation scripts into it. Automation scripts can be run at the physical layer just as easily as the virtual.  I believe this is really more of a design concept independent of any automation processes.

Even though you’ve created a virtual layer, you are still constrained by a physical attribute, that being the current size of your write-once media.  I get around this limitation through my normal use of catalog sets; which are completely unconstrained.

It’s true you do have the freedom in the virtual layer to re-organize your images as you see fit as long as you stay within the constraints of the current bucket size.  However, you also have that same freedom in the physical layer.  It’s just as easy to drag and drop physical folders as it is virtual (or write a script as the case may be).  Even though I’m calling it the physical layer, it’s still at the operating system level.  The files and folders are really just links to the physical blocks on the disks; hence, we already have our virtual connection in the O/S.

It’s inefficient to mirror or incrementally backup an entire archive for only a few daily or weekly changes.  I would imagine this could take several hours or more for a terabyte archive just to scan through it looking for changes.  My PC’s run many maintenance tasks throughout the night besides just backups.  I can’t afford an inefficient use of my computer resources even in the middle of the night.  Also, I would never incrementally backup my images, that’s unnecessary and requires too much disk space.  I use only mirror jobs for my media.

I really don’t see myself needing to go back and re-organize my physical buckets once they’ve been filled.  That’s what my catalog sets are for.  The only thing I see needing to do is encapsulate my smaller buckets into larger one’s as we move to new write-once media; which again is something I can do just as easily at the physical layer as I can at the virtual.  One problem I see with this virtual layer if I ever did decide to re-structure or migrate my archive to a new set of hard drives is not only would I have to reconnect my own catalog sets, but I would also have to reconnect all my virtual buckets; something that would probably have to be automated if I re-structured the buckets themselves.

I’m afraid at this point I still fail to see any real benefits here, only more maintenance and baggage in the catalog due to an extra set of links requiring automation scripts.  However, please feel free to implement this yourself and keep us updated on any progress or discoveries you make along the way.  I never rule out that I could be overlooking something.  At the least, this has been educational.

Best Regards,
Dan

[drmrbrewer]

Hi

It’s true you do have the freedom in the virtual layer to re-organize your images as you see fit as long as you stay within the constraints of the current bucket size.  

There are no constraints when I re-organise my images at the physical level with the virtual layer in place.  That's the point.  I would never have to reorganise at the virtual level, as such, because all I'd do is just run the script again to re-divide the images amongst the virtual buckets.  Re-organising to me is a gradual and careful transition from one state to another, with some sense of where you've just come from at each stage and also where you're going.  With virtual buckets, you just scrap the past entirely without looking back, and re-partition without any care in the world.

Also, I would never incrementally backup my images, that’s unnecessary and requires too much disk space.  I use only mirror jobs for my media.

I'm interested.  Please explain more.

The only thing I see needing to do is encapsulate my smaller buckets into larger one’s as we move to new write-once media; which again is something I can do just as easily at the physical layer as I can at the virtual.  

You're still left with a reminder of days gone by, though.  Presumably you still have all those little DVD-sized folders within the new, bigger, Bluray folders?  Either that or you empty the DVD buckets into the new buckets and mix them all up together in one great primordial soup.  And eventually when the mythical write-once media comes along that can store ALL your data, you just have one great big mass of files in a single directory (either that, or lots of little 4 gig folders).  I'd rather keep my images ordered meaningfully both at the physical level and the virtual level.

However, please feel free to implement this yourself and keep us updated on any progress or discoveries you make along the way.  

Progress so far:

1) Tested the Catalog Set scripting interface again in the latest build of MediaPro.
2) Still broken.
3) That's it.

Cheers,

Mike

[danaltick]

Mike,

Regardless of whether you burn your images from virtual buckets or physical buckets, you are still constrained to the size of the physical DVD.  Once again, you're clouding the picture with scripts.

For an explanation on incremental and mirror jobs and when to use each, see this thread http://thedambook.com/smf/index.php?topic=75.0.

One other reason for using mirror jobs with your media that I didn't mention in that thread is the fact that they are mirrors of the primary storage, making them hot swappable if you lose the primary.

Mike, I think we've beat on this one long enough, so I'm going to hand it over to the other members for further comments.  I wish you the best of luck; howeever, and do keep us updated.  It's been fun.

Dan

[drmrbrewer]

Hi Dan

Regardless of whether you burn your images from virtual buckets or physical buckets, you are still constrained to the size of the physical DVD.

I'd rather be constrained virtually and reversibly than physically and irreversibly.

Thanks for the link to the topic on backup.  Interesting.  This still leaves an issue unclear in my mind re backups and buckets...

1) Presumably you have data other than images which are important to you.  Would you use a DVD-sized bucket system for ALL your own data?
2) Say you are implementing a backup regime for a multiple user system for a medium or large corporation with terabytes of user data to protect.  Would you impose a DVD-sized bucket system on ALL your users for ALL their data?

If the answer to (1) or (2) is "no", how would you implement a backup regime to mirror only certain parts of the system?  Would you mirror all the user data in that case?  Or use an incremental backup?  Or what?

If the answer to (1) or (2) is "no", why not?

(Can I just check: by incremental I mean that the backup software only actually backs up what has changed since the last backup; I presume it means the same to you?)

Mike