Hi

It’s true you do have the freedom in the virtual layer to re-organize your images as you see fit as long as you stay within the constraints of the current bucket size.  

There are no constraints when I re-organise my images at the physical level with the virtual layer in place.  That's the point.  I would never have to reorganise at the virtual level, as such, because all I'd do is just run the script again to re-divide the images amongst the virtual buckets.  Re-organising to me is a gradual and careful transition from one state to another, with some sense of where you've just come from at each stage and also where you're going.  With virtual buckets, you just scrap the past entirely without looking back, and re-partition without any care in the world.

Also, I would never incrementally backup my images, that’s unnecessary and requires too much disk space.  I use only mirror jobs for my media.

I'm interested.  Please explain more.

The only thing I see needing to do is encapsulate my smaller buckets into larger one’s as we move to new write-once media; which again is something I can do just as easily at the physical layer as I can at the virtual.  

You're still left with a reminder of days gone by, though.  Presumably you still have all those little DVD-sized folders within the new, bigger, Bluray folders?  Either that or you empty the DVD buckets into the new buckets and mix them all up together in one great primordial soup.  And eventually when the mythical write-once media comes along that can store ALL your data, you just have one great big mass of files in a single directory (either that, or lots of little 4 gig folders).  I'd rather keep my images ordered meaningfully both at the physical level and the virtual level.

However, please feel free to implement this yourself and keep us updated on any progress or discoveries you make along the way.  

Progress so far:

1) Tested the Catalog Set scripting interface again in the latest build of MediaPro.
2) Still broken.
3) That's it.

Cheers,

Mike

Hi Dan, Al, Peter,

Dan wrote:

I'm actually a computer engineer and programmer studying to become a photographer.  One thing I've learned through all my years of programming is that unless creating a virtual layer actually offers you a degree of freedom that you wouldn't otherwise have via the physical layer, then the price you pay is usually performance and complexity due to an extra level of indirection.  That's really where my question still remains.  I can't say that I'm convinced at this point that an extra virtual layer is worth it.  All I see it really doing is creating a duplication of effort, once in the physical and again in the virtual; assuming that you are maintaining structure in your physical, which you say you are.

I would say that having the virtual layer does offer a significant degree of freedom over doing it at the physical layer instead.  It allows you the freedom to have whatever physical structure you like, unconstrained by whatever the current limit of technology is.

What's more, the process of splitting with virtual buckets is also completely reversible; in fact because it's virtual, there really is nothing to reverse.  Your physical structure remains intact, allowing you to move things around and re-organise in the future without any constraints.  With physical buckets you're stuck with what you decide at the time you decide to do it.  With virtual buckets you have the freedom to change.  

It's as if using physical buckets obeys the law of entropy, while using virtual buckets does not.  With physical buckets you divide everything up into little bits.  It's like dropping that cup on the floor and watching it shatter into pieces.  New items then go into different bits to the previous bits even if they are related to previous items.  With a 4 gig limit even a new batch of images will end up in different bits.

Re duplication of effort, and extra complexity.  There really is no effort in the physical layer.  It's all done by my import software.  I've decided on a convention and the import software does everything for me, putting the images in the right place with the right names.  There is effort in the virtual layer, but no more so than with physical buckets -- in fact perhaps even less so because I would hope to be able to script away a lot of the effort, for example by getting a script to find the most recent bucket and to place newly-imported items into that bucket up to the limit, creating a new bucket if it starts to overflow -- most of these sort of tasks would be automated.

Mike

What I have been mulling recently, would be a way to "lock" the Stars and iView Labels together, so that one always meant the other.  This way, I would have the convenience of Color-coding, and easier keyboard functionality, coupled with the added interface of stars for those times when that's the easier way to browse/scan images.

That would be simple to do as a separate, scripted, step (at least, when iView update their scripting engine).  This extra step would just align stars to labels, or vice versa, but it wouldn't of course happen real-time when you are annotating.

Mike

Hi Dan

You're right, there is nothing to prevent you from using virtual buckets in your cataloging software and just dumping all you images into a single folder on your hard drive.

But a key point of my argument in favour of using virtual buckets (my advantage (c) above) is that it allows you to retain whatever physical folder structure you like.  I wouldn't ever consider just dumping all my images into a single folder and subdividing only using virtual buckets.  My folders are divided according to Year and then Date of Shoot, as set out early on in this thread.  This structure means something to me, and it's easy to navigate.  All related images (not all images, just all related images) are in one place, not scattered around the place.

If you feel you will never have an archive that grows big enough to concern yourself with that, and as long as you aren't concerned with incremental backups of your entire archive, then by all means go ahead and use virtual buckets.

My feeling is that drive capacity and performance for the same money is increasing at least as fast as my image collection, so I'm unlikely ever to run into problems in this respect.  I expect I'll always be able to house my entire collection on a single drive.  And an incremental backup really doesn't take that long, and certainly no longer than 10 hours overnight.  It's more like about 2 minutes (maximum) at the moment for an incremental backup, and I can't see it going beyond 10 hours in my lifetime.

Be sure to let us know how it goes.

I will... all I have to do now is wait for iView to get their scripting engine sorted, because proper access to catalog sets via JS / VBS is currently broken.

I appreciate you sharing your views.  Many thanks.

Mike

Hi Dan

And yes, even if you could copy your virtual buckets to a physical folder before burning them, that still means you have to copy your buckets every time you want to burn...a major inconvenience; especially once we go to Blu Ray.  

I would be dragging and dropping from a MediaPro bucket set onto my DVD writing software.  This doesn't physically copy any files, it just transfers the paths of the media items into the DVD software.  Real quick.  It's a simple way of just letting your DVD software know what files to write, without having to have them all in the same physical folder.  Nice and easy... Ctrl-a in MediaPro to select all media items in the bucket set, drag and drop onto the DVD software window, press Go, and that's it.

Also, what about your backup software.  Current backup software, like Retrospect and GBM do not know anything about virtual buckets.

I wouldn't use my backup software to backup to DVD.  It wouldn't need to know about virtual buckets.  I would use my backup software to do incremental backups of my hard drive, e.g. overnight, to an external hard drive and/or an external ftp site I have set up.  It would by design only backup those images that have changed recently, wherever they might be in my physical data structure.  It doesn't need a bucket folder structure to tell it what has changed recently and what hasn't.

And you certainly don't want to be mirroring your entire physical archive.  I could just see Peter attempting to do that.  It would probably bring his whole operation to a crawl.   Ideally you only want to mirror the current bucket.  

Now put yourself in the position of someone other than yourself or Peter, like a keen amateur who doesn't really need to go to the extreme of mirroring certain parts of his/her disk to multiple drives, who can adequately protect his/her valuable images with incremental backups (and the occasional full backup) to multiple locations including to a set of DVDs.  (Peter's book is, after all, also aimed at such people.)  Would you still consider that a virtual bucket set system has no merits; such a person would find nothing in any of the advantages I gave previously?  I really don't understand your point of such a system making me more dependent on my cataloging software.  I'd really be no more dependent than I already am; I already make heavy use of catalog sets and would only realistically ever be able to leave MediaPro behind if I could take my catalog sets with me.  Besides, you can always script an exit strategy if there isn't one provided for you.

Mike

Hi Dan

If I'm understanding you correctly, it sounds like you are using virtual buckets (i.e. virtual sets) within your cataloging software to emulate Peter's physical bucket infrastructure.  

Well, yes and no.  I'm not actually doing this, but just exploring this as an option.  But yes, that's what I'm thinking.

You are then burning your DVD's directly from your cataloged virtual sets/buckets.

Yes.

If I'm correct here, on the surface this might seem more flexible because you're not limited by a physical infrastructure.  However, the problem I see with this is it ties you completely to your cataloging software to maintain your infrastructure as well as to burn your archive.  That's something I would be fearful of doing.  If you ever migrated to a new catalog that couldn't import your virtual bucket sets from your old catalog, it could be ground zero again.

That is a point, although MediaPro does enable you to sync catalog sets to the file, so in theory it should be possible to pick up those sets in the new software and use them in an analogous way with whatever interface it provides.  But yes, I can see that having your buckets independent of the software is more robust.  Especially since at the moment I think catalog sets are only synced to certain types of file (e.g. NOT to NEFs)... but I would hope naively that this will improve; iView seem to do the best they can to make it easy to get data out in as many ways as possible.

But then again, Peter advocates liberal use of Catalog Sets in his book to add value to your catalog by creating image groupings, and you would have exactly the same issues in respect of these when migrating to new cataloging software.  I bet you'd think twice about jumping ship to new software if you knew you'd lose all those.

Also, I wouldn't be reliant on the cataloging software to do the burning; I'd hope there is burning software out there that can accept dragged-and-dropped files from my MediaPro catalog sets to tell it what files to burn and where they are, with full hierarchy maintained.  I haven't looked into this.  Perhaps my current software can, I'm not sure.  But it would seem an obvious interface to have.  

Personally I don't really see any advantage to maintaining your bucketed infrastructure virtually because you still have to keep track of the bucket size and burn the DVD's just the same.

I can see advantages in:

(a) setting them up in the first place (just run a script);

(b) migrating them to a new media size (just run the script again with a larger number plugged in);

(c) allowing you to maintain a physical folder structure that means something to you, keeping related files together physically (although maybe in different virtual buckets), thereby allowing you easily to navigate to find files related to a particular shoot without relying on your cataloging software to get there; and

(d) that you're not tied to the limitations of a rapidly aging media technology.  I mean, 4 Gigs is really quite small nowadays, and the thought of splitting up my entire hard drive into tiny 4 Gig chunks doesn't appeal, especially knowing that I'll have to re-group everything in a couple of years' time for the new media format.  Doing it virtually -- well that's different.  To me it would be like sticking to eight-character filenames just to maintain compatibility with an old technology (OK that analogy is pushing it a little, I know).


Having said that, I really can see there are advantages in having your files split into physical folders that correspond exactly to the DVDs.  So I'm torn.

Thanks,

Mike

"Sometime later, you make a derivative file and put it back into that same bucket, making it larger than 4.4 Gigs."

But in my system I'm not putting new derivatives back into the old buckets and overflowing them.  I'm putting new derivatives into a new bucket.. but they're virtual buckets... my physical data structure would be completely free of bucket size and naming limitations... so the new file can actually live alongside the others from the original shoot and yet also live in a new bucket... a virtual catalog set bucket.  The new and old are in separate virtual buckets, and yet are still together on the physical drive.

Mike

Thanks for the tips, Peter.  I will re-read the sections of your book on this subject and review my thoughts afterwards.  In the meantime, you wrote:


But the later derivative will be in a new and quite separate catalog set bucket to the originals, and I will know that I only have to backup what's in the new catalog set bucket, not the old one.  The new and old may be mixed on the physical drive, but the catalog set "virtual" bucket structure allows you to pick out the new from the old, and only backup the new ones to DVD.  There's no need to touch the older DVDs because the contents of their respective buckets has not changed.  Maybe I'm missing something relating to another part of the backup process... I'll go back to the DAM book again

Many thanks for sharing your thoughts,

Mike

If it's a choice between Peter and myself over who is being dense, I'd go for myself.   I didn't write a book on the subject

I'm not trying to replicate a folder structure with catalog sets.  The point is that you can separate the folder structure (which can be anything you like) from the bucket structure (which would be named as you describe in your book).

For example, I have my images organised like:

..\My Pictures\YYYY\YYYY-MM-DD Short Description of Shoot

e.g. :

..\My Pictures\2006\2006-01-28 Hayden's Birthday Party

Now, I quite like having my images kept in their little family groups, for various reasons.  I know you advocate something different, but I was just wondering whether I could take some of your teaching (buckets) and keep some of my quirky feeling of reassurance by having images organised by shoot rather scattered amongst various buckets... by using catalog sets to create virtual buckets over whatever folder structure you may have.  I'm not a pro generating millions of images a year, so perhaps I am able to be a little more quirky 

So I'd run an automated script to create and fill catalog set buckets, say "DVD 001" to "DVD 427".  Then I'd create and fill further ones as I required.  If I create a new derivative of a master image back in DVD 002, then I'd just stick the derivative in the most recent catalog set bucket, DVD 428 or whatever (although the physical derivative file would be kept with the rest of its happy family in the same physical folder).  Then when a new storage medium comes along, I'd run the automated script again and create new catalog set buckets appropriate for the new media size, and away we go again.  Nice and simple.

Now, am I being dense by even thinking along these lines?  There are bound to be flaws... which is why I started this thread... to bring these points out for discussion.

Mike

P.S. I seem to have started a parallel thread going on iView's forum: http://www.iview-multimedia.com/forum/viewtopic.php?t=3300

Peter:

As I understand it, with your system you create physical folders on your drive for the buckets, and fill them up with images until they reach the limit, and then you start a new bucket.  Each new image goes into the most current bucket, and there is no danger of an image "missing" a bucket... it has to be stored somewhere and that somewhere is in the most recent bucket.

I was wondering how you could use MediaPro's Catalog Sets to create "virtual buckets", so that your actual folder structure can be anything you like, and your buckets are tracked with Catalog Sets; like virtual folders used as buckets.  When you move up to larger capacity storage in the future, you just divide your media amongst a new set of virtual buckets of the new size.

Robert: thanks for the link to the script.  It looks like it will be useful to set up the virtual buckets (when it works under 3!).

I was really pondering how you fit this into a practical, ongoing, workflow... because it seems to rely on you making sure that every new file that is imported into MediaPro finds its way into a bucket --- it also relies on you actually cataloging every new file in MediaPro --- otherwise files will be missed. 

I currently use folder Auto Update in MediaPro, and so the danger is that images are imported silently into MediaPro and I might well miss the manual step of assigning them to the most current bucket. 

I suppose folder watching without auto update would be better, so that you do a manual update, select the newly-imported images, and drag them into the most current bucket.  Even still, it doesn't seem completely foolproof.  Also, can you tell the total media size of a Catalog Set?  How do you know when it has reached its limit?

I guess these sort of things can be scripted (including checking that each media item is assigned to one -- and only one -- bucket), but before I even think about doing this I was just wondering whether anyone here had tried this sort of thing and ruled it out because it's impractical.  Any experiences / thoughts?

Thanks,

Mike

Is anyone here implementing Peter's bucket system with MediaPro's Catalog Sets instead of physical folders?  If so, would you mind sharing your experiences?  Any hints as to how to get this working in a practical and resilient way that ensures that none of your media items kinda miss the buckets entirely and end up on the floor?

Thanks,

Mike