Media Storage for major motion pictures‏

[jwballard]

I am trying to find information about how movie makers store their work in process.  The questions I would like to answer include:

What storage and backup systems do movie producers use?
How many petabytes do movie companies store for one movie, such as AVATAR?
Which storage companies are leaders in technology for movie-making digital storage?

Does anyone know these answers?  Does anyone have a suggestion about where I could look for answers?

John Ballard

[peterkrogh]

John,
Could you give a clue as to the context you are looking for?  Are you trying to accomplish this yourself, create a service, write a white paper, do an academic or school project?

>such as AVATAR?

I think there's probably only *one* movie with the storage requirements "such as Avatar", at the moment.
Peter

[jwballard]

Peter,
My interest in storage systems for movies is part of a personal interest in storage tech.  After we saw Avatar, my 16-yr-old tech-geek asked me how much storage was used in producing the movie. (500 terabytes? 5 petabytes?)  That is why I used Avatar as an example.
I would imagine that the makers of Avatar are at the leading edge of digital media storage technology, so I want to learn more about what they are doing. 
A few years back I bought a terabyte of NAS for personal use and was the "first on my block" to use something like that.  A few years in the future, photographers who shoot 30 megapixel images and tons of video will need more sophisticated systems than we use today.  I'd like to know where we are headed.
John

[ianw]

John,

According to this link found on Wikipedia - http://www.datacenterknowledge.com/archives/2009/12/22/the-data-crunching-powerhouse-behind-avatar/ - it required 17.28GB of storage per minute of the final film.  At 166 minutes long that makes it approaching 3TB if my math is right.  Somehow they'll squeeze that onto a 8.5GB dual-layer DVD or a 50GB blu-ray disk, neither of which, I'm sure, will do it justice.

Some of the other facts and figures for the data centre that handled the processing of the film are impressive: 40,000 processors with 104TB of memory.  That's some serious system.  I hope their backup strategy was sound!

Ian

[jwballard]

Ian,
Thank you for this link.  I have just started to read the available info, but the connections to WETA Digital, NetApp and BlueArc are just what I was looking for.  I know Net App, but not the other two firms.  I'm sure I can learn a lot through these links.

What is your interest in digital storage?

John

[markpirozzi]

...According to this link found on Wikipedia - http://www.datacenterknowledge.com/archives/2009/12/22/the-data-crunching-powerhouse-behind-avatar/ - it required 17.28GB of storage per minute of the final film.  At 166 minutes long that makes it approaching 3TB if my math is right.  Somehow they'll squeeze that onto a 8.5GB dual-layer DVD or a 50GB blu-ray disk, neither of which, I'm sure, will do it justice...

Ian

I wonder if all this storage per minute is based on the whole project and not just what takes made it to release. Plus the movie was filmed for IMax 3D and seemed to be a higher resolution than the average film.   Whatever the case, I'm sure they'll just downsample to fit home media - should look as good as any other movie released for home use.

Mark

[rogerhoward]

Mark - those figures are what the final, edited, full-resolution digital master requires - the actual storage requirements for the production assets on a project like this could easily be 100x or more, even with a great deal of temp data (draft renders, dailies, etc) not kept at all. A production like Avatar certainly generated data in the petabytes - how much of that is archived I don't know, but they'll certainly be keeping a lot more than just the final digital master.

Even within a single studio, the infrastructure and pipeline to produce a film like this will be almost entirely one-off - by the time the next comparable production comes online they'll be generations apart in terms of architecture and even vendors in some cases. There's also a huge range of storage requirements on a production, but just consider the storage for uncompressed, 4K film scans:

12 bits per channel @ 4k resolution @ 24 fps = 3.2TB per hour, uncompressed

[ianw]

Avatar would have needed petabytes of storage to hold everything required.  Of course a lot of this would have been duplication of data across different sites - you can't back this up to DVD or Blu-Ray!  Even a tape backup would be difficult due to the sheer amount of data.

I remember when Monsters Inc came out they boasted that they had animated a million hairs on the character Sully.  The storage and computational power required just for these hairs would have been incredible.  That was in 2001, which is a lifetime ago in computing terms.

At work we are upgrading our storage to EMC DMX4.  I've just looked at the specs for this ( http://uk.emc.com/collateral/hardware/specification-sheet/c1166-dmx4-ss.pdf ) and the racks can take between 96 and 1920 physical hard drives and we've got the largest.  The actual storage depends on the size of drives you install and which RAID setting you take.  I know we have gone for the smaller drives, I think the 146GB ones, spinning at 15,000 rpm.  More small drives gives a better throughput of data than fewer larger drives.  I think we have RAID 6.  Of course you don't just get one storage rack - we have 3.  There's a primary and a secondary, in different locations naturally, with the third rack being used to do any backups from.  Therefore some one has signed off on a very large purchase order, 3 racks and over 6,000 hard drives!  The application I'm responsible for only needs 1TB for our production system and 2TB for our development and test areas.  I don't know who's using the rest of it, just a shame I can't use it to back up my images on!

Ian

[jwballard]

Roger & Ian,
It sounds like you know much more about this topic than I do. 
I was guessing that the total storage needs of a movie like Avatar would be in petabytes.  I'm trying to track down someone with specific knowledge about this  at the Avatar production company or at WETA digital, the New Zealand special effects house that Ian's link led me to.  A WETA story mentions that:
"The computing core - 34 racks, each with four chassis of 32 machines each - adds up to some 40,000 processors and 104 terabytes of RAM. The blades read and write against 3 petabytes of fast fiber channel disk network area storage from BluArc and NetApp."
This give a sense of the scale of storage needed. 
I have no idea what hardware and configuration the Avatar producers used to store all the production assets.  I'll keep looking.
John Ballard

[peterkrogh]

Interesting discussion.
As to how much storage they *need*, defining need will be the rub. As Roger points out, there are certainly temporary renders, outtakes, proof of concept copies and more that might have value in the future. I suspect that every scrap of Star Wars footage, notes, outtakes, floorsweepings and the like has commercial value. The corresponding material from Ishtar does not.

I suspect that there is simply no way to know how much data is generated by such a project, because it will continue to grow. Unlike filmed footage, CGI generated from 3-d models can be reworked in lots of different ways.  I'd be surprised if it's not growing by terabytes daily (on average).

One thing we know - Avatar will be able to generate enough funds so that Cameron can afford to keep whatever he wants.

Peter

[jwballard]

I sent an inquiry to the publicity dept of WETA digital, the New Zealand special effects house about the equipment and configuration they use and the amount of storage used for their work on Avatar.  Here is the, not surprising, response:
We use high performance fibre channel disk Network Attached Storage (NAS) from Netapp and BlueArc as our primary storage. For lower
performance / lower cost storage, we use SATA disk NAS from Netapp.  To manage our intermediate and final images, we use a collection of
proprietary tools that are tightly coupled to our motion picture pipeline.

Apologies we can not be of further assistance due to the confidential nature of our work.
  (My emphasis added.)

When I sent followup question, it wasn't answered (for two days so far).

I guess WETA is a dead end.

John