By Andy Marken
While most of the attention at this year's FMS (Flash Memory Summit) in Santa Clara, CA focused on the enterprise, one of the most eye-opening sessions in the 3-day event was the M&E (Media & Entertainment) panel.
Tying the needs to solutions, we caught up with Larry O'Connor, founder of Other World Computing (OWC) who had spoken at an earlier consumer devices panel to discuss the changing needs of entertainment content production/post production.
Why the storage industry's "sudden" interest in movie/video?
O'Connor -- I wouldn't call it sudden. Mac systems have been the production tool of choice since they were first introduced; and according to industry experts we talk with, I would estimate that at least 90 percent of the production/post production is done with an Apple system. That's one of the reasons we keep adding to our add-in/add-on upgrade products as well as storage and cable products.
As Ms Catania pointed out during the M&E panel discussion, most new cameras today use SSD (solid state drives) to capture content because it provides relatively high capacity, low battery draw and is rugged. For obvious reasons, I'd like to see more camera manufacturers adopt the Blackmagic approach and use an industry-standard interface and then test/certify specific products for their cameras. That broadens the options for the cinematographer, lowers his/her storage costs and enables storage providers to add newer, better, higher-capacity products more quickly for today's creative artists.
Panelists Neil Smith, of LumaForge; Tony Cacciarelli, of AJA and Matthew Rehrer, of Harmonics; all agreed that storage requirements are growing so rapidly that physical, content management is becoming a major issue for producers and their teams.
What I found interesting is that SSD is the solution of choice for high-speed/hot data -- especially in the field where the production team has to work fast and not always under the most ideal conditions. However, hard drives are used for high capacity/low cost in the field and in the studio.
Another challenge that Ms. Catania pointed out was that organizations have to implement object-based storage for global content distribution.
Larry Jordan
Host of Digital Production BuZZ
I think Larry Jordan, panel moderator and host of Digital Production BuZZ, laid out the challenge best for the industry when he said SSD is the platform of choice IF!!! it becomes more affordable and supports larger storage sizes.
Is video creation/production moving too fast because we're just barely getting into 4K shows/movies and there's already discussion of 8K and beyond?
O'Connor – The industry pace is a challenge for the capture/production/distribution industry.
Equipment is reasonably priced but when the change is happening as rapidly as it is, it is difficult to justify, absorb and amortize the investment costs. Then too, there is the need to learn all the new features/nuances of the hardware/software/workflow, which is no small expense.
As video resolution and frame rates increase, camera image complexity increases and stereoscopic projects multiply. The storage capacity and bandwidth performance requirements become staggering.
A video engineer recently showed me that 16,000 x 8,000 pixel resolution, 24 bit/pixel, 300 fps raw video content could require 414 TB/hr. If the producer used four cameras to create data for a "free viewpoint" presentation, the raw data would be 1.66 PB for an hour of content.
Obviously, that is good news for storage solution producers that are dedicated to delivering high- performance/high- capacity solutions like OWC, but it places tremendous pressure on post video professionals.
Do you think the new less expensive and easier-to-use technologies are good or bad?
O'Connor -- The roadmap I just spelled out places more emphasis and value on professional content producers. It is true that low-cost 4K cameras are widely available and cloud-software makes it very affordable for more people to develop movie/video projects and post them on YouTube or over-the-air channels. As they say, competition is a good thing for the consumers who are becoming more and more discerning, but I also think it is a good thing for the producers. As more amateurs attempt their own productions, they personally gain appreciation for how amazing professional, high-quality productions really are.
I equate the dramatic growth in amateur video production to the old desktop pubishing products. People bought them and suddenly everyone was producing a newsletter/newspaper. Most were horrible and people quickly realized that there was a lot of creatively required in producing a paper people would enjoy reading.
That will happen with the DIY cinematographer because they don't want to do all of the painstaking detail work that produces a "wow" movie. They just want people to congratulate them for their work. They may shoot the project but they'll turn it over to the professional production/post person to do the heavy lifting.
For those professionals who are still trying to get the most of their last investment in the latest and greatest, the quality gap will become apparent very fast and they'll step up to the next generation.
Of course either way, I think there will be a greater appreciation of professional production/post production, so the work behind the scenes will gain the recognition it deserves.
How are storage considerations affecting a project – from capture, pre-production/post production, distribution?
O'Connor – Movie/video content is a vicious task master for people in the industry. It devours storage capacity at faster and faster rates. That's great for storage vendors, not so great for everyone else. Based on years of working with professionals, we always recommend carefully calculating the amount of storage needed and then err on the side of having more capacity ... just in case.
Honestly, that's not just to sell you more storage capacity either.
The work today doesn't just happen in a darkened room in front of a workstation.
We see more and more people doing pre-production on location, connected to an in-the-field system, so small, "portable" desktop or laptop storage is very important.
And as anyone who has used a camera for any time at all knows, nothing beats a good backup drive (or two).
Today's digital cameras use flash-based SD cards or SSDs, which can take the rugged environment/handling, but are more expensive per gigabyte. That's why external, portable hard drives that offer high capacity but are inexpensive are very popular.
The faster cards and drives can be used multiple times after the insurance copies are transferred to multiple larger-capacity, cheaper storage solutions. You never know what will happen on a shoot and it's often impossible to recreate that magic shot.
As a rule of thumb, our customers tell us that content acquisition consumes about 20 percent of their storage usage. Production/post production accounts for about 40 percent, distribution uses about 10 percent and then archiving takes up the remaining 30 percent or so.
That adds up quickly, so for any project, it's easy for storage needs to balloon. A 4K hour-long movie is four times more vivid than an HD video, but it often ends up consuming 10-12TB of storage.
Are there challenges you see in video content storage?
O'Connor - The most devastating thing I consistently hear is someone saying they thought their video content was safe because they used an external drive or a redundant RAID set-up.
Drive storage today is extremely reliable and continues to improve (and is becoming cheaper), but problems will occur and when they do, you want to be 100% sure you're data is safe, because the alternative is expensive and career threatening.
Data is truly secure only when it is exists in multiple locations.
We tell people to think about it in two ways.
First, does the data exist in full (full is the key) on more than one drive or set of drives? If it doesn't, then any problem, theft or loss can be catastrophic.
Next, does the data exist in full in more than one physical location? If your two or three copies of the data are in the same building and that building burns down or is otherwise destroyed, then you're courting disaster.
Nothing provides 100 percent assurance, but multiple copies in multiple locations improve your odds. Those copies are cheap compared to trying to reshoot a scene or recreate an entire video project – if that is even possible!
How should cinematographers and production teams approach their storage needs?
O'Connor -- Individuals and teams really need to sit down and assess if there are points where all of the work they are doing has a single point of vulnerability.
I know people who have very sound setups that also make sure they have a copy of their content in another state or country and additional copies in safety deposit boxes.
The best place to start though is in choosing one thing, say raw data and the various trial and error FX work and thinking about where all of that content is located.
If that material is just in the computer because you're in the middle of the project and it's easier to work on everything when it is right there, then you need to take the first important step of adding an external drive. That includes implementing an automated back-up program or Time Machine solution so when content is added or new sequences are worked on they are immediately mirrored to your external storage device.
Once that is done and the process is as second nature as breathing, then you have to determine what added steps you want to take, such as copies in multiple locations, that you can live with and work with.
Keep in mind that there is a point where you aren't simply professional, you're paranoid. And as Andy Grove of Intel said, "Only the paranoid survive."
What challenges does OWC face in meeting the broad spectrum of production needs – one-person Indies, small teams, large organizations?
O'Connor - Everyone needs storage and lots of it.
What we see in the larger studios is a growing problem with data management and access. When one person needs access, that is easy; but as soon as you have multiple people accessing files (especially if they are in far-flung locations as is often the case in post work with specialists located across town or maybe even half-way around the globe), you have to start looking at a more sophisticated workflow management system like Harmonics, AJA or our recent offering.
That was one of the challenges we addressed with our Jupiter SAN. The central production team can grant any number of people access to the entire project or specific scenes – often at the same time – while still ensuring that each workstation or location has the capacity and speed they need to do their color correcting, editing, special effects and changes/enhancements.
At the same time, the project lead can ensure the individual's work isn't comingled with the complete project until it is fully reviewed/approved.
There is a lot of "controversy" on doing everything in the cloud everything local. Is there a balance for production people?
O'Connor - The cloud is a blessing to content project owners and to production/post production professionals, but it has also been oversold.
The cloud has given production/post production people the freedom to live/work where they want. If you're a seasoned professional in a particular phase of the workflow, you can be in Kauai; New Zealand; Croatia; Singapore; Austin, TX or Woodstock, IL. The Cloud (internet) puts your expertise, talents and tools in the center of the workflow.
Producers/project owners know they aren't limited to the people across the hall or across the street. They pick the best talent regardless of physical location. This provides cost savings and schedule benefits for the producers, while letting professionals live where they are happiest and have the best quality of life.
In other words, working through the cloud does offer a lot of benefits.
The downside comes more from the security and accessibility side. Internet connections are not 100 percent reliable and often not as fast as video projects demand. It is also still a new industry where security/privacy/content protection standards are a work in progress. Some have called Web work and storage the new Wild, Wild West.
Even when the content is fully encrypted, there is a chance that it can be hacked or stolen. However as long as you retain the original and only send work in progress segments, you can be relatively sure your work product will remain intact.
Would I send a complete or nearly complete project through the cloud? No.
Would I use the cloud to store my completed work as most cloud provider's advocate? Not as the primary, but the cloud is definitely a viable option as a secondary location.
I prefer local, physical storage (and in multiple locations) because it provides faster access and minimizes points of failure, loss, and corruption.
Speaking of local physical coverage what is new for cinema production people?
O'Connor – Products with the Thunderbolt interface and software RAID.
Thunderbolt is faster, better, more robust for video production/post production. It opens up the pipe for content throughput (typically 20GB/s) and that makes a huge difference in the speed and ease of doing movie projects.
In fact, we recently demonstrated connecting three ThunderBay 4 units to one of the new Mac Pros. It provided 36TB of capacity and benchmarked performance was clocked at nearly 4000MB/s. That translates into shorter production time and getting the content to the screen for viewing more rapidly.
In addition, we've reached the stage where software RAID can give greater application freedom and outperforms hardware RAID.
Back in the early days, MPEG-2 cards were widely used to handle the production process because the system processors simply didn't have the ability to handle the heavy workload. Today, we have tons of processing power to efficiently handle the software RAID with almost no effort.
You're not tied to a specific structure forever, you can move drives around and adjust your RAID from 0 to 5 to meet your changes in workload and workflow. In addition, something I think is great is that software RAID will constantly monitor the health of all of your drives and even send you an email when the health of a drive is "questionable."
The only way you can lose a video segment or frame is by ignoring the alert!
What is your main take-away from the panel session and FMS?
O'Connor – Because I've talked with a lot of cinema and videographers around the country, I've become increasingly aware of their storage needs. But, Ms. Catania gave me a new appreciation for how much capacity is required and why.
Cirina Catania
Founder, Lead Creative
The Catania Group, Inc.
She probably surprised everyone in the FMS audience when she spelled out the number of storage copies a film project produces:
- Original content (RAW) as shot
- Protection clone (never touched)
- Worker copy (files renamed, organized)
- Protection copy of worker
- Studio copy of worker
- 3rd clone of worker...just in case
- Output of the project
- Clone, 2nd protection
The result (which isn't surprising to the people who do this for a living) is that a project will end up consuming 10X the amount of storage as the original shoot.
No wonder storage capacity and storage management is such a big issue in the industry.
That's one of the reasons we spend so much time listening to customers so we can deliver a more positive/rewarding experience.
www.markencom.com
