Issue link: https://digital.copcomm.com/i/860302
www.postmagazine.com 29 POST AUGUST 2017 and the Denial of Service attack that is Framestore's on-premise and cloud-based rendering," he says. So "accommodating scale, production fluctuations and hostile performance requirements across many petabytes of high-performance storage required a bit of thought." After a period of analysis and discovery, a formal storage architecture was proposed and adopted. Over the past 18 months the technology team at Framestore has phased it in with great success, MacPherson reports. Utilizing enterprise-grade commodity hardware from the likes of Dell and NetApp/LSI, Framestore runs proprietary software designed in conjunction with its systems, systems development and pipe- line teams to deliver a high degree of performance predictability and availability. Using a combination of Serial Attached SCSI (SAS) and Near Line SAS (NLSAS) disks running on the ZFS file system, each cluster is configured in High Availability (HA) failover pairs. The studio is experimenting with different operating systems for the head; Solaris is the current default, although Framestore is also exploring Linux and FreeBSD as a foundation for next-generation file severs. "Framestore maintains a high commitment to soft- ware development and is currently deep within the NFS code looking to further extend server function- ality," MacPherson says. Framestore partitions its storage into three primary categories: Hot, Cold and Frozen. The Hot zone is based primarily on fast SAS/SSD and is ded- icated to active production where "multiple artists, multiple departments and multiple sites can work on the same shot simultaneously sharing work and progressing through that work, together or sepa- rately, maintaining both a high degree of visibility and creative autonomy," he explains. Framestore's in-house tools perform scene assembly and versioning for the artists and work in concert with the underlying hardware systems. "This makes it easy for artists to locate, load and begin work on the shots/sequences they are assigned," he says. "The Hot, Cold and Frozen components that lie beneath provide a common set of methods for the technical and production departments to discuss provisioning and deployment of the storage infrastructure. Artists may or may not be interested in the mechanics of how this is architected, the key point being that a deep understanding of how our storage is delivered is no longer mandatory to com- municate requirements and predictions." Essential to Framestore's system is "a high level of redundancy within each individual Hot server zone — power supplies, controllers, RAID striping, et cetera," he notes. "In order to protect against cata- strophic zone failure, each zone exists in a matched set where the alternative can assume duties should an individual zone server become unavailable. "In the event of storage array failure, we also maintain a spare zone that can be repopulated from cold storage," MacPherson says. "The critical factor here is TTR (Time To Recovery) and the time it takes to repopulate. We have two general catego- ries of a Hot data — 'keep,' which is data where we keep two copies at all times and 'churn,' which is data where the effort required to backup and main- tain that data exceeds the effort to recreate it from source materials." Cold storage is simply high-capacity disks de- signed to maintain a second copy of all 'keep' data. As data is removed from Hot, a second copy is cre- ated and kept in Frozen storage. Upon completion, the show is cleaned by pipeline rulesets, archived to LTO tape and ultimately stored in a physically secure environment. Framestore's storage architecture enables the company "to wrest ourselves from letting others determine our choices, from the limitations and cost associated with a specific vendor's approach," says MacPherson. "This is in no way intended to be critical of vendors: There are many outstanding solutions that solve well-understood problems, and we have deployed them with great success in the past. What we saw was an opportunity to build something that could scale easily in accordance and tightly coupled to the way our productions work. By being soft- ware-defined, we are also ready to leverage new storage technologies as they become available." It's necessary for Framestore to "accommodate the requirements of multiple, large studio projects without disruption," he notes. "As we scale both our internal data centers and also our ability to exploit cloud-based options, we realized that investing in R&D liberated us from the general purpose software requirements of the industry. Instead of implement- ing solutions that partially address key fundamen- tals, we could deliver a set of services closely aligned to the business and production require- ments of Framestore." Framestore's UK film division (bottom, right) reports that assets for such projects as Marvel's Guardians of the of the of Galaxy Vol. Galaxy Vol. Galaxy 2 (above) and Disney's Beauty and Beauty and Beauty the and the and Beast (right) require huge amounts of storage.