Issue link: https://digital.copcomm.com/i/1505509
ile and object storage are both common technologies for per- sistently storing and organizing digital information. While files and objects have many similarities, there are some nota- ble differences. A principal difference is that the con- tent of a file can be changed at any time during its lifetime, while the contents of an object cannot be changed after it is initially created. An object can be replaced with an entirely new object with the same "key" by deleting and recreat- ing it, but parts of an object cannot be changed independently. This simplifica- tion facilitates improved scaling, reliabili- ty and data durability. There are also key differences in how files and objects are identified. A file is part of a file system with a hierarchi- cal naming structure, and files can be attached at any point in this naming hierarchy. Files can be renamed and moved within the naming hierarchy of a file system efficiently without copying data. Objects, on the other hand, are ref- erenced by a key, which doesn't change over the object's lifetime. Object keys can be created to mimic a file path and name, but changing this name still requires creating a new object with a new key and deleting the old object. Again, this sim- plification removes complexity when all that is required is a mechanism to reliably store and retrieve fixed blobs of data. These naming differences are connect- ed to differences in how files and objects are accessed. Applications access files using an operating system specific API on a computer (with operations like: open, seek, read, write, close) while ob- ject storage is accessed like a web server over a network (with operations like: put, get). While there are standards for both interaction mechanisms and standard protocols for accessing files over net- works, accessing objects over a network is a primary design consideration. Of course, most people interact with both files and objects through applica- tions, which can abstract a lot of these differences. Desktop applications have historically been written to work with files, and web applications are typically written to work with objects. There's nothing preventing a web application from working with files, but those files have to be made accessible via a web service. Further, there is nothing stop- ping a desktop application from working with objects, but appropriate API calls for interacting with web services need to be used. The challenges of working with media assets that live in object storage Given most people interact with storage through applications, application support is a central concern. Web applications tend to abstract how underlying data is stored. They often bundle storage as an integral part of the offering, and any external access to that data must be performed through the application. Collaboration applications like Frame.io and Media Silo, and document centric file sharing systems like Google Drive and Dropbox fall into this category. These applications provide no choice over how and where data is stored, which is an important consideration when working with large media files. Most desktop applications have no support for native interaction with object storage. While desktop applications typically offer a lot of flexibility regarding how and where files are stored, to inter- act with object storage without native support for object storage, desktop applications need some form of storage gateway. These gateways usually map files to objects in a manner that makes it difficult, if not impossible, to access the data in object storage any way other than through the gateway, again limiting choice of how and where data is stored. Media objects that are written once and read many times are ideally suited to object storage. When media objects are edited, the editing process usually just re- cords editing decisions without changing the underlying media. New media is gen- erated through a flattening or rendering process that applies the edit decisions or rendering rules. Different characteristics of storage and the applications that in- teract with them make them more or less applicable at different stages in the media life cycle. Combining the accessibility of a web application with the functionality of desktop application and the freedom to choose where and how media is stored provides significant benefits. File to object mapping determines how applications that straddle the file and object world translate object keys to file names and vice versa. Storage gateways typically use internal keys for objects and break files into multiple ob- jects tracked with an internal database. Web applications also often use internal keys and internal mechanisms to map applications data to objects. Most media tools, like editing tools, have no native ability to interact with object storage. Media stored as an object must first be copied into working file storage. This assumes the correct media can be found in the first place. Tools for searching object storage and previewing assets found in object storage are rare. If only a part of an identified asset is re- quired, the entire object must usually be copied into working file storage to trim the asset. This wastes time, bandwidth and working storage space. Signiant Media Engine facilitates user-friendly search and preview of media assets stored as objects and allows retrieval of a user-specified portion of the asset. Users can specify a frame accurate mark-in and mark-out point, and apply the trim before the asset is moved. This type of functionality helps bridge the gap between object storage and desktop applications, such as editing software. Performance issues and scalability While accessing object storage over a network is an integral part of the design, long-distance networks still present chal- lenges. The HTTP-based object storage access protocol is orders of magnitude better over long distances than protocols like NFS, used to access file systems over a network, but we can still do better. Different versions of the HTTP protocol have varying levels of efficiency when dealing with small objects. Interacting with large objects can be made more efficient by reading or writing the data in parts in parallel, but for maximum performance the right number of simultaneous parallel parts and part size must be determined. Further, HTTP operations are performed over TCP, which itself can suffer throughput issues on long-distance, high-bandwidth pipes. Signiant Intelligent Transport acceler- ates long distance access to storage by optimizing the choice of transport proto- col (Signiant UDP or TCP) and application FILE STORAGE VS. OBJECT STORAGE BY IAN HAMILTON CHIEF TECHNOLOGY OFFICER SIGNIANT WWW.SIGNIANT.COM WHAT ARE THE SIMILARITIES AND DIFFERENCES? F STORAGE www.postmagazine.com 30 POST JULY/AUG 2023