The design of appenders for Object Storage:

Object storage has established itself as a “standard storage” in the enterprise and cloud. As it brings many of the storage best practice to provide durability, scalability, availability and low cost to its users, it could expand its role from being a storage layer to one that facilitates data transfer. We focus on the use cases of object storage so that the onerous tasks of using object storage can be automated. Object storage then transforms from being a passive storage layer to one actively participating in data transfers to and from different data sinks and sources simply by attaching appenders. Earlier, we described the data transfer to the object storage with the help of connectors. The purpose of this document is to describe appenders that facilitate intelligent automations of data transfer to external data sinks with minimal or no disruption to their operations.  

File-systems have long been the destination to capture traffic and while file system has evolved to stretching over clusters and not just remote servers, it remains inadequate as a blob storage. The use of connectors and appenders enhance the use of object storage and do not compete with the usages of elastic file stores.  

We describe the appenders not by the data sink they serve but by the technology behind the appenders. We enumerate synchronous send and receive over protocols, asynchronous publisher-subscriber model, compressed data transfer, deduplicated data transfer and incremental rsync based backups to name a few.  We describe their advantages and disadvantages but do not provide a prescription to the retail world which allows them to be used on a case by case basis and with flexible customizations. In this sense, the appenders can be presented in the form of a library with the object storage so that commodity and proprietary data sinks may choose to receive data rather than actively poll for it. 

While public cloud object storage solutions offer cloud based services such as Amazon’s Simple Notification Services and Azure Notification hubs, on-premise object storage has the ability to make the leap from standalone storage offering to a veritable solution integration packaging. Public-Cloud offers robust myriad features for their general-purpose cloud services while the appenders specialize in the usages of the object storage and come as an out of box feature for the object storage. 

The data that flows out of object storage is often pulled from various receivers. That usage continues to be mainstream for the object storage. However, we add additional usages where the appenders push the data to different data sinks. Previously there were in-house scripts for such data transfers. Instead we make it part of the standard storage and provide users just the ability to configure it to their purpose.  

The ability to take the onerous routines of using object storage as a storage layer from the layer above across different data sinks enables a thinner upper layer and more convenience to the end user. The customizations in the upper layer are reduced and the value additions bubble up the stack.   

On-premise object storage is no more a standalone. The public cloud has moved towards embracing on-premise compute resources and their management via System Center integration. The same applies to on-premise object storage as well.  Consequently, the technologies behind the appenders are not only transparent but they are also setup for being monitored and reported via dashboards and charts This improves visibility into the data transfers while enabling cost accounting.   

An Object Storage offers better features and cost management, as it continues to stand out against most competitors in the unstructured storage. The appenders lower the costs of usage so that the total cost of ownership is also lowered making the object storage whole lot more profitable to the end users.