Friday, September 28, 2018

Object Storage is not inherent to a cluster. It does not participate in another cluster. Many applications utilize a cluster to scale. And they don’t interact with each other by any means other than as application endpoints or a shared volume.  A database, file-system or unstructured storage may be hosted elsewhere and then used with a cluster within an application. Consequently, the storage and the application form separate layers.  An application that utilizes a cluster specifically for messaging is a message queue server also called a message broker.  Message Queues facilitate an alternative for applications and services to send and receive data. Instead of directly calling the receiver by the sender, a message broker allows the sender to leave a message and proceed. This makes a microservice architecture become even more lean and focused while the messaging framework is now put in its own layer. Message queues also enable a number of processors to operate on a variety of messages while being resilient to errors in general. Since messages can grow to arbitrary size and their numbers can be mind boggling, the messages need to be saved where they can be retrieved and updated.  
Object Storage can store messages very well. The Queues are not nested and the hierarchy within object storage allows for grouping of these messages as easily as in a queue. Generally, a database is used for the storage of these messages but it is not for the transactional semantics. Even a file system could be sufficient for these messages. Object storage, on the other hand, is perceived as backup and tertiary storage otherwise. This may come from the interpretation that this storage is not suitable for read and write intensive data transfers that are generally handled by file-system or database. However, not all data needs to be written deep into the object storage at once. The requirements for object storage need not even change while the reads and writes from the applications can be handled with a background processor. There can be a middle layer as a proxy for a file system to the application while utilizing the object storage for persistence.  This alleviates performance considerations to read and write deep into the private cloud each time. Therefore, a Queue Layer may use the Object Storage with slight modifications. And it offers the same performance as it continued to provide. The Queues not only work as a staging for application data but also as something that asynchronously dispatches into object storage. 
Queue service has been a commercially viable offering and utilize a variety of protocols. Message Queue is an example of a Queue service that has shown substantial improvements to APIs. Since objects are also accessed via S3 web Apis, the use of such Queue service works very well if each message is stored and retrieved individuallyTraditional Queue services have usually maintained ordered delivery of messages, retries, dead letter handling, along with journaled messages and Queue writes and their writes have been mostly write-throughs which reach all the way to the disk. This service may be looked at in the form of a cloud service that not only maintains its persistence in the object storage but also uses the hierarchy for isolating its queues.

No comments:

Post a Comment