Wednesday, September 26, 2018

The design of Queue Layer over object storage  
Problem statement:
Object Storage is perceived as backup and tertiary storage. This may come from the interpretation that this storage is not suitable for read and write intensive data transfers that are generally handled by file-system or database. However, not all data needs to be written deep into the object storage at once. The requirements for object storage need not even change while the reads and writes from the applications can be handled with a background processor. There can be a middle layer as a proxy for a file system to the application while utilizing the object storage for persistence.  This alleviates performance considerations to read and write deep into the private cloud each time. That is how this Queue Layer positions itself. It offers the same performance as enterprise message queue does to handle the workload and while it may use its own intermediate storage, it works as a staging for the data so that the data has a chance to asynchronously dispatched into object storage.
Queue service has been a commercially viable offering. Message Queue is an example of a Queue service that has shown substantial improvements to APIs. Since objects are accessed via S3 Apis, the use of such Queue service works very well. However, traditional Queue services have usually maintained ordered delivery of messages, retries, dead letter handling, along with journaled messages and Queue writes and their writes have been mostly write-throughs which reach all the way to the disk. This service may be looked at in the form of a cloud service that not only maintains its persistence in the object storage but also uses the hierarchy for isolating its queues.
Queue Service works closely with a cluster based storage layer or a database server and have traditionally been long standing products in the marketplace. RabbitMQ is a message broker that relays messages to processors. This Queue layer is well-positioned for web application traffic as well as those that utilize S3 APIs directly. Data reads and writes need not be synchronous anymore and callers and clients may do well to delegate it to the queues as a background task. Moreover the object storage can leverage geographical replication of objects within its layer. As long as this queuing layer establishes sync between say a distributed or cluster file system and object storage with duplicity-tool like logic, it can roll over all data eventually to persistence.
#codingexercise
Void generateAlternateCubes() {
Var cubes = GetCubes ();
cubes.Enumerate( (I , e) => { if ( (i%2 == 0) { Console.writeline (e); }} );
}

No comments:

Post a Comment