Yesterday we were discussing the use case of stream storage with message broker. We continue the discussion today.

As compute, network and storage are overlapping to expand the possibilities in each frontier at cloud scale, message passing has become a ubiquitous functionality. While libraries like protocol buffers and solutions like RabbitMQ are becoming popular, Flows and their queues are finding universal recognition in storage systems. Messages are also time-stamped and can be treated as events.A stream store is best suited for a sequential event storage. 

Since Event storage overlays on Tier 2 storage on top of blocks, files, streams and blobs, it is already transferring data to those dedicated stores. The storage tier for the message broker with a stream storage system only brings in the storage engineering best practice. 

The programmability of streams has a special appeal for the message processors. Runtimes like Apache Flink already supports user jobs which have rich APIs to work with unbounded and bounded data sets. 

The message broker now has the ability to host a runtime that is even more performant and expressive than any of the others it supported.  

The design for message brokers has certain features that stream storage can handle very well. For example, the MSMQ has a design which includes support for dead letter and poison letter queues, support for retries by the queue processor, a  non-invasive approach which lets the clients send and forget, ability to create an audit log, support for transactional as well as non-transactional messages, support for distributed transactions, support for clusters instead of standalone single machine deployments and ability to improve concurrency control. Stream storage provides excellent support for queues, logs, audit, events, along with transactional semantics.  

This design of MSMQ is somewhat different from that of System.Messaging. The system.messaging library transparently exposes the underlying Message queuing windows APIs . For example, it provides GetPublicQueues method that enumerates the public message queues. It takes the message queue criteria as a parameter. This criterion can be specified with parameters such as category and label. It can also take machine name or cluster name, created and modified times as filter parameters. The GetPublicQueuesEnumerator is available to provide an enumerator to iterate over the results.  

Though these designs are different, the stream store provides guarantees for durability, performance and storing consistency models that work with both.