We were discussing the use case of stream storage with message broker. We continue the discussion today.



A message broker can roll over all data eventually to persistence, so the choice of storage does not hamper the core functionality of the message broker.



A stream storage can be hosted on any Tier2 storage whether it is files or blobs. The choice of tier 2 storage does not hamper the functionality of the stream store. In fact, the append only unbounded data nature of messages in the queue is exactly what makes the stream store more appealing to these message brokers

A message broker is supposed to assign the messages If it sends it to the same single point of contention, it is not very useful When message brokers are used with cache, the performance generally improves over what might have been incurred in going to the backend to store and retrieve That is why different processors behind a message broker could maintain their own cache.  A dedicated cache service like AppFabric may also be sufficient to distribute incoming messages. In this case, we are consolidating processor caches with a dedicated cache. This does not necessarily mean a single point of contention. Shared cache may offer at par service level agreement as an individual cache for messages. Since a message broker will not see a performance degradation when sending to a processor or a shared dedicated cache, it works in both these cases. Replacing a shared dedicated cache with a shared dedicated storage such as  a stream store is therefore also a practical option.

We were discussing the use case of stream storage with message broker. We continue the discussion today.

A message broker can roll over all data eventually to persistence, so the choice of storage does not hamper the core functionality of the message broker.

A stream storage can be hosted on any Tier2 storage whether it is files or blobs. The choice of tier 2 storage does not hamper the functionality of the stream store. In fact, the append only unbounded data nature of messages in the queue is exactly what makes the stream store more appealing to these message brokers

Chained message brokers
This answers the question that if the mb is a separate instance, can message broker be chained across object storage. Along the lines of the previous question, if the current message broker does not resolve the queue for a message located in its queues, is it possible to distribute the query to another message broker. These kinds of questions imply that the resolver merely needs to forward the queries that it cannot answer to a default pre-registered outbound destination. In a chained message broker, the queries can make sense simply out of the naming convention and say if a request belongs to it or not. If it does not, it simply forwards it to another message broker. This is somewhat different from the original notion that the address is something opaque to the user and does not have any interpretable part that can determine the site to which the object belongs.  The linked message broker does not even need to take time to local instance is indeed the correct recipient. It can merely translate the address to know if it belongs to it with the help of a registry. This shallow lookup means a request can be forwarded faster to another linked message broker and ultimately to where it may be guaranteed to be found. The Linked message broker has no criteria for the message broker to be similar and as long as the forwarding logic is enabled, any implementation can exist in each of the message broker for translation, lookup and return. This could have been completely mitigated if the opaque addresses were hashes and the destination message broker was determined based on a hash table.  Whether we use routing tables or a static hash table, the networking over the chained message brokers can be its own layer facilitating routing of messages to correct message broker. 

We were discussing the use case of stream storage with message broker. We continue the discussion today.


A message broker can roll over all data eventually to persistence, so the choice of storage does not hamper the core functionality of the message broker.


A stream storage can be hosted on any Tier2 storage whether it is files or blobs. The choice of tier 2 storage does not hamper the functionality of the stream store. In fact, the append only unbounded data nature of messages in the queue is exactly what makes the stream store more appealing to these message brokers



The message broker is more than a publisher subscriber. It is the potential for a platform for meaningful insight into events.

The rules of a message broker need not be AMQP endpoint based address lookup. We are dealing with messages that are intended for multiple recipients. For that matter instead of addresses, regex and rich expressions may be allowed. and internally the same message may be relayed to different queues at lightning speed from queue specific initiation. We are not just putting the message broker on steroids, we are also making it smarter by allowing the user to customize the rules. These rules can be authored in the form of expressions and statements much like a program with lots of if then conditions ordered by their execution sequence. The message broker works more than a gateway server. It is a lookup of objects without sacrificing performance and without restrictions to the organization of objects within or distributed stores. It works much like routers and although we have referred to message broker as a networking layer over storage, it provides a query execution service as well.

Why gateway service cannot be implemented like this
For (Zone zone : ZoneScanner.getZones ()) {
    If (zone.equals ( currentZone)) {
    }
}
Imagine if this were to be performed at every data-path it would be repeated and costly.

We were discussing the use case of stream storage with message broker. We continue the discussion today.

A message broker can roll over all data eventually to persistence, so the choice of storage does not hamper the core functionality of the message broker.

A stream storage can be hosted on any Tier2 storage whether it is files or blobs. The choice of tier 2 storage does not hamper the functionality of the stream store. In fact, the append only unbounded data nature of messages in the queue is exactly what makes the stream store more appealing to these message brokers

The message broker is more than a publisher subscriber. It is the potential for a platform for meaningful insight into events.  

Message broker is also a networking layer that enables a P2P network of different local storage which could be on-premise or in the cloud. A distributed hash table in this case determines the store to go to. The location information for the data objects is deterministic as the peers are chosen with identifiers corresponding to the data object's unique key. Content therefore goes to specified locations that makes subsequent requests easier. Unstructured P2P is composed of peers joining based on some rules and usually without any knowledge of the topology. In this case the query is broadcast and peers that have matching content return the data to the originating peer. This is useful for highly replicated items. P2P provides a good base for large scale data sharing. Some of the desirable features of P2P networks include selection of peers, redundant storage, efficient location, hierarchical namespaces, authentication as well as anonymity of users.  In terms of performance, the P2P has desirable properties such as efficient routing, self-organizing, massively scalable and robust in deployments, fault tolerance, load balancing and explicit notions of locality.  Perhaps the biggest takeaway is that the P2P is an overlay network with no restriction on size and there are two classes structured and unstructured. Structured P2P means that the network topology is tightly controlled and the content is placed on random peers and at specified location which will make subsequent requests more efficient. 

Public cloud technologies can be used to improve our products and their operations even if they are deployed on-premise. Public clouds have become popular for their programmability which endear them to developers while providing unparalleled benefits such as no maintenance, lower total cost of ownership, global reachability and availability for all on-premise instances, unified and seamless operations monitoring via system center tools and improved performance via almost limitless scalability. 

The public cloud acts like a global sponge to collect data across on-premise deployments that serve to give insights into the operations of those deployments. Take the use case of collecting metrics such as total size and object count from any customer deployment. Metrics like these are lightweight and have traditionally been passed into special purpose on-premise storage stacks which again pose the challenge that most on-premise deployments are dark for monitoring remotely. On the other hand, the proposal  here is that if these choice metrics were to fly up to the public clouds like pixie dusts from each of the customer deployments, then they can improve visibility into the operations of all customer deployments while expanding the horizon of monitoring and call home services. A service hosted on the public cloud becomes better suited to assuming the role of a network operations center to receive callback-home alerts, messages, events and metrics. This is made possible with the rich development framework available in the public cloud which allows rapid development of capabilities in the area of operations monitoring, product support and remote assistance to customers for all deployments of software products and appliances. It also lower costs for these activities. Metrics is just one example of this initiative to move such developmental work to the public cloud. All services pertaining to the operations of customer deployments of our products are candidates for being built and operated as tenants of the public cloud.  

The proposals of this document include the following: 1) improving visibility of on-premise appliance deployments to services in the public cloud, 2) Using the public cloud features to improve manageability of all deployed instances, 3) Using a consolidated reporting framework via services built in the cloud that map all the deployed instances for their drill down analytics and 4) using the public cloud as a traffic aggregator for all capabilities added to the on-premise storage now and in the future. 

Specifically, these points were explained in the context of knowing the size and count of objects from each deployed instance. Previously, there was no way of pulling this information from customer deployments by default. Instead the suggestion was that we leverage the on-premise instance to publish these metrics to a service hosted in the public cloud. This service seemed to be available already in both the major public clouds. The presentation demonstrated Application Programming Interface aka API calls made against each public cloud to put and get the total size and count of objects. 

These API calls were then shown as introduction to several powerful features available from the public cloud for value additions to the manageability of deployed instances. The latter part of the presentation focused on the involvement of public cloud for the monitoring and operations of geographically distributed on-premise customer deployments.  

As a mention of a new capabilities made possible with the integration of on-premise deployment with manageability using public cloud, it was shown that when this information is collected via the API calls made, they become easy to report with drill downs on groups of deployments or aggregation of individual deployments. 

Finally, the public cloud was suggested to be used as a traffic aggregator aka sponge for subscribing to any light-weight data not just the size and count described in this presentation and thus become useful in alleviating publishing-subscribing semantics from on-premise deployments. Together, these proposals are a few of the integration possibilities of on-premise deployments with capabilities hosted in the public cloud.

#codingexercise

Determine the max size of a resource from allresources


Int getMaxResourceLength(List<String > input) {
return input.stream().map(x -> getSize(x))
                          .max(Comparator.comparing(Integer::valueOf))
                          .get();

}

Both message broker and stream store are increasingly becoming cloud technologies rather than on-premise. This enables their integration much more natural as the limits for storage, networking and compute are somewhat relaxed.

This also helps with increasingly maintenance free deployments of their components. The cloud technologies are suited for load balancing, scaling out, ingress control, pod health and proxies that not only adjust to demand but also come with all the benefits of infrastructure best practices

The networking stacks in the host computers have maintained a host centric send and receive functionality for over three decades. Along with the services for file transfers, remote invocations, peer-to-peer networking and packet capture, the networking in the host has supported sweeping changes such as web programming, enterprise computing, IoT, social media applications and cloud computing. The standard established for this networking across vendors was the Open Systems Interconnection model. The seven layers of this model strived to encompass all but the application logic. However, the networking needs from these emerging trends never fully made its feedback to the host networking stack. This article presents the case where a message broker inherently belongs to a layer in the networking stack. Message Brokers have also been deployed as a standalone application on the host supporting Advanced Message Queuing Protocol. Some message brokers have demonstrated extreme performance that has facilitated the traffic on social media.

The message broker supports an observer pattern that allows interested observers to listen from outside the host. When the data unit is no longer packets but messages, it facilitates data transfers in units that are more readable. The packet capture used proxies as a man in the middle which provides little or no disruption to ongoing traffic while facilitating the storage and analysis of packets for the future. Message subscription makes it easier for collection and verification across hosts.

The notion of data capture now changes to the following model:

Network Data subscriber

Message Broker

Stream storage

Tier 2 storage



This facilitates immense analytics directly over the time series database. Separation of analysis stacks implies that the charts and graphs can now be written with any product and on any host.

This way a message broker becomes a networking citizen in the cloud.

Both message broker and stream store are increasingly becoming cloud technologies rather than on-premise. This enables their integration much more natural as the limits for storage, networking and compute are somewhat relaxed.
This also helps with increasingly maintenance free deployments of their components. The cloud technologies are suited for load balancing, scaling out, ingress control, pod health and proxies that not only adjust to demand but also come with all the benefits of infrastructure best practices
The networking stacks in the host computers have maintained a host centric send and receive functionality for over three decades. Along with the services for file transfers, remote invocations, peer-to-peer networking and packet capture, the networking in the host has supported sweeping changes such as web programming, enterprise computing, IoT, social media applications and cloud computing. The standard established for this networking across vendors was the Open Systems Interconnection model. The seven layers of this model strived to encompass all but the application logic. However, the networking needs from these emerging trends never fully made its feedback to the host networking stack. This article presents the case where a message broker inherently belongs to a layer in the networking stack. Message Brokers have also been deployed as a standalone application on the host supporting Advanced Message Queuing Protocol. Some message brokers have demonstrated extreme performance that has facilitated the traffic on social media.
The message broker supports an observer pattern that allows interested observers to listen from outside the host. When the data unit is no longer packets but messages, it facilitates data transfers in units that are more readable. The packet capture used proxies as a man in the middle which provides little or no disruption to ongoing traffic while facilitating the storage and analysis of packets for the future. Message subscription makes it easier for collection and verification across hosts.
The notion of data capture now changes to the following model:
Network Data subscriber
Message Broker
Stream storage
Tier 2 storage

This facilitates immense analytics directly over the time series database. Separation of analysis stacks implies that the charts and graphs can now be written with any product and on any host.
This way a message broker becomes a networking citizen in the cloud.