Cluster computing

Tuesday, December 8, 2020

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

There are very few cases where the system or the process goes wrong. On the other hand, there is a lot of advantage to trusting the process. It is better not to try to beat the system but to make it work for you.

As with any product, a networking product also qualifies for the Specific-measureable-attainable-realistic-timely aka SMART process where improvements can be measured and the feedback used to improve the process and the product.
Every process change comes with an onus and is probably the least favored by employees but the rewards generally outweigh the costs when it is reasoned and accepted by all. The six-sigma process for example sets a high bar for quality because it eliminates errors progressively.

The iterations for six-sigma were high so it takes a long time and the results are not always available in the interim. The agile development processes allowed results to be incremental.

The agile methodology improved the iterations over the features in such a way that it did not impact the rest of the product. This enables faster feature development

The continuous integration and continuous deployment model made the individual feature improvements available for use because the changes were build, tested and deployed in lock step with development.

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

There are several formats of metrics and monitoring data and generally, they are proprietary. Utilizing an external stack for these purposes via APIs helps alleviate the concerns from the storage service.

Virtually every service utilized from the infrastructure is a candidate for standardization and consistency so that one component/vendor in the infrastructure may be replaced with another with little or no disruption

There are several stack frames that a developer must traverse in order to find the code path taken by the execution thread and they don’t always pertain to layers but if the stack frames can get simpler for the developer, the storage product overall improves tremendously. This is not a rule but just a rule of thumb that the simpler the better.

As with all one-point maintenance code, there is bloating and complexity to handle different use cases from the same code. Unfortunately, developers don’t have the luxury to rewrite core components without a significant investment of time and effort. Therefore version 1 of the product must always strive for building it right from the get-go.

As use cases increase and the business improves, the product management pays a lot of attention to sustainable growth in the face of business needs. It is at this cusp of technology and business that the system architecture plays its best.

There are notions of SaaS, PaaS and, IaaS with clear separation of concerns in the cloud. The same applies to any solution deployed to these.

The organization of services in the cloud does not limit the number and type of solutions hosted in the cloud. The same holds true for the feature as services within a multi-tenant hybrid cloud with proper isolation.

The benefits that come with the cloud can also come from a networking layer and software defined stack. If an application uses cloud only to aggregate traffic, it could consider moving to on-premise with a fully managed networking layer.

There are times when the networking layer will have imbalanced load. They will need to be load balanced. Since this is an ongoing activity, it can be periodically scheduled or responded when thresholds are crossed. A load balancer does that automatically.

When the layers of infrastructure and networking services are clearly differentiated, the upper layer may utilize the alerting from the lower layers for health checks and to take corrective actions.

There are several ways to monitor networking whether it is for performance, statistics or health checks. A system center management system can consolidate and unify the operations management. The networking layer merely needs to publish to a system center.

Networking products embrace compute as much as the services that rely on networking but the line of separation between compute and networking layer remains clear in solutions that use networking. The purer the compute over the storage, the better for the storage.
For example, an application may use peer to peer networking or cluster mode with consensus messaging protocol but it tends to become simpler if the networking layer handles these.

A networking product may use a cluster for scalability or a peer to peer networking for distribution and offer this as a service so that the applications do not have to handle these.
The dependency of networking layer on a quorum of healthy nodes in an ensemble of a cluster is met with the help of detection and auto-remedial measures. If the administrator does not have to rush to replace a bad unit, it saves time and cost.

There are notions of SaaS, PaaS, and, IaaS with clear separation of concerns in the cloud. The same applies to networking applications in terms of how it is deployed.

The organization in the cloud does not limit the number and type of services available from the cloud. The same holds true for the feature as services within a networking product.

There are times when the networking product will have an imbalanced load. They will need to be load balanced. Since this is an ongoing activity, it can be periodically scheduled or responded to when thresholds are crossed.

When the layers of infrastructure and networking services are clearly differentiated, the upper layer may utilize the alerting from the lower layers for health checks and to take corrective actions.

1. The rules are scoped to the artifacts they secure. For system-wide resources, there is only a singleton. For user resources, they can be dynamically fetched and executed if they are registered.
1. This export of logic is very helpful in overcoming the limitations of static-configuration and reload of service. Regardless of the need for a runtime to execute the logic, even listenable config values can help with changes to rules.
1. The type of rules and the classes of outcome generally don’t change even in the most heavily used filters. IPSec, for example, has a lot of attributes to secure the network but its type of rules and outcomes are well-known. Rules can therefore be rewritten periodically to make them more efficient.
1. Networking products must accumulate user artifacts such as rules, containers, and settings. It should be easy to migrate and upgrade them.
1. The migration mentioned above is preferable to be done via a user-friendly mechanism because they matter more to the user than to the system.