Network engineering continued ...

  This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html 

Memory configuration: In a cluster environment, most of the nodes are commodity. Typically, they have a reasonable memory. However, the data that can be processed by a node depends on fitting the corresponding data structures in memory. The larger the memory, the higher the capability of the server component in the control path. Therefore, there must be some choice of memory versus capability in the overall topology of the server so that it can improve the overall network.  

Cpu Configuration: Typically, VMs added as nodes to cluster come in T-shirt size configurations with the number of CPUs and the memory configuration defined for each T-shirt size. There is no restriction for the networking service to be deployed in a container or a single Virtual Machine. And since the virtualization of the computing makes it harder to tell the scale-up of the host, the general rule of thumb has been more the better. This does not need to be so and a certain configuration may provide the best advantage. Again, the choice and the recommendation must be appropriate to the demand. 

Serverless Computing: Many functions of a networking server/product are written in the form of microservices or perhaps as components within layers if they are not separable. However, the notion of modularity can be taken in the form of serverless computing so that the lease on named compute servers does not affect the functionality of the networking server. 

Expansion: Although some configurations may be optimal, the networking server must be flexible to what the end-user wants as configuration for the network. Availability of acceleration, chimney offloads, and external additions to the hardware can all be used to expand the capabilities of the network. 

Upgrade scenario: As with any other product, a networking server also has similar concerns for changes to data structures or requests/responses. While it is important for each feature to be backward compatible, it is also important to have a design that can introduce flexibility without encumbrance. 

Network Engineering Continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html 

Contract:  Service-oriented architecture framework of providing web services defined contract and behavior in addition to address and binding for services but the general shift in the industry has been towards RESTful services from that architecture. This paradigm introduces well-known verbs for operations permitted. Networking products that provide RESTful services must conform to the well-defined mapping of verbs to create-update-delete operations on their resources. 

Workload profiles: Every network engineering product will have data I/O and one of the ways to try out the product is to use a set of workload profiles with varying patterns of data access.  These traffic generators can be T-shirt sized into small, medium and large.

Intra/Inter: Since data I/O crosses multiple layers, a lower layer may perform operations that are similarly applicable to artifacts in another layer at a higher scope. For example, replication may be between copies of objects within a single PUT operation and may also be equally applicable to objects spanning sites designated for content replication.  This not only emphasizes reusability but also provides a way to check the architecture for consistency. 

Local/Remote: While many components within the storage server take the disk operations to be local there are certain components that gather information across disks and components directly writing to it. In such case, even if the disk is local, it would prove consistent to access local via a loopback and simplify the logic to assuming every such operation as remote. 

Resource consumption: We referred to performance engineering for improving the data operations. However, the number of resources used per request was not called out because it may have been perfectly acceptable if the elapsed time was within bounds. However, resource conservation has a lot to do with reducing interactions which in turn leads to efficiency. 

Network engineering continued ...

95. This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

    
    

    Reconfiguration: Most networking products are subject to some pools of available resources managed by some policies that can change from time to time. Whenever the network resources are changed, they must be done in one operation so that the network presents a consistent view for all usages going forward. Such a network-wide reconfiguration is supported across network servers. 
     

96. Auto-tuning: This is the feedback loop cycle with which network server/appliance/product can be made to perform better because the dynamic parameters are adjusted to values that better suit the workload. 
     

97. Acceptance: This is the user-defined level of service-level agreement for the APIs to the storage server so that they maintain satisfactory performance with the advantage that the clients can now communicate with a pre-existing contract. 
     

98.  Address: This defines how network service is discovered by clients. For example, if there were a network local address or a bridged mode bidirectional reachability, this would define how the host / service would be reached. If there were a network share on the host, this would define who can access it. Enabling DMZ is a way to prevent reachability in unwanted cases. Since it can be virtual some network products may provide a gateway to those addresses. 
     

99. Binding: A binding protocol defines the transport protocol, encoding, and security requirements before the data transfer can be initiated. Although network products may default to tcp binding for internal networks and http binding for external by default, this could determine stateful or stateless connections. 

Network engineering continued

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

Tracers: The network usage by a given workload may not always be known. In such cases, it is better for the  networking server to inject markers or tracers to view the data path. 

User versus system boundary: Many security vulnerabilities manifest themselves when the code gets executed with user context rather than with system context. The execution of code in system context is privileged and maintains a few assumptions including one that it is the source of truth. Therefore, the switching from user to system context is required wherever we can demarcate the boundary. If the context switching is missing then it is likely that the code can be executed with user context.

Lines of control – even when the code path for admission into the system has a clear user and system context defined, the user context is established only when the execution traverses the lines of authentication and authorization Consequently all user facing entry points need to guarantee proper exception handling to minimize security risks from the line of control

Impersonation – Usually identities are not switched by the system because most of the system code is executed with its own identity. However, there are cases when code is executed in user context in which case a system thread may need to use the security context of the user. Impersonation opens up a new dimension for tests when identities are switched between two user accounts and is generally best avoided.

Routines at the user-system boundary- When the boundary between user and system is clearly demarcated and secured, it facilitates the execution of common routines such as auditing, logging, exception handling and translations, resetting contexts and so on. In fact, the user-system context boundary is a convenient way to enforce security as well as collect data on the traffic.

Data Types: There are some universally popular data types such as integers, float, date and string that are easy to recognize even on the wire and most network facing api also see them as ways of packing bytes in a byte sequence. However, network facing apis could do with more data types such as files, Json, Xml and plain old objects improves encapsulation within a request. Log indexes for example ingest data without user interaction.  Most products work largely by inferring fields rather than types within the data because it gives them a way to allow users to search and analyze using tags that are familiar to them. If types could be recognized, looking up fields together as types only adds a little bit more logic on the server side while improving the convenience to the users

Network engineering continued...

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

 Application Configuration – Most networking products are best served with a static configuration that can determine the parameters of the system. Since product usages span a wide variety of deployments, most products offer at least a few parameters to tune the system. Configuration files have been used with applications and servers on the Unix flavor platform and networking products also make use of it. Files also make it easy for listeners to watch for changes. 

 Dynamic Configuration – Applications and services have not only used configuration based on static files but also used dynamic configuration which may be obtained from external configuration sources. Since the value is not updated by touching the production server, this finds appeal in cases where parameters need constant tuning and have to be done without involving the production server. 

Platform independent client library – Most frontends for networking products rely on some form of JavaScript for client-side scripting. However, JavaScript is also popular in server-side programming as a NodeJs server. While portals and application servers are portable when written in JavaScript, it applies equally to any client library or agent interaction for the cloud server 

External management tools – For object storage, S3 has been a primary API for control and data path to the storage server. Management tools that are cloud agnostic provide tremendous leverage for bulk automation and datacenter computing. Consequently, storage products must strive for conforming to these tools in ways that utilize already streamlined channels such as well-published APIs whenever possible. 

Statistics – We referred to statistics enabled counters earlier for the components of the storage server. This section merely refers to client-based statistics for the entire storage product whenever possible so that there can be differentiated tuning to workloads based on the data gathered from the server usage. 

Network engineering continued...

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

Cachepoints – Cachepoints are used with consistent hashing. Cachepoints are arranged along the circle depicting the key range and cache objects corresponding to the range. Virtual nodes can join and leave the network without impacting the operation of the ring. 

Stream/Batch/Sequential of packets: Networking products are notorious for queues. And the sliding window processing is not restricted to TCP. There are other ways to process queues which include stream, batch, and sequential processing. 

Strategies – Implementation of a certain network processing logic within a networking product may often have a customized implementation and maintained with the component as it improves from version to version. Very little effort is usually spent on externalizing the strategy across components to see what may belong to the shared category and potentially benefit the components. Even if there is only one strategy every used with that component, this technique allows other techniques to be tried out independent of the product usage. 

Plug and Play architecture – the notion of Plug'n'play may have had some roots in networking. Plugins work irrespective of the components and layers in a networking stack of the host. Yet the standardization of the interface such that it is applicable across implementations is often left pending for later. Instead, the up-front standardization of interfaces promotes the eco-system and adds convenience to the user. 

Interoperability – Most networking products work well when the clients are running on a supported flavor of an operating system. However, this consideration allows the product to expand its usage. Interoperability is not just a convenience for the end-user, it is a reduction in management cost as well.  

Network engineering continued ...

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

Topology: Most networks are being deployed as new networks and often with software-defined stacks. The layout that the user might choose has to remain flexible to growth and scale to their requirements. The topology that they choose to deploy their network, might keep the edge more compute-intensive as opposed to the core of the network. This facilitates admission control and scales to any number of resources within the network.

Virtual time: As the storage server virtualizes storage over heterogeneous media and expands elastically for demand, there is a need to co-ordinate activities across participating agents. In such cases, the only event sequence that can be established correctly is the one based on virtual time.  

Gossip protocol: In a distributed environment, the best way to detect failures and determine memberships is with the help of gossip protocol. When an existing node leaves the network, it may not respond to the gossip protocol so the neighbors become aware.  The neighbors update the membership changes and copy data asynchronously. 

Paxos Algorithm: Some systems utilize a state machine replication such as Paxos that combines transaction logging for consensus with write-ahead logging for data recovery. If the state machines are replicated, they are fully Byzantine tolerant. 

Consistent hashing – Data is partitioned and replicated using consistent hashing to achieve scale and availability. Consistency is facilitated by object versioning. Replicas are maintained during updates based on a quorum like technique.