Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

638. As tasks appear and disappear, it is sometimes too tedious to perform all the chores for each task. In such cases, we merely difference the new tasks and add them to the list. This prevents the cleanup on each job as they are left. A large-scale global shutdown may suffice later. 

639. 
     

639. If there are multiple registrations that need to be kept in sync, they get harder to maintain. It is easier if the lists can be combined or there is a one-to-one mapping between the lists 

640. 
     

640. Failed tasks may require new tasks to be added in which case, it is better to find the failed tasks as separate from the otherwise new tasks. 

641. 
     

641. When the tasks are constantly replenished, it is helpful to keep track of in versus out. 

642. The tasks that are out are candidates for cleanup. 

643. 
     

643. The tasks that are in are either existing or new. They are mutually exclusive so it is easy to tell the new ones from the old ones. 

644. 
     

644. The tasks that are new will need things set up for them to execute. It involves initialization so that they can be included in the list 

Network engineering continued ...

This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

633) The state of an object is authoritative. If it weren’t the source of truth, the entries themselves cannot be relied on without involving validation logic across entries. There is no problem performing validations but doing them over and over again not only introduces delays but can be avoided altogether with a clean state.

634) The states are also representative and unique. The entries are not supposed to be in two or more states at once. It is true that bitmask can be used to denote conjunctive status but a forward only discrete singular state is preferable.

635) The attributes in an entry are often added on a case by case basis since it is expedient to add a new attribute without affecting others. However, the accessors of the entry should not proliferate the attributes. If the normalization of the attribute can serve more than one accessor, it will provide consistency across accesses.

636) Background tasks may be run or canceled. Frequently these tasks need to be canceled. If they don’t do the proper cleanup, they can leave their results in a bad state. The shutdown helps release the resources properly

637) The list of background tasks may need to include and exclude the tasks as they appear or disappear. This is, in addition, to start and stop on each task. If the start and registration are combined, the stop and deregistration must also be combined.

Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

626. Listings are great to use when they are in a single location. They are often scoped to a parent container. If the parent containers are distributed, the listings tend to be multiple. In such cases, the effort is repeated. 

627. 
     

627. When the listings are separated by locations, the results from the search may be fewer than the expected total if only one of the locations is searched. This has often been encountered in deployments. 

628. The listings do not need to be aggregated across locations in all cases. Sometimes, only the location is relevant, and the listing and the search can be scoped to it. 

629. 
     

629. Iterating the listings has proved banal in most cases both for the system and for the user. Consequently, either an identifier is used to go directly to the entry in the listing, or a listing is reserved so that only that listing is accessed. 

630. 
     

630. The listing can be cleaned up as well. There is no need to keep it growing with outdated entries and then archived by age. The cleaning can happen in the background so that list iterations skip over entries or do not see the entries that appear as removed. 

631. 
     

631. Listing entry values are particularly interesting. In addition to the type of attributes in an entry, we can take advantage of the range of values that these attributes can take. For example, we can reserve boundary values and extremely tiny values that will not be encountered in the real world at least for most cases.  

Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

618. If the listing is common to several consumers, a key may be used for each consumer for their specific needs. If this approach does not scale, then the listing may be retrieved in ranges and the filtering may be taken over on the compute side. 

619. 
     

619. The keys will need to have backward compatibility otherwise they break existing scenarios. 

620. When a new key is added, it may not impact existing keys, but it does affect the overall space consumption of the listing depending on the size and number. 

621. 
     

621. The keys can have as many fields as necessary. If this becomes slow, the lookups can be faster when there are only a few keys to compare.  

622. 
     

622. Key comparison can be partial or full. Partial keys are useful to match duplicates. The number of keys that share the same subkeys can be many. This form of comparison is very helpful to group entries. 

623. Grouping of entries also helps with entries that span groups based on subkeys. These work across groups 

Network engineering continued ...

  This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html 

612. The results from the background tasks mentioned above might also take a long time to accumulate. They can be made available as they appear or batched. 

613. 
     

613. The load balancer works very well to enable background tasks to catch up by not overloading a single task and distributing the online activities to ensure that the background task has light load 

614. 
     

614. The number of background tasks or their type should not affect online activities. However, systems have known to be impacted when the tasks are consuming memory or delay garbage collection 

615. 
     

615. If one or more background tasks takes plenty of shared resources, they can be canceled and they are written to be fault tolerant so that they can pick up from where they left off. 

616. 
     

616. The use of lookup keys is common to find entries in listings. The key generally comprises of a prefix and one or more values pertaining to related entities.  

617. 
     

617. The type and number of keys can change even in the same listing. They can also be used as markers 

618. If the listing is common to several consumers, a key may be used for each consumer for their specific needs. If this approach does not scale,  the listing may be retrieved in ranges and the filtering may be taken over on the compute side. 

Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html 

606. Networking, compute and storage are inter-dependent. We use data structures to keep the information we want to access in a convenient form. When this is persisted, it mitigates faults in the processing. Artifact brings in additional chores and maintenance while it is cheaper to execute the logic. Versioning of logic helps track the changes. When there is a trade-off between computing and storage for numerous small and cheap artifacts, it is better to generate them dynamically.

607.  

607. The above has a far-reaching impact when there are layers involved and the cost incurred in the lower layer bubbles up to the top layer. 

608. 
     

608. Compute tends to be distributed in nature while storage tends to be local. They can be mutually exclusive in this regard. 

609. 
     

609. Compute-oriented processing can scale up or out while storage must scale-out.

610.  

610. Compute oriented processing can assign priority dynamically but storage tends to remain in a class  

611. 
     

611. Background tasks may sometimes need to catch up with the current activities. In order to accommodate the delay, they may either be run upfront so that changes to be processed are incremental or they can increase in number to divide up the work.