Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

632. 638. The tasks that run long need to indicate progress in some way so that the scheduler knows that this task is still active and not stuck. 

          646. When the tasks must sort the results, the sorting order might change as the listing changes. It is helpful to refresh the listing before sorting. 

          647. 
               

          647. If the listing is large, it is not easy to refresh without taking a cost on the overall query time. In such cases, it helps to have a progressive listing. Where the changes are made to one end of the listing while the other end remains as is.  As the listings are added to the tail, the stats from unchanged can be reused for the new entries. 

          648. 
               

          648. If the listing is distributed, it helps to have a map-reduce on the listing 

          649. 
               

          649. If the entries vary widely affecting the overall results at a high rate, it is easier to take on the changes on the compute side but allow the storage of the listings to be progressive. This way tasks can communicate the changes to their respective sortings to the scheduler which can then adjust the overall sort order 

          650. 
               

          650. If the listing is a stream, processing on a stream works the same way as a cursor on a database adjusting the rankings gathered so far for each entry as they are encountered. 

     639. 
          

     640. 
          

     641. 
          

     642. 
          

Network engineering continued ...

    This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

632. 638. As tasks appear and disappear, it is sometimes too tedious to perform all the chores for each task. In such cases, we merely difference the new tasks and add them to the list. This prevents the cleanup on each job as they are left. A large-scale global shutdown may suffice later. 

     639. 
          

     639. If there are multiple registrations that need to be kept in sync, they get harder to maintain. It is easier if the lists can be combined or there is a one-to-one mapping between the lists 

     640. 
          

     640. Failed tasks may require new tasks to be added in which case, it is better to find the failed tasks as separate from the otherwise new tasks. 

     641. 
          

     641. When the tasks are constantly replenished, it is helpful to keep track of in versus out. 

     642. The tasks that are out are candidates for cleanup. 

     643. 
          

     643. The tasks that are in are either existing or new. They are mutually exclusive so it is easy to tell the new ones from the old ones. 

     644. 
          

     644. The tasks that are new will need things set up for them to execute. It involves initialization so that they can be included in the list 

Network engineering continued ...

   This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

632. 
     

633. The state of an object is authoritative. If it weren’t the source of truth, the entries themselves cannot be relied on without involving validation logic across entries. There is no problem performing validations but doing them repeatedly not only introduces delays but can be avoided altogether with a clean slate.

634.  

634. The states are also representative and unique. The entries are not supposed to be in two or more states at once. It is true that bitmask can be used to denote conjunctive status, but a forward only discrete singular state is preferable. 

635. 
     

635. The attributes in an entry are often added on a case-by-case basis since it is expedient to add a new attribute without affecting others. However, the accessors of the entry should not proliferate the attributes. If the normalization of the attribute can serve more than one accessor, it will provide consistency across accesses. 

636. 
     

636. Background tasks may be run or canceled. Frequently these tasks need to be canceled. If they don’t do the proper cleanup, they can leave their results in a bad state. The shutdown helps release the resources properly 

637. 
     

637. The list of background tasks may need to include and exclude the tasks as they appear or disappear. This is, in addition, to start and stop on each task. If the start and registration are combined, the stop and deregistration must also be combined. 

Network engineering continued ...

  This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

645. 628. The listings do not need to be aggregated across locations in all cases. Sometimes, only the location is relevant and the listing and the search can be scoped to it. 

     629. 
          

     629. Iterating the listings has proved banal in most cases both for system and for user. Consequently, either an identifier is used to go directly to the entry in the listing or a listing is reserved so that only that listing is accessed. 

     630. 
          

     630. The listing can be cleaned up as well. There is no need to keep it growing with outdated entries and then archived by age. The cleaning can happen in the background so that list iterations skip over entries or do not see the entries that appear as removed. 

     631. 
          

     631. Listing entry values are particularly interesting. In addition to the type of attributes in an entry, we can take advantage of the range of values that these attributes can take. For example, we can reserve boundary values and outliers that will not be encountered in the real world at least for most cases.  

     632. 
          

     632. When the values describe the size of an associated object, the size itself can be arbitrary and it is not always possible to rule out a size for a user object no matter how unlikely it seems. However, when used together with other attributes such as status, they become usable as representative of some object state that is otherwise not easily found. 

Network engineering continued ...

 This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html

645. The tasks that run long need to indicate progress in some way so that the scheduler knows that this task is still active and not stuck. 

646. 
     

646. When the tasks must sort the results, the sorting order might change as the listing changes. It is helpful to refresh the listing before sorting. 

647. 
     

647. If the listing is large, it is not easy to refresh without taking a cost on the overall query time. In such cases, it helps to have progressive listing where the changes are made to one end of the listing while the other end remains as is.  As the listings are added to the tail, the stats from unchanged can be reused for the new entries. 

648. 
     

648. If the listing is distributed, it helps to have a map-reduce on the listing.

649. 
     

649. If the entries vary widely affecting the overall results at a high rate, it is easier to take on the changes on the compute side but allow the storage of the listings to be progressive. This way tasks can communicate the changes to their respective sortings to the scheduler which can then adjust the overall sort order 

650. If the listing is a stream, processing on a stream works the same was as cursor on a database adjusting the rankings gathered so far for each entry as they are encountered.