Today we continue discussing the best practice from storage engineering:

656) Different containerization technologies have proprietary format and semantics for the storage application to use. Of these the most critical aspect of the storage server has been networking because the nodes often scale out and the nodes need to be declared to the containers for proper connectivity. NAS storage is not going away as long as hard drives continue to be nearline storage.

657) These same containerization technologies allow the storage services to be deployed as images. This is very helpful to run the same image across a variety of hosts without having to recompile and build the package.

658) Such an image needs to be built with the help of declared dependencies for the storage service often called custom resources in the containerization parlance and a logic to reconcile the state of the deployment with the intended declarations. This process is often aided by the sdk from the containerization technology.

659) The sdk from containers can also help with the scaffolding code generation for use with the storage service to build and deploy the service as an image. This code is called an operator

660) The operator used to build and deploy the storage server as an image for the container is usually one per product. The flavors of the product and its releases can be detailed in the definitions for the same operator.

Today we continue discussing the best practice from storage engineering:

649) If the entries vary widely affecting the overall results at a high rate, it is easier to take on the changes on the compute side but allow the storage of the listings to be progressive. This way tasks can communicate the changes to their respective sortings to the scheduler which can then adjust the overall sort order

650) If the listing is a stream, processing on a stream works the same was as cursor on a database adjusting the rankings gathered so far for each and every entry as they are encountered.

651) The processing of stream is facilitated with compute packages from Microsoft and Apache for example. These  kind of packages highlight the stream processing techniques that can be applied to stream from a variety of storage.

652) About the query and the algorithm be it mining or machine learning can be externalized. This can work effectively across storage just as much as it is applicable to specific data.

653) The algorithms vary widely in their duration and convergence even for the same data. There is usually no specific rule to follow when comparing algorithms in the same category

654) The usual technique in the above case is to use a strategy pattern that interchanges algorithms and evaluated them on trial and error basis.

655) Storage services can take advantage of containerization just like other applications. Provisioning the service over the containers while allowing the nodes to remain part of the cluster is both scale-able and portable.

Today we continue discussing the best practice from storage engineering

648) If the listing is distributed, it helps to have a map-reduce on the listing

649) If the entries vary widely affecting the overall results at a high rate, it is easier to take on the changes on the compute side but allow the storage of the listings to be progressive. This way tasks can communicate the changes to their respective sortings to the scheduler which can then adjust the overall sort order

650) If the listing is a stream, processing on a stream works the same was as cursor on a database adjusting the rankings gathered so far for each and every entry as they are encountered.

651) The processing of stream is facilitated with compute packages from Microsoft and Apache for example. These  kind of packages highlight the stream processing techniques that can be applied to stream from a variety of storage.

652) About the query and the algorithm be it mining or machine learning can be externalized. This can work effectively across storage just as much as it is applicable to specific data.

653) The algorithms vary widely in their duration and convergence even for the same data. There is usually no specific rule to follow when comparing algorithms in the same category

654) The usual technique in the above case is to use a strategy pattern that interchanges algorithms and evaluated them on trial and error basis.

Today we continue discussing the best practice from storage engineering:

639) If there are multiple registrations that need to be kept in sync, they get harder to maintain. It is easier if the lists can be combined or there is a one to one mapping between the lists

640) Failed tasks may require new tasks to be added in which case, it is better to find the failed tasks as separate from the otherwise new tasks.

641) When the tasks are constantly replenished, it is helpful to keep track of in versus out.

642) The tasks that are out are candidates for cleanup.

643) The tasks that are in are either existing or new. They are mutually exclusive so it is easy to tell the new ones from the old.

644) The tasks that are new will need things setup for them to execute. It involves initialization so that they can be included in the list

645) The tasks that run long need to indicate progress in some way so that the scheduler knows that this task is still active and not stuck.

646) When the tasks have to sort the results, the sorting order might change as the listing changes. It is helpful to refresh the listing before sorting.

647) If the listing is large, it is not easy to refresh without taking a cost on the overall query time. In such cases, it helps to have progressive listing. Where the changes are made to one one end of the listing while the other ends remains as is.  As the listings are added to the tail, the stats from unchanged can be reused for the new entries.

Today we continue discussing the best practice from storage engineering :

633) The state of an object is authoritative. If it weren’t the source of truth, the entries itself cannot be relied on without involving validation logic across entries. There is no probllem performing validations but doing them over and over again not only introduces delays but can be avoided altogether with clean state.

634) The states are also representative and unique. The entries are not supposed to be in two or more states at once. It is true that bitmask can be used to denote conjunctive status but a forward only discrete singular state is preferable.

635) The attributes in an entry are often added on a case by case basis since it is expedient to add a new attribute without affecting others. However, the accessors of the entry should not proliferate the attributes. If the normalization of the attribute can serve more than one accessor, it will provide consistency across accesses.

636) Background tasks may be run or canceled. Frequently these tasks need to be canceled. If they don’t do proper cleanup, they can leave their results in bad state. The shutdown helps release the resources properly

637) The list of background tasks may need to include and exclude the tasks as they appear or disappear. This is in addition to start and stop on each task. If the start and registration are combined, the stop and deregistration must also be combined.

638) As tasks appear and disappear, it is sometimes too tedious to perform all the chores for each task. In such cases, we merely difference the new tasks and add them to the list. This prevents the cleanup on each job as they are left. A large-scale global shutdown may suffice later.

639) If there are multiple registrations that need to be kept in sync, they get harder to maintain. It is easier if the lists can be combined or there is a one to one mapping between the lists

640) Failed tasks may require new tasks to be added in which case, it is better to find the failed tasks as separate from the otherwise new tasks.

Today we continue discussing the best practice from storage engineering:

631) Listing entry values are particularly interesting. In addition to the type of attributes in an entry, we can take advantage of the range of values that these attributes can take. For example, we can reserve boundary values and extremely tiny values that will not be encountered in the real world at least for the majority of cases.

632) When the values describe the size of an associated object, the size itself can be arbitrary and it is not always possible to rule out a size for a user object no matter how unlikely it seems. However, when used together with other attributes such as status, they become usable as representative of some object state that is otherwise not easily ffound.

633) The state of an object is authoritative. If it weren’t the source of truth, the entries itself cannot be relied on without involving validation logic across entries. There is no probllem performing validations but doing them over and over again not only introduces delays but can be avoided altogether with clean state.

634) The states are also representative are also unique. The entries are not supposed to be in two or more states at once. It is true that bitmask can be used to denote conjunctive status but a forward only discrete singular state is preferable.

635) The attributes in an entry are often added on a case by case basis since it is expedient to add a new attribute without affecting others. However, the accessors of the entry should not proliferate the attributes. If the normalization of the attribute can serve more than one accessor, it will provide consistency across accesses.

636) Background tasks may be run or canceled. Frequently these tasks need to be canceled. If they don’t do proper cleanup, they can leave their results in bad state. The shutdown helps release the resources properly

Today we continue discussing the best practice from storage engineering:

626) Listings are great to use when they are in a single location. However, they are often scoped to a parent container. If the parent containers are distributed, the listings tend to be multiple. In such cases the effort is repeated.

627) When the listings are separated by locations, the results from the search may be fewer than the expected total if only one of the locations is searched. This has often been encountered in deployments.

628) The listings do not need to be aggregated across locations in all cases. Sometimes, only the location is relevant and the listing and the search can be scoped to it.

629) Iterating the listings has proved banal in most cases both for system and for user. Consequently, either an identifier is used to go directly to the entry in the listing or a listing is reserved so that only that listing is accessed.

630) The listing can be cleaned up as well. There is no need to keep it growing with outdated entries and then archived by age. The cleaning can happen in the background so that list iterations skip over entries or do not see the entries that appear as removed.

631) Listing entry values are particularly interesting. In addition to the type of attributes in an entry, we can take advantage of the range of values that these attributes can take. For example, we can reserve boundary values and extremely tiny values that will not be encountered in the real world at least for the majority of cases.