Today we continue discussing the best practice from storage engineering :

255) Update in place and historical queries real-time challenges. If the values of the updates are maintained with their chronological order, then the queries may simply respond with the values of recent past. Such a collection of queries with answers from the same point of time are compatible

256) Catalogs maintained in storage products tend to get large. A sufficient allocation may need to be estimated. This is not always easy to do with the back of the envelope calculation but allowing the catalog to grow elastically is a good design practice

257) Memory allocation is a very common practice in data path. The correct management of memory is required for programming as well as performance. A context based memory allocator is often used. It involves the following steps: A context is created with a given name or type. A chunk is allocated within a context The chunk of memory is deleted within a context after use. The context is then deleted and then reset. Alternatively, some systems are implemented in languages with universal runtime and garbage collection to utilize the built-in collection and finalization.

258) Memory contexts provide more control than garbage collectors. Developers can provide both spatial and temporal locality of deallocation. Garbage collectors work on all of a program’s memory but some languages offer ways to tune up the garbage collection for specific servers.

259) Overheads may be reduced by not requiring as many calls to kernel as there are user based requests. It is often better to consolidate them in bulk mode so that majority of the calls return in a shallow manner.

260) Administrators are often tasked with provisioning sufficient memory on all compute resources associated with the storage product. With t-shirt sized commodity virtual machines, this is only partially addressed because that only specifies the physical memory.  Virtual memory and it a usage must be made easier to query so that corrective measures may be taken.

Today we continue discussing the best practice from storage engineering:

250) The algorithm for load-balancing can even be adaptive based on choosing appropriate metrics to determine traffic patterns that are well-known.  We start with a single number to quantify load on each partition and each server and then use the product of request latency and request rate to represent loads.

251) Bitmap indexes are useful for columns with small number of values because they take up less space than B+ tree which requires a value and record pointer tuple for each record. Bitmap are also helpful for conjunctive filters.

252) B+ trees are helpful for fast insertion, delete and update of records. They are generally not as helpful to warehouses as Bitmaps

253) Bulk-load is a very common case in many storage products including data warehouses. They have to be an order of magnitude faster than individual insertions. Typically they will not incur the same overhead for every record and will take up the overhead upfront before the batch or stream into the storage.

254) Bulk Loads may not be as prevalent as when the storage product is already real-time. The only trouble with real-time products is that the read write is not separated from read only and they may contend for mutual exclusion. Moreover sets of queries may not see compatible answers.

255) Update in place and historical queries real-time challenges. If the values of the updates are maintained with their chronological order, then the queries may simply respond with the values of recent past. Such a collection of queries with answers from the same point of time are compatible

A use case for visibility of storage products: https://1drv.ms/w/s!Ashlm-Nw-wnWuDSAzBSGbG3Wy6aG 

Today we continue discussing the best practice from storage engineering:

245) The process pool per disk worker model has alleviated the need to fork processes and tear down and every process in the pool is capable of executing any of the read-writes from any of the clients. The process pool size is generally finite if not fixed. This has all of the advantages from the process per disk worker model above and with the possibility of differentiated processes in the pool and their quota.
246) When compute and storage are consolidated, they have to be treated as commodity and the scalability is achieved only with the help of scale-out. On the other hand, they are inherently different. Therefore, nodes dedicated to computation may be separated from nodes dedicated to storage. This lets them both scale and load balance independently.
247) Range-based partitioning /indexing is much more beneficial for sequential access such as with stream which makes enumeration easier and faster because of the locality of a set of ranges. This helps with performance. Hash based indexing is better when we have to fan out the processing in their own partitions for performance and all the hashes fall in the same bucket. This helps with load balancing.
248) Third, throttling or isolation is very useful when accounts are not well behaved.  The statistics is collected by the partition server which keeps track of request rates for accounts and partitions.  The same request rate may also be used for load balancing.
249) Automatic load balancing can now be built on range based partitioning approach and the account-based throttling.  This improves multi-tenancy in the environment as well as handling of peaks in traffic patterns.
250) The algorithm for load-balancing can even be adaptive based on choosing appropriate metrics to determine traffic patterns that are well-known.  We start with a single number to quantify load on each partition and each server and then use the product of request latency and request rate to represent loads.

Today we continue discussing the best practice from storage engineering:

240) Sparse data storage involves setting a large number of attributes as null. For a table, this means many of the columns will be empty. The solution to overcoming tremendous disk space waste is to re-organize the data in terms of columns of data tables rather than the rows. This column-oriented storage is very popular in massive data such as with Google’s BigTable, TaggedColumns used by Microsoft Active Directory, and the Resource Description Framework for Semantic Web.
241) Flash Memory is viable and supported in a broad market. It provides notable cost/performance trade-off relative to disk and RAM. Yet disks are not going away anytime soon. They may even show significant catch-up in terms of being intelligent with respect to power management and time-based scheduling of I/O. 
242) Clusters deal with nodes and disks as commodity making no differentiation in terms of capacity improved or nodes added. They are tolerant to nodes going down and view the disk array as Network Access Storage. If they could improve resource management with storage classes where groups of disks are treated differently based on power management and I/O scheduling, it will provide tremendous quality of service levels to workloads.
243) While there can be co-ordination between the controller nodes and data nodes in a cluster, an individual disk or a group of disks in a node does not have a dedicated disk worker to schedule I/O to the disks since storage has always been progressive towards higher and higher disk capacity. When the disks are so much cheaper that their expansion by way of numerous additions and earmarking for purposes are possible, then the dispatcher and execution worker model can even be re-evaluated.
244) The process per disk worker model is still in use today. It was used by early DBMS implementations.  The I/O scheduling manages the time sharing of the disk workers and the operating system offers protection. This model has been helpful to debuggers and memory checkers.
245) The process pool per disk worker model has alleviated the need to fork processes and tear down and every process in the pool is capable of executing any of the read-writes from any of the clients. The process pool size is generally finite if not fixed. This has all of the advantages from the process per disk worker model above and with the possibility of differentiated processes in the pool and their quota.

Today we continue discussing the best practice from storage engineering:

233) Strategies never remain the same if the data and the business change. Consequently, even the longest running strategy is constantly re-evaluated to see if it can still perform as well. This has in fact been demonstrated in commercial database systems with the use of query compilation and recompilation and holds equally true for classifiers and other kinds of analysis.

234) Since strategy is best described by logic, it is very helpful to export is as a module so that it can run anywhere after being written once. This has been demonstrated by my machine learning packages and data mining algorithms regardless of the domain in which the data exists.  At a low-level, the same applies to strategies within individual components because even if they are not immediately re-used, it will be helpful to have version control on them.

235) Optimization of an execution does not merely depend on the data and the strategy. It involves hints from the users, environmental factors and parameters.  All of this play a role in driving down the costs and some are easier to tweak than others.

236) Builtins from storage products are helpful to customers because they are already available and tested. However the use of builtins may not always be deterministic and free of side-effects. Care must be taken to document how the behavior might change.

237) Customers might not see how a speed up may occur in complex systems. While auto-tuning may cover some scenarios, the bulk of scenarios at user level maybe covered with automation.

238) Parametrization helps queries because the plans remains the same. Even if the storage
product merely implements API for listing resources, it could parametrize the filters to apply and the same may be applied to all other similar API.

#codingexercise
Int GetSortedTail(String a) {
For (int i = a.length - 1; i >=0; i--)
    For (int j = a+1; j < a.length; j++)
      {
            If (a[i] > a[j]) {
                      Return len - i;
             }
      }
Return len;
}

Today we continue discussing the best practice from storage engineering:

230) The results of query execution and materialized views are equally helpful to be cached and persisted separately from the actual data. This reduces the load on the product as well as makes the results available sooner to the queries. 

231) Sampling: Sometimes it is difficult to estimate cost without actually visiting each and every value via estimation. If it were feasible to analyze and summarize the distribution of values with the help of histograms, then it is easier to make a call. Instead we could use sampling techniques to get an estimation without having to exhaust the scan. 

232) Full Iterations are sometimes the only way to exhaust the search space. In top-down approach, at least an early use of Cartesian product for instance can be helpful. This has been acknowledged even in the determination of plan space where the base tables are nested as right- hand inputs only after the Cartesian product has been estimated. 

233) Strategies never remain the same if the data and the business change. Consequently, even the longest running strategy is constantly re-evaluated to see if it can still perform as well. This has in fact been demonstrated in commercial database systems with the use of query compilation and recompilation and holds equally true for classifiers and other kinds of analysis. 

234) Since strategy is best described by logic, it is very helpful to export is as a module so that it can run anywhere after being written once. This has been demonstrated by my machine learning packages and data mining algorithms regardless of the domain in which the data exists.  At a low-level, the same applies to strategies within individual components because even if they are not immediately re-used, it will be helpful to have version control on them. 

235) Optimization of an execution does not merely depend on the data and the strategy. It involves hints from the users, environmental factors and parameters.  All of this play a role in driving down the costs and some are easier to tweak than others.

Today we continue discussing the best practice from storage engineering:


225) A shared nothing system must mitigate partial failures. This is a term used to describe the condition when one or more of the participating nodes goes down. In such cases the mitigation may be one of the following: 1) bring down all of the nodes when any one fails which is equivalent to a shared –memory system, 2) use “data skipping” where queries are allowed to be executed on any node that is up and the data on the failed node is skipped and 3) use as much redundancy as necessary to allow queries access to all the data regardless of any unavailability.

226) Search algorithm over data in storage tend to be top-down. Top-down search implies lower costs because it can prune the query plan to what is just relevant. However, top-down search can exhaust memory. It is often helpful, if there could be additional hints taken from the user and the storage system be capable of using the hints.

227) A single query may be run synchronously and serially usually. However, if the user does not see it and there are ways to gain improvement by parallelizing the workers, then it is always better to use that. The caveat here is there forms of two stages: first the work estimation and then the workload distribution

228) Any storage system can be made to perform better with the help of “auto-tuning” In this method, the same workload is studied with different “what-if” plans so that the outcome is chosen as the one most beneficial.

229) Storage queries that are repeated often are useful to cache because chances are the data has not changed significantly to alter the plan that best suits the execution of the query. While the technique has been very popular with relational databases, it actually holds true for many forms of queries and storage products.

230) The results of query execution and materialized views are equally helpful to be cached and persisted separately from the actual data. This reduces the load on the product as well as makes the results available sooner to the queries.

Today we continue discussing the best practice from storage engineering:


221) Sometimes it is helpful to phase out decisions to multiple tiers. For example, with admission control, the tier that handles the connections and dispatches processes may choose to keep the number of client connections below a threshold. At the same time the inner system layer might determine whether the execution is postponed, begins execution with fewer resources or begins execution without restraints.

222) The decision on the resources can come from the cost involved in the query plan. These costs might include the disk devices that the query will access, the number of random and sequential I/Os per device, the estimates of the CPU load of the query, the number of key-values to process and the amount of memory foot-print of the query data structures.

223) With a shared-nothing architecture, there is no sharing at the hardware resource level. In such cases multiple instances of the storage product may be installed or a cluster mode deployment may be involved. Each system in the cluster stores only a portion of the data and requests are sent to other members for their data. This facilitates the horizontal partitioning of data.

224) When the data is partitioned with different collections rather than the same collection but different ranges over participating nodes, it is referred to as vertical partitioning. There are some use cases for this where data may have groups and a group might not require partitioning.

225) A shared nothing system must mitigate partial failures. This is a term used to describe the condition when one or more of the participating nodes goes down. In such cases the mitigation may be one of the following: 1) bring down all of the nodes when any one fails which is equivalent to a shared –memory system, 2) use “data skipping” where queries are allowed to be executed on any node that is up and the data on the failed node is skipped and 3) use as much redundancy as necessary to allow queries access to all the data regardless of any unavailability.

Today we continue discussing the best practice from storage engineering:

215) Fault domains are a group covering known faults in an isolation. Yet some faults may occur in combinations. It is best to give names to patterns of faults so that they can be included in the design of components.

216) Data driven computing has required changes in storage products. While previously, online transactional activities were read-write intensive and synchronous, today most processors including order and payments are done asynchronously on data driven frameworks usually employing a message queueing. Storage products do better with improved caching for these processing

217) The latency involved in the execution of an order and the prevention of repeats in the order processing has not been relaxed. A gift card run through a store register for renew must make its way to the backend storage so that it is not charged again when run through and before the prior execution is completed. A storage product does not need to be a database with strong ACID guarantees but it should support serialized readability against all read-write operations

218) When the processing requirements are relaxed from ACID to eventual consistency, the latency may not have been relaxed. Consequently, it is important for the storage to return the results as early as possible. In such cases, it is helpful to evaluate the results such that partial results can be returned.

219) Multiple active versions of a record need not be maintained. If there are several ongoing read writes of the records leading to multiple versions, they may merely hold a reference to a version and those older versions can be brought back by going back in time from the latest version of the record by means of the logs that were captured. This is how Read-Consistency isolation level works in database products that have known to favor versions.

220) Between the choice of versions and timestamps for capturing the data changes, it is always better to denote the name and the version for an entity as it narrows down the iteration of the records after they have been fetched.

Today we continue discussing the best practice from storage engineering:
210) Location-based services are a key addition to many data stores simply because location gives more interpretations to the data that solve major business use cases. In order to facilitate this the location may become part of the data and maintained routinely or there must be a service close to the storage that relates them

211) Although storage servers share similar capabilities as compute layer api servers, they are not as transaction oriented as the api layer in many cases.   This calls for a more relaxed execution that includes among other background tasks and asynchronous processing. This affects some of the choices that may be different from compute layer

212) The consensus protocol used in distributed components of a storage system, is required to be fault-tolerant. However, the choice of consensus protocol may vary from system to system

213) Message passing between agents in a distributed environment is required. Any kind of protocol can suffice for this. Some storage systems like to use open source in this regard while others build on message queuing.

214) Every storage server prepares for fault tolerance. Since faults can occur in any domain, temporarily or permanently, each component determines which activities to perform and how to overcome what is not available.

215) Fault domains are a group covering known faults in an isolation. Yet some faults may occur in combinations. It is best to give names to patterns of faults so that they can be included in the design of components .

Today we continue discussing the best practice from storage engineering:

205) Social engineering applications also have a lot of load balancing requirements and therefore more number of servers may need to be provisioned to handle their load. Since a storage tier does not necessarily expose load balancing semantics, it could call out when an external load balancer must be used.

206) Networking dominates storage for distributed hash tables and message queues to scale to social engineering applications. Whatsapps’ Erlang and FreeBSD architecture has shown unmatched symmetric multiprocessing (SMP) scalability

207) Unstructured data generally becomes heterogenous because there is no structure to hold them consistent. This is a big challenge for both data ingestion as well as machine parsing of data. Moreover, the data remains incomplete as its form and heterogeneity increases.

208) Timeliness of executing large data sets is important to certain systems that cannot tolerate wide error margins. Since the elapsed and execution time differ depending on the rate at which tasks get processed, the overall squeeze meets rock hard limits unless there are ways to tradeoff between compute and storage.

209) Spatial proximity of data is also important to prevent potential congestion along routes. In the cloud this was overcome with dedicated cloud network and direct access for companies but this is not generally the norm for on-premise storage.

210) Location-based services are a key addition to many data stores simply because location gives more interpretations to the data tha solve major business use cases. In order to facilitate this the location may become part of the data and maintained routinely or there must be a service close to the storage that relates them

Today we continue discussing the best practice from storage engineering:
200) Data modeling and analysis: Data model may be described with entity-relationship diagrams, json documents, objects and graph nodes. However, the models are not final until several trials. Allowing the versions of data models to be kept also helps subsequent analysis.

201) Data Aggregation depends largely on the tools used. A sql query can perform rollups. A map-reduce can perform summation. They are very different usages but the storage tier can improve the performance if it is dedicated to either. In order to separate out the usages on a shared storage tier, we could classify the workloads and present different sites with redundant copies or materialized views.

202) Even if we separate out the query processing use of the storage tier from the raw data transfer into the storage, we need to maintain separate partitions for read-only data from read-write data. This will alleviate performance considerations as well as inconsistent views.

203) Social engineering data has had a phenomenal use case of using unstructured data storage instead of relational databases. This trend only expands and the requirements for the processing of chat messages and group discussions are way different from conventional file or object-based storage.

204) The speed of data transfer and not just the size is also very critical in the case of social engineering applications such as Facebook and Twitter. In such cases, we have to support a large number of concurrent message. A message platform such as one written in Erlang for Whatsapp may be more performant than servers written with extensive inter process communication

205) Social engineering applications also have a lot of load balancing requirements and therefore more number of servers may need to be provisioned to handle their load. Since a storage tier does not necessarily expose load balancing semantics, it could call out when an external load balancer must be used.

Today we continue discussing the best practice from storage engineering:

195) User’s location, personally identifiable information and location-based services are required to be redacted. This involves not only parsing for such data but also doing it over and over starting from the admission control on the boundaries of integration. If the storage tier stores any of these information in the clear during or after the data transfer from the pipeline, it will not only be a security violation but also fail compliance.

196) Data Pipelines can be extended. The storage tier needs to be elastic and capable of meeting future demands. Object Storage enables this seamlessly because it virtualizes the storage. If the storage spans clusters, nodes can be added. Segregation of data is done based on storage containers.

197) When data gets connected, it expands the value. Even if the storage tier does not see more than containers, it does very well when all the data appears in its containers. The connected data has far more audience than it had independently. Consequently, the storage tier should facilitate data acquisition and connections

198) Big Data is generally accumulated from some source.  Sensor data for example can be stored in NoSQL Databases. However, the data is usable when the right metadata is also recorded with the observational data. To do this continuously, the storage tier must facilitate metadata acquisition.

199) Cleaning and parsing: Raw data is usually noisy and imperfect. It has to be carefully parsed. For example, with full text analysis, we perform stemming and multi-stage pre-processing before the analysis. This applies to admission control and ingestion as well.

200) Data modeling and analysis: Data model may be described with entity-relationship diagrams, json documents, objects and graph nodes. However, the models are not final until several trials. Allowing the versions of data models to be kept also helps subsequent analysis.


#codingexercise
How does SkipList work?
SkipList nodes have multiple next pointers that point to adjacencies based on skipping say 4,2,1
In a sorted skiplist this works as follows:

SkipListNode skipAhead(SkipAheadNode a, SkipAheadNode b) {
SkipListNode cur = a
SkipListNode target = b;
If ( a == null) return a;
For ( cur; cur.next && cur.next.data <= target.data; ) {
// skip by 4, if possible
If (cur.next && cur.next.next && cur.next.next.next && cur.next.next.next.next &&
cur.next.next.next.next <= target.data)
cur = cur.next.next.next.next;
// Skip by 2, if possible
If (cur.next && cur.next.next &&
cur.next.next.next. <= target.data)
cur = cur.next.next;
// Skip by 1, if possible
If (cur.next  &&
cur.next <= target.data)
cur = cur.next;
}
Return cur.next;
}
Since the SkipList already has the links at skip levels  of 4,2,1 etc we avoid the checking and using next.next.next notations

The number of levels for skipping may not be restricted to using 4,2,1 only.

Today we continue discussing the best practice from storage engineering:

191) A pipeline must hold against a data tsunami. In addition, data flow may fluctuate and the pipeline must hold against the ebb and the flow. Data may be measured in rate, duration and size and the pipeline may need to become elastic. If the storage tier cannot accommodate a single pipeline from being elastic, it must mention its Service Level Agreement clearly

192) Data and their services have a large legacy in the form of existing products, process and practice. A storage tier cannot disrupt any of these and therefore must provide flexibility to handle such diverse data transfers as Extract-Transform-Load and analytical pipeline.

193) Extension: ETL may require flexibility in extending logic and in their usages on clusters versus servers. Microservices are much easier to be written. They became popular with Big Data storage. Together they have bound compute and storage into their own verticals with data stores expanding in number and variety.  Queries written in one service now need to be written in another service while the pipeline may or may not support data virtualization. Depending on the nature of the pipeline, the storage tier may also change.

194) Both synchronous and asynchronous processing need to be facilitated so that some data transfers can be run online while others may be relegated to the background. Publisher-subscriber message queues may be used in this regard.  Services and brokers do not scale as opposed to cluster- based message queues. It might take nearly a year to fetch the data into the analytics system and only a month for the analysis. While the benefit for the user may be immense, their patience for the overall time elapsed may be thin. Consequently, a storage tier can at best not require frequent and repeated data transfers from the user. Object Storage, for instance, handles multi-zone application automatically

195) User’s location, personally identifiable information and location-based services are required to be redacted. This involves not only parsing for such data but also doing it over and over starting from the admission control on the boundaries of integration. If the storage tier stores any of these information in the clear during or after the data transfer from the pipeline, it will not only be a security violation but also fail compliance.

185) Reliability of data: A storage platform can provide redundancy and availability but it has no control on the content. The data from pipelines may sink into the storage but if the pipeline is not the source of truth, the storage tier cannot guarantee that the data is reliable.  Garbage in Garbage out applies to storage tier also.

186) Cost based optimization: When we are able to determine the cost function for a state of the storage system or the processing of a query, we naturally try to work towards the optimum by progressively decreasing the cost. Some methods like simulated annealing serves this purpose. But the tradeoff is that the cost function is an oversimplification the trend to consistently lower the costs as a linear function does not represent all the parameters of the system. Data mining algorithms may help here better if we can form a decision tree or a classifier that can encapsulate all the logic associated with the parameters from both supervised and unsupervised learning

187) AI/ML pipeline: One of the emerging trends of vectorized execution is its use with new AI/ML packages that are easy to run on GPU based machines and pointing to the data from the pipeline. While trees, graphs and forests are a way to represent the decision-making models of the system, the storage tier can enable the analysis stack with better concurrency, partitions and summation forms.

188) Declarative querying:  SQL is a declarative querying language. It works well for database systems and relational data. It’s bridging to document stores and Big Data is merely a convenience. A storage tier does not necessarily participate in the data management systems. Yet the storage tier has to enable querying.

189) Support for transformations from batch to Stream processing using the same Storage Tier: Products like Apache Flume are able to support dual mode processing by allowing transformations to different stores. Unless we have a data management system in place a storage tier does not support SQL query keywords like Partition, Over, On, Before, TumblingWindow. The support for SQL directly from the storage tier using an iteration of storage resources, is rather limited. However if the support for products like Flume is there, then there is no difference to analysis whether the product is a time series database or an Object Storage.

190) A pipeline may use the storage tier as a sink for the data. Since pipelines have their own reliability issues, a storage product cannot degrade the pipeline no matter how many pipelines share the storage.

Today we continue discussing the best practice from storage engineering:

181)  Vectorized execution means data is processed in a pipelined fashion without intermediary results as in map-reduce. The push-based execution means that the relational operators push their results to their downstream operators, rather than waiting for these operators to pull data. If the storage can serve both these models of execution, then they work really well.

182)  Data types – their format and semantics have evolved to a wide degree. The pipeline must at least support primitives such as variants, objects and arrays. Support for data stores has become more important than services Therefore datastores do well to support at least some of these primitives in order to cater to a wide variety of analytical workloads.

183) Time-Travel - Time Travel walking through different versions of the data In SQL queries, this is done with the help of AT and BEFORE keywords. Timestamps can be absolute, relative with respect to current time, or relative with respect to previous statements. This is similar to change data capture in SQL Server so that we have historical record of all the changes except that we get there differently

184) Multi-version concurrency control - This provides concurrent access to the data in what may be referred to as transactional memory. In order to prevent reads and writes from seeing inconsistent views, we use locking or multiple copies of each data item.Since a version is a snapshot in time and any changes result in a new version, it is possible for writers to rollback their change while making sure the changes are not seen by others as we proceed.

185) Reliability of data: A storage platform can provide redundancy and availability but it has no control on the content. The data from pipelines may sink into the storage but if the pipeline is not the source of truth, the storage tier cannot guarantee that the data is reliable.  Garbage in Garbage out applies to storage tier also.

Today we continue discussing the best practice from storage engineering :

175) Storage can be used with any analysis package. As a storage tier, a product only serves the basic Create-update-delete-list operations on the resources.  Anything above and beyond that is in the compute layer. Consequently, storage products do well when they integrate nicely with popular analysis packages.
176) When it comes to integration, programmability is important. How well a machine learning SDK can work with the data in the storage, is not only important for the analysis side but also for the storage side. The workloads from such analysis are significantly more different that others because they require GPUs for heavy iterations in their algorithms.
177) There are many data sources that can feed the analysis side but the storage tier is uniquely positioned as a staging for most in and out data transfers. Consequently, the easier it is to to trace the data through the storage tier, the more popular it becomes for the analysis side
178) Although we mentioned documentation, we have not elaborated on the plug and play kind of architecture. If the storage tier can act as a man in the middle with very little disruption to ongoing data I/O, it can become useful in many indirect ways that differ from the usual direct storage of data.
179) A key aspect of storage tier is the recognition of a variety of data formats or in a specific case file types. If the file is an image versus a text file, then it does not help with certain storage chores such as dedeplication and compression.  Therefore, if the storage product provides ability to differentiate data natively, it will help in its acceptance and popularity
180) File types are equally important for applications. Workloads become segregated by file types. If a storage container has all kinds of files, the ability to work with them as if the file types were independent would immensely benefit in separating workloads and providin differential treatment.

Today we continue discussing the best practice from storage engineering:

165) Event Subscription versus appenders: Just like log appenders, there is a possibility to transfer the same event collection result to a large number of destinations simultaneously. These destinations can include files, databases, email recipients and so on.

166) Optional versus mandatory. Every feature in the storage server that is not on the data critical path, is candidate for being turned off to save on resources and improve the data path. This allows equally to components that are not essential. Reducing the number of publisher subscribers is another example of this improvement

167) The number of layers encountered in some operations may become a lot. In such case layering can accommodate components that directly talk to lower layers Layering is not always stripped. However, the justifications to bypass layers must be well made. This counts towards performance by design.

168) There are time periods of peak workload for any storage product. These products serve annual holiday sales, specific anniversaries and anticipated high demand. Utilization of the product under such activity is unusually high. While capacity may be planned to meet the demand, there are ways to tune the existing system to extract more performance. Part of these efforts include switching from being disk intensive to performing more in-memory computation and utilizing other software products to be used with the storage server such as memcache.

169) When the load is high, it is difficult to turn on the profiler to study bottlenecks. However, this can be safely done in advance in performance labs. However, there is an even easier strategy of selectively turning off components that are not required and scaling the component that is under duress.   A priority list of mitigatory steps may be predetermined prior to periods of heavy loads.

170) The monitoring of the Service Level Agreements on the storage server allows us to determine the steps needed to maintain the optimum performance of the system. Maintaining standbys for servers and replacements for cluster nodes or spare hardware either on the chassis or as a plugin helps with the speedy resolution of outages.

# usage example
https://1drv.ms/w/s!Ashlm-Nw-wnWuCiBH-iYeyyYCrFG

Today we continue discussing the best practice from storage engineering:

165) Event Subscription versus appenders: Just like log appenders, there is a possibility to transfer the same event collection result to a large number of destinations simultaneously. These destinations can include files, databases, email recipients and so on.
166) Optional versus mandatory. Every feature in the storage server that is not on the data critical path, is candidate for being turned off to save on resources and improve the data path. This allows equally to components that are not essential. Reducing the number of publisher subscribers is another example of this improvement
167) The number of layers encountered in some operations may become a lot. In such case layering can accommodate components that directly talk to lower layers Layering is not always stripped. However, the justifications to bypass layers must be well made. This counts towards performance by design.
168) There are time periods of peak workload for any storage product. These products serve annual holiday sales, specific anniversaries and anticipated high demand. Utilization of the product under such activity is unusually high. While capacity may be planned to meet the demand, there are ways to tune the existing system to extract more performance. Part of these efforts include switching from being disk intensive to performing more in-memory computation and utilizing other software products to be used with the storage server such as memcache.
169) When the load is high, it is difficult to turn on the profiler to study bottlenecks. However, this can be safely done in advance in performance labs. However, there is an even easier strategy of selectively turning off components that are not required and scaling the component that is under duress.   A priority list of mitigatory steps may be predetermined prior to periods of heavy loads.
170) The monitoring of the Service Level Agreements on the storage server allows us to determine the steps needed to maintain the optimum performance of the system. Maintaining standbys for servers and replacements for cluster nodes or spare hardware either on the chassis or as a plugin helps with the speedy resolution of outages.

Today we continue discussing the best practice from storage engineering:

160) Nativitity of registries – User registries, on the other hand, are welcome and can be arbitrary. In such cases, the registries are about their own artifacts. However, such rregsitries can be stored just the same way as user data. Consequently, the system does not need to participate in the user registries and they can ear mark storage designated storage artifacts for this purpose.

161) Sequences – Sequences hold a special significance in the storage server. If there are several actions taken by the storage server and the actions don’t belong to the same group and there is no way to assign a sequence number, then we rely on the names of the actions as they appear on the actual timeline such as in the logs. When the names can be collected as sequences, we can perform standard query operations on the collections to determine patterns. This kind of pattern recognition is very useful when there are heterogenous entries and the order in which the user initiates them is dynamic.

162) Event driven framework: Not all user defined actions are fire and forget. Some of them may be interactive and since there can be any amount of delay between interactions, usually some form of event driven framework consistently finds its way into the storage server. From storage drivers, background processors and even certain advanced UI controls use event driven framework.

163) Tracing: The most important and useful application of sequences is the tracing of actions for any activity. Just like logs, event driven framework may provide the ability to trace user actions as different system components participate in their completion. Tracing is very similar to profiling but there needs to be a publisher-subscriber model. Most user mode completion of tasks are done with the help of a message queue and a sink.

164) Event Querying: Event driven framework have the ability to not just operate on whole data but also on streams. This makes it very popular to write stream-based queries involving partitions and going over a partition.

165) Event Subscription versus appenders: Just like log appenders, there is a possibility to transfer the same event collection result to a large number of destinations simultaneously. These destinations can include files, databases, email recipients and so on.

Today we continue discussing the best practice from storage engineering:

155) Pooled external resources: It is not just the host and its operating system resources that the storage product requires, it may also require resources that are remote from the local stack. Since such resources can be expensive, it is helpful for the storage product to be thrifty by pooling the resources and servicing as many workers in the storage product as possible.

156) Leveraging monitoring of the host: When an application is deployed to Platform-as-a-service, it no longer has the burden of maintaining its own monitoring. The same applies to storage server as a product depending on where it is deployed. The deeper we go in the stack including the fabric below the storage server, the more amenable they are for the monitoring of the layer above.

157) Immutables: Throughout the storage server we have to use constants for everything from identifiers, names and even temporary data as immutables. While we can differentiate them with number sequences, it is more beneficial to use strings. Strings not only allow names to be given but also help prefix and suffix matching. Even enums have names and we can store them as single instances throughout the system.

158) System artifacts must have names that are not user-friendly because they are reserved and should potentially not come in the way of the names that the user wants to use. Moreover, these names have to be somewhat hidden from the users

159) Registries – When there are collections of artifacts that are reserved for a purpose, they need to be maintained somewhere as a registry. It facilitates lookups. However, registries like lists cannot keep piling up. As long as we encapsulate the logic to determine the lsit, the list is redundant because we can execute the logic and over again. However, this is often hard to enforce as a sound architecture principle

160) Nativitity of registries – User registries, on the other hand, are welcome and can be arbitrary. In such cases, the registries are about their own artifacts. However, such rregsitries can be stored just the same way as user data. Consequently, the system does not need to participate in the user registries and they can ear mark storage designated storage artifacts for this purpose.

Int getIndexOf(node*root, node* moved){
if (root == null) return 0;
if (root.next == null && root == moved) return 0;
if (root.next == null && root != moved) return -1;
int count = 0;
node* tail=root;
while (tail->next) {
If (tail == moved) break;
 tail = tail->next;
count++;
}
if (count ==0) return -1;
Return count;
}

Today we continue discussing the best practice from storage engineering:

Querying: Storage products are not required to form a query layer. Still they may participate in the query layer on enumeration basis. Since the query layer can come separately over the storage entities, it is generally written in the form of a web service. However, SQL like queries may also be supported on the enumerations as shown here: https://1drv.ms/w/s!Ashlm-Nw-wnWt1QqRW-GkZlfg0yz

152) Storage product code is usually a large code base. Component interaction diagrams can be generated and reorganizations may be done to improve the code.  The practice associated with large code bases applies to storage server code as well.

153) Side-by-Side: Most storage products have to be elastic to growing data. However, customers may choose to user more than one instances. In such a case, the storage product must work well in a side by side manner. This is generally the case when we don’t bind to pre-determined operating system resources and acquire them on initialization using the Resource Acquisition is Initialization principle.

154) Multiplexing: By the same argument above, it is not necessary for the Operating System to see multiple instances of the storage product as separate processes. For example, if the storage product is accessible via an http endpoint, multiple instances can register different endpoints while using a host virtualization for utilizing operating system resources. This is the case with Linux containers.

155) Pooled external resources: It is not just the host and its operating system resources that the storage product requires, it may also require resources that are remote from the local stack. Since such resources can be expensive, it is helpful for the storage product to be thrifty by pooling the resources and servicing as many workers in the storage product as possible.

Today we continue  discussing the best practice from storage engineering

148) Standalone mode: Most storage products offer capabilities in a standalone mode. This makes it easy to try out the product without dependencies. One Box deployments also work in this manner. When we remove the dependency on the hardware, we enable the product to be easier to study and try out the features.

149) End User License Agreement:  Products have EULAs which the user is required to accept. If the EULA conditions are not read or accepted, certain functionalities of the server are turned off. Typically, this check happens at the initialization time of the product or the restart of the service. However, the initialization code is global and has very sensitive sequence of steps. Therefore, placing the EULA criteria without leaving the product in a bad state is necessary.

150) Provisioning: Storage artifacts are not just created by user. Some may be offered as built-ins so that the user can start working with their data rather than handling the overhead of setting up containers and describing them. These ready to use built-in storage artifacts lets the user focus on their tasks rather than the system tasks. A judicious choice of all such artifacts and routines are therefore useful to the user.

151) Querying: Storage products are not required to form a query layer. Still they may participate in the query layer on enumeration basis. Since the query layer can come separately over the storage entities, it is generally written in the form of a web service. However SQL like queries may also be supported on the enumerations as shown here: https://1drv.ms/w/s!Ashlm-Nw-wnWt1QqRW-GkZlfg0yz

152) Storage product code is usually a large code base. Component interaction diagrams can be generated and reorganizations may be done to improve the code.  The practice associated with large code bases applies to storage server code as well.

#codingexercise
int getCountEven(node* root)
{
if (root == null) return 0;
int count;
while (root)  {
if (root->value %2 == 0) {
    count++;
}
root = root->next;
}
return count;
}

Today we continue  discussing the best practice from storage engineering:

146) Footprint: The code for a storage server can run on any device. Java for example runs on billions of devices and a storage server written in Java can run even on pocket devices. If the storage is flash and the entire storage server runs only on flash, it makes a great candidate for usability.

147) Editions: Just like the code for storage server can be made suitable for different devices, it can ship as different editions. One way to determine the different kinds of editions is to base it on where the customers demand it. Although there are many perspectives in these decisions, the ultimate service of the product is for the customer.

148) Standalone mode: Most storage products offer capabilities in a standalone mode. This makes it easy to try out the product without dependencies. One Box deployments also work in this manner. When we remove the dependency on the hardware, we enable the product to be easier to study and try out the features.

149) End User License Agreement:  Products have EULAs which the user is required to accept. If the EULA conditions are not read or accepted, certain functionalities of the server are turned off. Typically, this check happens at the initialization time of the product or the restart of the service. However, the initialization code is global and has very sensitive sequence of steps. Therefore, placing the EULA criteria without leaving the product in a bad state is necessary.

150) Provisioning: Storage artifacts are not just created by user. Some may be offered as built-ins so that the user can start working with their data rather than handling the overhead of setting up containers and describing them. These ready to use built-in storage artifacts lets the user focus on their tasks rather than the system tasks. A judicious choice of all such artifacts and routines are therefore useful to the user.

#codingexercise
int getCountOdd(node* root)
{
if (root == null) return 0;
int count;
while (root)  {
if (root->value %2 == 1) {
    count++;
}
root = root->next;
}
return count;
}

Today we continue  discussing the best practice from storage engineering:

143) One of the most prone software faults is heap memory usage especially by the java virtual machine. This requires a lot of effort to investigate and narrow down. Often the remedial steps taken are to increase the memory usage all the way to 4GB for the process. Since leaks have their own stacktrace if they occur deterministically, the finding of the root cause involves trials.

144) Among the various usages of process level statistics, one of the most reviewed usage is memory. Since functionalities within the storage server can be broken down as processes for microservices, we are able to narrow down on individual process. Since processes can be restarted, their restarts is a good indication of malfunction

145) Distributed ledger: this is gaining popularity where there is no central ledger and no requirement for a central ownership for verification of grants and revocations. It mitigates tampering. It is a great storage for storing and processing digital identity data and works for products that do not need belong to an organization or cloud.

146) Footprint: The code for a storage server can run on any device. Java for example runs on billions of devices and a storage server written in Java can run even on pocket devices. If the storage is flash and the entire storage server runs only on flash, it makes a great candidate for usability.

147) Editions: Just like the code for storage server can be made suitable for different devices, it can ship as different editions. One way to determine the different kinds of editions is to base it on where the customers demand it. Although there are many perspectives in these decisions, the ultimate service of the product is for the customer.

#codingexercise
In the segregated list discussed earlier, find the count of nodes swapped:
int getCount Swapped (node* root) {
      return getCount (root)/2;
}

Today we're discussing the best practice from storage engineering:
140) The topology for data transfers has been changing together with the technology stack. Previously even master data or the product catalog of a company was a singleton and today there is a practice to rebuild it constantly.  The data is also allowed to be stagnant as with data lakes and generally hosted in the cloud. On-Premise servers and SAN are being replaced in favor of cloud technologies wherever possible. Therefore, toolsets and operations differ widely and a conformance to ETL semantics for data transfers from the product will generally be preferred by their audience.
141) Most storage products work best when they are warmed up. A storage product may use its own initialization sequence and internal activities to reach this stage. Since the initialization is not done often, it is a convenient time to put all the activities together so that the subsequent operations are efficient. This has been true at all levels starting from class design to product design.
142) The graceful shutdown of products and recovery from bad states has been equally important to most products even if they are not required to maintain strong guarantees. This is particularly the case with those that are susceptible to a large number of faults and failures.  For, example, faults may range from disk and node errors to power failures, network issues, bit flip and random hardware-failures. Reed Solomon erasure coding or Pelican coding try to overcome their faults by determining the m number of error correction chunks from n number of data chunks.

143) One of the most prone software faults is heap memory usage especially by the java virtual machine. This requires a lot of effort to investigate and narrow down. Often the remedial steps taken are to increase the memory usage all the way to 4GB for the process. Since leaks have their own stacktrace if they occur deterministically, the finding of the root cause involves trials.

Today we continue discussing the best practice from storage engineering:

136) Data transfer does not always have to be to and from the software product. Transfer within the product using its organizational hierarchy could also be supported. For example, object storage provides the convenience of copying bucket of objects at a time even he objects may have any folder path like prefix.

137) One of the overlooked facts about data transfer is that it is often done by production support personnel because of the sensitivity of the data involved. They prefer safe option to complicated and most efficient operations.  If the data transfer can be done with the help of a tool and a shell script, it works very well for such transfers. Consequently, there must be a handoff between the developer and the production support and the interface must be something that is easier to use from the production support side.

138) The administrative chores around production data also increases significantly as compared to the datasets that the product is build on and tested. There is absolutely no room for data corruption and outages that are unplanned. If the data transfer tool itself is defective, it cannot be handed over to production. Consequently, the data transfer tools must be proven and preferably part of the product so that merely the connections can be setup.  

139) The data transfers that involve read only operations on the production data are a lot more favored over write-only data. Together this reason and the above constitute the general shift towards Extract-Transform-Load packages to be used with production data instead of writing and leveraging any code for such customization.  

140) The topology for data transfers has been changing together with the technology stack. Previously even master data or the product catalog of a company was a singleton and today there is a practice to rebuild it constantly.  The data is also allowed to be stagnant as with data lakes and generally hosted in the cloud. On-Premise servers and SAN are being replaced in favor of cloud technologies wherever possible. Therefore, toolsets and operations differ widely and a conformance to ETL semantics for data transfers from the product will generally be preferred by their audience.

Today we continue discussing the best practice from storage engineering:

134) Data migration wizards help movement of data as collections of storage artifacts. With the judicious choice of namespace hierarchy and organizational units, the users are relieved from the syntax of storage artifacts and their operations and view their data as either in transit or at rest.

135) Extract-Transform-Load operations are very helpful when transferring data between data stores. In such cases, it is best to author them via user Interface with the help of designer like tools. Logic for these operations may even be customized and saved as modules.

136) Data transfer does not always have to be to and from the software product. Transfer within the product using its organizational hierarchy could also be supported. For example, object storage provides the convenience of copying bucket of objects at a time even he objects may have any folder path like prefix.







#codingexercise
Segregate odd and even nodes in a linked list

node* segregate(node* root) {

if (root == null) return root;
if (root.next == null) return root;
node* prev = null;
node* current = root;
node* next = current->next;
node* head = root;
node* tail=head;

while (tail->next) tail = tail->next;
node* mid= tail;
while( next && next->next &&
       prev!= mid && current != mid && next != mid) {
prev = current;
current = cut (next, current);
prev.next = current;

tail.next = next;
tail = next;
tail.next = null;
next = current->next;

}

return root;

}

node* cut (node* next, node* current) {
current.next = next.next;
next.next = null;
current = current.next;
return current;
}

Today we continue discussing the best practice from storage engineering
:
131) Virtual Timeline: Most entries captured in the log are based on actual timeline. With messages passed between distributed components of the storage, there is a provision for using sequence numbers as a virtual timeline. Together boundaries and virtual timeline enable spatial and temporal capture of the changes to the data suitable for watching. Recording the virtual time with the help of a counter and using it for comparisons is one thing, reconstructing the event sequence is another.
132) Tags: Storage artifacts should support tags so that the users may be able to form collections based on tags. With the help of tags, we can specify access control policies, cross-region replication and manage object life-cycles.
133) Classes: Classes are service levels of storage. Object Storage for instance supports two or more storage classes. A class named standard is used with objects that require frequent access of data. A class named infrequent may be used with objects that have less frequently-accessed data. Objects that have changing access patterns over time may be placed in yet another class.
134) Data migration wizards help movement of data as collections of storage artifacts. With the judicious choice of namespace hierarchy and organizational units, the users are relieved from the syntax of storage artifacts and their operations and view their data as either in transit or at rest.
135) Extract-Transform-Load operations are very helpful when transferring data between data stores. In such cases, it is best to author them via user Interface with the help of designer like tools. Logic for these operations may even be customized and saved as modules.

#codingexercise


List<int> SegregateBasedOddAndEvenValueOfElements(List<int> A)
{
List<int>result = new List<int>();
A.forEach((x,I) => {if (A [i]%2 == 0) result.add(x);});
Var odd = A.Except(result).toList();
result.AddRange(odd);
return result;
}

Today we continue discussing the best practice from storage engineering: 

127) Directory Tree: Perhaps files and folders are the most common way of organizing data and many other forms of storage also enable some notion of such organization. While documents and graphs and other forms are also becoming popular forms of storage, they generally require their own language for storage and are not amenable as a tree. Storage organized as trees should facilitate importing and exporting from other data stores so that the user is freed up from the notion that a storage product is defined by the syntax of storage and transfers the data regardless of whether it is stored on file shares or databases.

128) Runtime: Storage has never intruded into compute. Programs requiring storage have no expectation for storage to do their processing. Yet predicates are often pushed down within the compute so that it is as close to data as possible. However, storage does have a notion of callbacks that can be registered. FileSystemWatchers are an example of this requirement. It is also possible for the Storage Layer to host a runtime so that the logic on its layer can be customized.

129) Privacy: guest access to artifacts and routines are sometimes allowed in the product to provide a minimum functionality that is available outside provisioning. The same access can also be used when the user does not need to be identified. In such case, there is no audit information available on the user and the actions

130) Boundaries: The boundaries in a continuous stream of data is usually dependent on two things: a fixed length boundary of segments or a variable length depending on the application logic. However, application refers to the component and the scope performing demarcation on the data. Therefore, boundaries can be nested, non-overlapping and frequently require translations.

131) Virtual Timeline: Most entries captured in the log are based on actual timeline. With messages passed between distributed components of the storage, there is a provision for using sequence numbers as a virtual timeline. Together boundaries and virtual timeline enable spatial and temporal capture of the changes to the data suitable for watching.

Today we continue discussing the best practice from storage engineering: 

123) StackTraces:  When the layers of a software product are traversed by shared data structures such as login contexts , then it is helpful to capture and accumulate the stacktraces at the boundaries for troubleshooting purposes. These ring buffer of stack traces provide instant history for the data structures

124) Wrapping: This is done not just for certificates or exceptions. Wrapping works for any artifact that is refreshed or renewed and we want to preserve the old with the new. This may apply even to headers and metadata of data structures that fall in the control path.

125) Bundling: Objects are not only useful as a  standalone. They appear in collections. Files for instance can be archived into a tar ball that behaves like any other file. The same is true for all storage artifacts that can be bundled and products do well to promote these.

126) Mount: A remote file share may be mounted such that it appears local to the user. The same is true for any storage product that does not need to be reached over http. File protocols already enable remote file shares and there are many protocols to choose from. Syslog, ssh are also used to transfer data. Therefore, a storage product may choose from all conventional modes of data transfer from remote storage.

127) Directory Tree: Perhaps files and folders are the most common way of organizing data and many other forms of storage also enable some notion of such organization. While documents and graphs and other forms are also becoming popular forms of storage, they generally require their own language for storage and are not amenable as a tree. Storage organized as trees should facilitate importing and exporting from other data stores so that the user is freed up from the notion that a storage product is defined by the syntax of storage and transfers the data regardless of whether it is stored on file shares or databases.