Today we continue discussing the best practice from storage engineering:

395) Catalogs also need to be served to a variety of devices. Websites tailored for mobile and desktop differ even in the content that is presented and not just the style, markup, script or logic. There is virtually no restriction to how much resource can be stored in the object storage and these can co-exist.

396) Similar to catalogs but in the form of document collections, libraries of digital content are just as easy to collect in organizations as any other repository. Most of these document libraries are using relational databases but they have no difference from object storage in terms of the use of the content and since versioning is supported.

397) These libraries differ from the catalogs in that they not only read-only traffic but also read-write on the documents in the collection. It is also internal to the organization as opposed to public catalogs

398) These libraries also participate in a variety of workflows which were earlier subject to limitations of the service as well as the portal where users sign in to access their documents. The use of an object storage on the other hand removes such restrictions

399) Unlike catalogs, libraries have to provide significant resource access control. Object storage with its S3 api is suitable for this purpose.

400) Unlike catalogs libraries don’t need to be served to multiple devices. However, libraries tend to grow in number. Therefore, object storage can encompass them all and provide limitless storage.

Today we continue discussing the best practice from storage engineering:

391) Some companies in the retail industry have a lot of catalogs. Although there is significant investment in Master Data management, solutions similar to those can be built on top of object storage. This is definitely a niche space and one that can support an emerging trend.

392) These retail companies process significant read-only traffic for their catalogs with the help of http proxies and web services. The investment can be maintained the same so long as the read only operations on the backend translate to fetching objects from the object store. This can help ease the transition to directly serving it from the object storage.

393) Catalogs participate in a variety of workflows such as rewards service, promotions and campaigns and so on.  When the catalogs are served from the service, they are subject to the limitations of the service. When the catalog is directly served from the object storage, then it becomes far easier to start new services.

394) Catalogs typically require no access controls since they are served to the public. This makes it more appealing to move it to object storage where content distribution, replication and multi-site support is available out of the box.

395) Catalogs also need to be served to a variety of devices. Websites tailored for mobile and desktop differ even in the content that is presented and not just the style, markup, script or logic. There is virtually no restriction to how much resource can be stored in the object storage and these can co-exist.

Today we continue discussing the best practice from storage engineering:

383) Health data is not just sharded by customer but also maintained in isolated shared-nothing pockets with their own management systems. Integration of data to represent a whole for the same customer is the new and emerging trend in the health industry. Organizations and companies are looking to converge the data for an individual without losing privacy or failing to comply with government regulations.

384) Health data has numerous file types for the data captured from the patients. These could range from small text documents to large images. Unlike cluster file systems that consolidate data to a cluster, these data artifacts are scattered throughout repositories. In addition, there is a lot of logic to who can access what data leading to some bulky user interface for read and edit by providers, insurance, administrators and end users.

385) In addition to access over health data, agencies and providers frequently exchange health records which leads to a high traffic of data from all the data sources. Virtually no data is erased from the system and historical records going back several years are maintained. The accumulation of data records also has no chance to go to a warehouse because it is always active and online.

386) Retail industry has traditionally embraced mammoth sized databases and some even on large Storage Area Networks. Their embrace of databases and data warehouses have been banner use cases for online transaction processing and online analytical processing. Yet vectorized execution models are gaining ground in nascent retail companies where they want to wrap all purchases, rental payments and servicing fees as billing events that flow to processors. It is highly unlikely that they will switch to management and analytics solutions overnight that are based on key value stores or object stores.



387) Unlike health industry data stores, Retail industry data stores are all self contained homogenous and full service management systems. Writing a new service for retail industry merely points to other existing services as data stores or shared databases. Even store front devices such as point of sale registers point to queues which inevitably process their messages from back-end databases

388) While these industries may view data stores, queues, services and management systems as data sources, they did not have the opportunity until recently to consolidate their data sources with storage first design.

#codingexercise

maximize the coin collection when two players take turns when picking out the coins from either end

Int GetCoins(List<int> coins, int i, int j)
{
int n = coins.count;
If (i >= j) return 0;
If (i==j) return coins[i];

If (j == i+1) return max(coins[i], coins[j]);
Return max(coins[i] + max(GetCoins(coins, i+2, j), GetCoins(coins, i+1,j-1)) ,
                      coins[j] + max(GetCoins(coins,i+1, j-1), GetCoins(coins,i,j-2)));
}

378. Today we continue discussing the best practice from storage engineering:
379. 
     
380. 378) Finance data is also heavily regulated. The Sarbanes-Oxley act sought to bring control to the corporations in order to avoid accounting scandals. It specified disclosure controls, audit and the compliance terms
381.   

379. 379) ETL tools are widely used for in house finance data. On the other hand, global indexes, ticker price and tradings are constantly polled or refreshed. In these cases, it is hard to unify storage across departments, companies and industries. A messaging framework is preferred instead. Object storage could be put to use in these global stores but their adoption and usage across companies is harder to enforce. 

380. 
     
381. 380) Web services and gateways are preferred to distribute data and different kinds of finance data processing systems evolve downstream. These systems tend to have their own storage while transforming and contributing to the data in flight. The web-services are also popular for using as data source in parallel analysis usages. Since distribution and flow is more important for the data, the origin is considered the source of truth and the data changes are not propagated back to the origin. 

381. 
     
382. 381) Health industry data is another example with its own needs around data compliance. The Health insurance portability and Accountability act required a lot of controls around who, when and where can get access to personally identifiable information 

382. 
     
383. 382) Health data is often tightly integrated into proprietary stacks and organizations. Yet they are also required to participate in providing web access to all the information surrounding an individual at the same place. This makes them require a virtualized cross company data storage. 

383. 
     
384. 383) Health data is not just sharded by customer but also maintained in isolated shared-nothing pockets with their own management systems. Integration of data to represent a whole for the same customer is the new and emerging trend in the health industry. Organizations and companies are looking to converge the data for an individual without losing privacy or failing to comply with government regulations. 

Today we continue discussing the best practice from storage engineering:

375. 375) Most storage products don’t differentiate between human and machine data because it involves upper layers of data management. However, dedicated differentiation between human and machine data can make the products more customized for these purposes. 

376. 
     
377. 376) Data storage requirements change from industry to industry. Finance data storage is largely in the form of distributed indexes and continuous data transfers. A cold storage product does not serve its needs even if the objects are accessible over the web 

377. 
     
378. 377) Finance data is subject to a lot of calculations and proprietary and often well-guarded calculators that have largely relied on relational databases. Yet these same companies have also adopted NoSQL storage in favor of their warehouses. As their portfolios grow, they incubate new and emerging features and increasingly favor new technologies 

378. 
     
379. 378) Finance data is also heavily regulated. The Sarbanes-Oxley act sought to bring control to the corporations in order to avoid accounting scandals. It specified disclosure controls, audit and the compliance terms  

379. 
     
380. 379) Health industry data is another example with its own needs around data compliance. The Health insurance portability and Accountability act required a lot of controls around who, when and where can get access to personally identifiable information 

380. 
     
381. 380) Health data is often tightly integrated into proprietary stacks and organizations. Yet they are also required to participate in providing web access to all the information surrounding an individual at the same place. This makes them require a virtualized cross company data storage.

Today we continue discussing the best practice from storage engineering
:
371) Data management software such as Cloudera can be deployed and run on any cloud. It offers an enterprise data hub, an analytics DB, and operational DB, data science and engineering and essentials. It is elastic and flexible, it has high performance analytics, it can easily provision over multiple clouds and it can be used for automated metering and billing.  Essentially they allow different data models, real-time data pipelines and streaming applications with their big data platform. They enable data models to break free from vendor lockins and with the flexibility to let it be community defined.

372) The data science workbench offered from Cloudera involves a console on a web browser that users can authenticate themselves with using Kerberos against the cluster KDC. Engines are spun-up and we can seamlessly connect with Spark, Hive, and Impala. The engines are spun up based on engine kernels and profiles.

373) Cloudera Data Science workbench uses Docker and Kubernetes. Cloudera is supported on dedicated Hadoop hosts.  Cloudera also adds a data engineering service called Altus. It’s a platform that works against a cloud by allowing clusters to be setup and torn down and jobs to be submitted to those clusters. Clusters may be Apache Spark, MR2 or Hive.

374) Containerization technologies and Backend as a service aka lambda functions can also be supported by products such as Cloudera which makes them usable with existing public clouds while it offers an on-premise solution

375) Most storage products don’t differentiate between human and machine data because it involves upper layers of data management. However, dedicated differentiation between human and machine data can make the products more customized for these purposes.

Today we continue discussing the best practice from storage engineering :

366) Sometimes performance drives the necessity to create other storage products. Social engineering utilizes storage products that are not typical to enterprise or cloud storage. This neither means that social engineering applications cannot be built on cloud services nor does it mean that the on-premise storage products necessarily have to conform to organizational hardware or virtualware needs.

367) To improve performance and scalability, Facebook had to introduce additional parallelization in the runtime and the shared contexts which they called "WorkerContext". Bottlenecks and overheads such as checkpointing were addressed by scheduling. This was finer level than what the infrastructure provided.

368) Facebook even optimized the memory utilization of the graph infrastructure because it allowed arbitrary vertex id, vertex value, edge and message classes. They did this with 1) serializing edges with byte array and 2) serializing messages on the server.

369) Facebook improved parallelization with sharded aggregators that provided efficient shared state across workers. With this approach, each aggregator gets assigned to a randomly picked worker which then gathers the values, performs the aggregation and distributes the final values to master and other workers. This distributes the load that was otherwise entirely on the master.

370) Many companies view graphs as an abstraction rather than an implementation of the underlying database. There are two reasons for this:
First, Key-value stores suffice to capture the same information in a graph and can provide flexibility and speed for operations that can be translated as queries on these stores. Then these can be specialized for the top graph features that an application needs.
Second, different organizations within the company require different stacks built on the same logical data for reasons such as business impact, modular design and ease of maintenance.

Today we continue discussing the best practice from storage engineering :

360) A rewarding service increases appeal and usage by customers. This is what makes Blockchain popular.

361) Many database clusters are used as a failover cluster and not as a performance or scaleable cluster. This is primarily because the storage server is designed for scale-up versus scale-out.  This is merely an emphasis on the judicious choice of technology for design.

362) Some database server uses SAN, it is a shared storage and does not go offline.  Most storage products have embraced Network Access Storage. Similar considerations are true when the database server is hosted on a container and the database is on a shared volume

363) The clustering does not save space or efforts for backup or maintenance. And it does not scale out the reads for the database. Moreover, it does not give a 100% uptime.

364) Reporting stack is usually a pull and transformation operation on any database and is generally independent of the data manipulation from online transactions. Therefore, if a storage product can simplify its design by offloading reporting stack to say time-series database, grafana and charting stack, then it can focus on storage server related design.

365) The above is not necessarily true for analysis stacks which often produce a large number of artifacts during computations and as such are heavily engaged in the read-write on the same storage stack.

Today we continue discussing the best practice from storage engineering:

355) P2P is considered a top-heavy network. Top-heavy means we have an inverted pyramid of layers where the bottom layer is the network layer.  This is the substrate that connects different peers.  The overlay nodes management layer handles the management of these peers in terms of routing, location lookup and resource discovery. The layer on top of this is the features management layer which involves security management, resource management, reliability and fault resiliency.

356) Let us take a look at contracts. We discussed this earlier in reference to Merkle trees.  A smart contract can even include code and evaluate dynamically just like a classifier with rules in program order.

357) A utility token issued for authorization need not be opaque. It can be made up of three parts – one which represent the user, another which represents the provider, and an irrefutable part that is like a stamp of authority. The section from the provider may also include scope for the resources authorized.

358) The attestation mechanism itself might vary. This might include, for example, Merkle tree where each node of the tree represents an element of Personally-Identifiable-Information (PII) along with the hash and the hash of the hashes of the child nodes.  The root hash then becomes the fingerprint of the data being attested.

359) An immutable record which has its integrity checked and agreed on an ongoing basis provides a venerable source of truth.

360) A rewarding service increases appeal and usage by customers. This is what makes Blockchain popular.

List <List <int>> getAllArrangements (List <List <int>> selected) {
List <List <int>> ret = new List < List <int>>();
Selected.distinct ().forEach ( x -> ret.addAll (getPermutations (x)));
Return ret;
}

Today we continue discussing the best practice from storage engineering: 

351) P2P can be structured or unstructured.

352) In a structured topology, the P2P overlay is tightly controlled usually with the help of a distributed hash table (DHT). The location information for the data objects is deterministic as the peers are chosen with identifiers corresponding to the data object's unique key. Content therefore goes to specified locations that makes subsequent query easier.

353) Unstructured P2P is composed of peers joining based on some rules and usually without any knowledge of the topology. In this case the query is broadcast and peers that have matching content return the data to the originating peer. This is useful for highly replicated items but not appropriate for rare items. In this approach, peers become readily overloaded and the system does not scale when there is a high rate of aggregate queries.

354) Recently there have been some attempts at standardization on the key based routing KBR API abstractions and OpenHash - an open publicly DHT service that helps with a unification platform.

355) P2P is considered a top-heavy network. Top-heavy means we have an inverted pyramid of layers where the bottom layer is the network layer.  This is the substrate that connects different peers.  The overlay nodes management layer handles the management of these peers in terms of routing, location lookup and resource discovery. The layer on top of this is the features management layer which involves security management, resource management, reliability and fault resiliency.

Today we continue discussing the best practice from storage engineering :
345) A P2P network introduces a network first design where peers are autonomous agents and there is a protocol to enable them to negotiate contracts, transfer data, verify the integrity and availability of remote data and to reward with payments. It provides tools to enable all these interactions. Moreover, it enables the distribution of the storage of a file as shards on this network and these shards can bee stored using a distributed hash table. The shards themselves need not be stored in this hash table, rather a distributed network and messaging could facilitate it with location information.
346) Messages are helpful to enforce consistency as nodes come up or go down. For example, a gossip protocol may be used for this purpose and it involves propagating updates via message exchanges.
347) Message exchanges can include state or operation transfers. Both involve the use of vector clocks.
348) In state transfer model, each replica maintains a state version tree which contains all the conflicting updates. When the client sends its vector clock, the replicas will check whether the client state precedes any of its current versions and discard it accordingly. When it receives updates from other replicas via gossip, it will merge the version trees.
349) In operation transfer model, each replica has to first apply all operations corresponding to the cause before those corresponding to the effect. This is necessary to keep the operations in the same sequence on all replicas and is achieved by adding another entry in the vector clock, a V-state, that represents the time of the last updated state. In order that this causal order is maintained, each replica will buffer the update operation until it can be applied to the local state A tuple of two timestamps - one from the client's view and another from the replica's local view is associated with every submitted operation.
350) Since operations are in different stages of processing on different replicas, a replica will not discard the state or operations it has completed until it sees the vector clocks from all others to have preceded it

Today we continue discussing the best practice from storage engineering :

341) Emerging trends like Blockchain have no precedence for storage standards. A blockchain is a continuously growing list of records called blocks which are linked and secured using cryptography. Since it is resistant to tampering, it becomes an open distributed ledger to record transactions between two parties.
342) Blockchain is used for a variety of purposes. Civic for instance enables its users to login with their fingerprints. It uses a variety of smart contracts, an indigenous utility token and new software applications. A Merkel tree is used for attestation.
343) Storage servers to store such distributed ledgers is not standardized yet though any stream based store would help. Object Storage suitability for storage ids is not popular yet.
344) A Peer-to-Peer (P2P) network is popular In production storage, peer to peer networks were not popular as data because data accrues based on popularity not on utility.
345) A P2P network introduces a network first design where peers are autonomous agents and there is a protocol to enable them to negotiate contracts, transfer data, verify the integrity and availability of remote data and to reward with payments. It provides tools to enable all these interactions. Moreover, it enables the distribution of the storage of a file as shards on this network and these shards can bee stored using a distributed hash table. The shards themselves need not be stored in this hash table, rather a distributed network and messaging could facilitate it with location information.

Today we continue discussing the best practice from Storage Engineering:

Storage units such as files and objects may be stored as shards on different constituents of the network. The location information in such cases becomes part of the messages between the peers.

Distributed Hash Table has gained widespread popularity in distributing load over a network. There are some well-known players in this technology. Kademlia for instance proves many of the theorems for DHT.

Messages are routed through low latency paths and use parallel asynchronous queries. Message queuing itself and its protocol is an excellent communication mechanism for a distributed network.

Integrity of the data is verified with the help of Merkle trees and proofs in such distributed framework. Others use key based encryptions.

Partial audits reduce overhead in compute and storage. For example, shards of data may be hashed many times to generate pre-leaves in a Merkle tree. And the proof may be compact but the tree may not be compact. Partial audits therefore save on compute and storage.

Data does not always need to be presented in entirety and always from the source of truth. The data may remain completely under the control of the owner while a challenge-proof may suffice

Messages may contain just the hash of the data and the challenge while the responses may contain just the proof. This enables the message exchange to be meaningful while keeping the size within a limit.

Contracts are signed and stored by both parties. Contracts are just as valuable as the sensitive data. Without a contract in place, there is no exchange possible.

Parties interested in forming a contract use a publisher-subscriber model that is topic-based and uses a bloom filter implementation which tests whether an element is a member of a set. 

Today we continue discussing the best practice from storage engineering:

326) A higher layer that manages and understands abstractions, provides namespace and data operation ordering is present in almost all storage management systems

327) A lower layer comprising of distribution and replication of data without actually requiring any knowledge of abstractions maintained by the higher layer is similarly present in almost all storage products.

328) Similarly, the combination of the two layers described above is almost always separated from the front-end layer interpreting and servicing the user requests.

329) Working with streams is slightly different from fixed sized data. It is an ordered set of references to segments. All the extents are generally immutable.

330) If a shared resource can be represented as a pie, there are two ways to enhance the usage: First, make the pie bigger and the allocate the same fractions in the increment. Second, dynamically modify the fractions so that at least some of the resource usages can be guaranteed some resource.

331) In order to remove the notion of a centralized storage provider, some storage products have preferred distributed cloud storage network. A distributed hash table overlays the network.

332) Storage units such as files and objects may be stored as shards on different constituents of the network. The location information in such cases becomes part of the messages between the peers.

333) Distributed Hash Table has gained widespread popularity in distributing load over a network. There are some well-known players in this technology. Kademlia for instance proves many of the theorems for DHT.

334) Messages are routed through low latency paths and use parallel asynchronous queries. Message queuing itself and its protocol is an excellent communication mechanism for a distributed network.

335) Integrity of the data is verified with the help of Merkel trees and proofs in such distributed framework. Others use key based encryptions

Today we continue discussing the best practice from storage engineering:

323) Cost of storage is sometimes vague because it does not necessarily encompass all operational costs for all the units because the scope and purpose changes for the storage product. The cost is not a standard but we can get comparable values when we take the sum of the costs and divide it for unit price.
324) Cost is always a scalar value and usually calculated by fixing parameters of the system. Different studies may use the same parameters but have widely different results. Therefore, it is not good practice to compare studies unless they are all relatively performing the same.
325) The total cost of ownership encompasses cost for operations and is usually not reflected on new instances of the storage product. It is used with products that have been used for a while and are becoming a liability.
326) A higher layer that manages and understands abstractions, provides namespace and data operation ordering is present in almost all storage management systems
327) A lower layer comprising of distribution and replication of data without actually requiring any knowledge of abstractions maintained by the higher layer is similarly present in almost all storage products.
328) Similarly, the combination of the two layers described above is almost always separated from the front-end layer interpreting and servicing the user requests.
329) Working with streams is slightly different from fixed sized data. It is an ordered set of references to segments. All the extents are generally immutable.
330) If a shared resource can be represented as a pie, there are two ways to enhance the usage: First, make the pie bigger and the allocate the same fractions in the increment. Second, dynamically modify the fractions so that at least some of the resource usages can be guaranteed some resource.

The design of appenders for Object Storage:

Object storage has established itself as a “standard storage” in the enterprise and cloud. As it brings many of the storage best practice to provide durability, scalability, availability and low cost to its users, it could expand its role from being a storage layer to one that facilitates data transfer. We focus on the use cases of object storage so that the onerous tasks of using object storage can be automated. Object storage then transforms from being a passive storage layer to one actively participating in data transfers to and from different data sinks and sources simply by attaching appenders. Earlier, we described the data transfer to the object storage with the help of connectors. The purpose of this document is to describe appenders that facilitate intelligent automations of data transfer to external data sinks with minimal or no disruption to their operations.  

File-systems have long been the destination to capture traffic and while file system has evolved to stretching over clusters and not just remote servers, it remains inadequate as a blob storage. The use of connectors and appenders enhance the use of object storage and do not compete with the usages of elastic file stores.  

We describe the appenders not by the data sink they serve but by the technology behind the appenders. We enumerate synchronous send and receive over protocols, asynchronous publisher-subscriber model, compressed data transfer, deduplicated data transfer and incremental rsync based backups to name a few.  We describe their advantages and disadvantages but do not provide a prescription to the retail world which allows them to be used on a case by case basis and with flexible customizations. In this sense, the appenders can be presented in the form of a library with the object storage so that commodity and proprietary data sinks may choose to receive data rather than actively poll for it. 

While public cloud object storage solutions offer cloud based services such as Amazon’s Simple Notification Services and Azure Notification hubs, on-premise object storage has the ability to make the leap from standalone storage offering to a veritable solution integration packaging. Public-Cloud offers robust myriad features for their general-purpose cloud services while the appenders specialize in the usages of the object storage and come as an out of box feature for the object storage. 

The data that flows out of object storage is often pulled from various receivers. That usage continues to be mainstream for the object storage. However, we add additional usages where the appenders push the data to different data sinks. Previously there were in-house scripts for such data transfers. Instead we make it part of the standard storage and provide users just the ability to configure it to their purpose.  

The ability to take the onerous routines of using object storage as a storage layer from the layer above across different data sinks enables a thinner upper layer and more convenience to the end user. The customizations in the upper layer are reduced and the value additions bubble up the stack.   

On-premise object storage is no more a standalone. The public cloud has moved towards embracing on-premise compute resources and their management via System Center integration. The same applies to on-premise object storage as well.  Consequently, the technologies behind the appenders are not only transparent but they are also setup for being monitored and reported via dashboards and charts This improves visibility into the data transfers while enabling cost accounting.   

An Object Storage offers better features and cost management, as it continues to stand out against most competitors in the unstructured storage. The appenders lower the costs of usage so that the total cost of ownership is also lowered making the object storage whole lot more profitable to the end users.