We were discussing data transfers in and out of Object Storage It is a limitless storage. It is used as-is in many deployments and scales to any workload with the help of monitoring over capacity. While it is easier to add a lot of disks when capacity runs low, it is not easy to reserve space for a high priority workload in preference to others under the same disk space. Even if the workload is just backup, there is no differentiation of one backup from another. Also, if the workloads are not supposed to be throttled, this can be an administrator only feature that enables them to classify storage pools for workloads. That said, the benefits of object storage are clear to all workloads which we enumerate below. 
Object storage is a zero-maintenance storage. There is no planning for capacity and elastic nature of its services may be taken for granted. The automation of asynchronous writes, flush to object storage and sync of data in Connectors to that in object storage is now self-contained and packaged into this Connectors. 
The cloud services are elastic, both the storage and the Connectors along with the compute necessary for these purposes can remain in the cloud enabling not just the same benefits to one application and client for an organization but to be universal across departments, organizations, applications, services and workloads. 
Object storage coupled with this Connectors is also suited to dynamically address the needs for client and application because the former may have been a global store but the latter is able to determine the frequency depending on the workload.  Different applications may tune the governing to its requirements. 
Performance increases dramatically when the resources are guaranteed rather than commissioning them on demand. This has been one of the arguments for Connectors in general. 
Such service is hard to mimic individually within each application or client. Moreover, optimizations only happen when the compute and storage are elastic where they can be studied, Connectors, and replayed independent of the applications. 
The move from tertiary to secondary storage is not a straightforward shift from NAS storage to object storage without some form of chores over time. A dedicated product likes this takes the concern out of the picture. 

Object storage has established itself as a “standard storage” in the enterprise and cloud. As it brings many of the storage best practice to provide durability, scalability, availability and low cost to its users, it could expand its role from being a storage layer to one that facilitates data transfer. We focus on the use cases of object storage so that the onerous tasks of using object storage can become part of it. Object storage then transforms from being a passive storage layer to one actively pulling data from different data sources simply by attaching connectors. The purpose of this document is to describe such connectors that facilitate intelligent automations of data transfers from data sources with minimal or no disruption to their operations.
File-systems have long been the destination to capture traffic and while file system has evolved to stretch over clusters and not just remote servers, it remains inadequate as a blob storage. The connectors enhance the use of object storage and do not compete with the usages of elastic file stores.
We describe the connectors not by the data source they serve but by the technology behind the connectors. We enumerate synchronous send and receive over protocols, asynchronous publisher-subscriber model, compressed data transfer, deduplicated data transfer and incremental rsync based backups to name a few.  We describe their advantages and disadvantages but do not provide a prescription to the retail world which allows them to be used on a case by case basis and with flexible customizations. In this sense, the connectors can be presented in the form of a library with the object storage.
While public cloud object storage solutions offer cloud based services such as Amazon’s Simple Notification Services and Azure Notification hubs, on-premise object storage has the ability to make the leap from standalone storage offering to a veritable solution integration packaging. Public-Cloud offer robust myriad features for their general-purpose cloud services while the connectors specialize in the usages of the object storage.
The data that flows into object storage is often pushed from various senders. That usage continues to be mainstream for the object storage. However, we add additional usages where the connectors pull the data from different data sources. Previously there were in-house scripts for such data transfers. Instead we make it part of the standard storage and provide users just the ability to configure it to their purpose.
The ability to take the onerous routines of using object storage as a storage layer from the layer above across different data sources enables a thinner upper layer and more convenience to the end user. The customizations in the upper layer are reduced and the value additions bubble up the stack.
On-premise object storage is no more a standalone. The public cloud has moved towards embracing on-premise compute resources and their management via System Center integration. The same applies to on-premise object storage as well.  Consequently, the technologies behind the connectors are not only transparent but they are also setup for being monitored and reported via dashboards and charts This improves visibility into the data transfers while enabling cost accounting.
An Object Storage offers better features and cost management, as it continues to stand out against most competitors in the unstructured storage. The connectors lower the costs of usage so that the total cost of ownership is also lowered making the object storage whole lot more profitable to the end users.

Today we continue discussing the best practice from storage engineering :

300) Disks compete not only with other disks but also with other forms of storage such as Solid-State Drives. Consequently, disks tend to become cheaper, capable and smarter in their operations with added value in emerging and traditional usages. Cloud Storage costs have been said to follow a trend that asymptotically reaches zero with current price today at about 1c per GigaByte per month for cold storage. The average cost per GigaByte per drive has come down by half from 4c per GigaByte between 2013 and 2018.

301) Solid State Drives are considered as replacements for memory, L1 and L2 cache with added benefits. This is not really the case. It is a true storage even if it does wear out. Consequently, programming should be more mindful of the reads and writes to data and if they are random, store those data structures on the SSD drives.

302) The use of sequential data structures is very common in storage engineering. While some components go to great length to make their reads and writes access sequential, other components may simplify their design by storing on SSD.

303) Reads and writes are aligned on page size on Solid State Drives while erasures are on a block level. Consequently, data organized in data structures can leverage these criteria for reading some or all at once. If we are writing less than a page more frequently, we are not making good usage of the SSD. We can use buffering to aggregate writes.

304) The internal caching and readahead mechanism in SSD controller prefer long continuous reads and writes rather than simultaneous multiple reads and writes and performs them in one large chunk. This means we open up iterations and aggregate reads and writes to do them all together

305) Random writes perform just as well as sequential writes on SSD as long as the data are comparable. If the data size is small and the random writes are numerous, it may affect performance.

Object storage has established itself as a “standard storage” in the enterprise and cloud. As it brings many of the storage best practice to provide durability, scalability, availability and low cost to its users, it could draw further inspiration and become more differentiated in its usages.  The move towards storage classes in the public cloud that differentiate frequent and infrequent access of data is a step in this direction. When data backups are merely for digital preservation, their costs can be further reduced by drawing inspiration from cold storage systems such as Pelican. 

While public cloud object storage solutions are making progress, on-premise solutions have yet to catch up on these trends. Also, the use cases of on-premise storage are different from cloud storage. However, there is no denying that object storage- both on-premise and cloud are often used for frequent access by workloads as well as infrequent access that are backup oriented. As data ages and it is less frequently used, there are many gains to be made in terms of cost and efficiency such as with de-duplication and lowering power consumption.  

Pelican aims to be better than its overprovisioned counterpart racks using computations in software stacks. Pelican is not merely a power consumption optimization system. Pelican provides both a lower capital cost per disk as well as a lower running cost while previous work improved the latter only. Similarly, deduplication appliances can reduce the archival cost because it is performed only once with massive gains in disk usage. Together these kinds of savings further improve the appeal for object storage as it aims to differentiate usages to serve them best. 

Pelican is an extreme example for a storage class. It differentiates from the standard storage class very well in that the writes are scheduled and the power consumption is reduced. Workloads that use this tier are well known.  

However most of the workloads fall somewhere between these extremes. It is difficult for users to decide which storage class their workload should be assigned. In such cases, it may be better to assign the workload to an intermediate, which we can call the “intelligent storage class” and use it for determining whether a workload should be assigned to a different class.

Today we continue discussing the best practice from storage engineering:

295) A RAID level 4 uses a block inter-leaved parity with a striping unit of a disk block. Block-level striping has the advantage that read requests the size of a disk block can be served entirely by the disk where the requested block resides.  The write of a single block still requires a read-modify-write cycle, but only one data disk and the check disk are involved and the difference between the old data block and the new data block is noted

296)
A RAID level 5 uses a block inter-leaved distributed parity. The parity blocks are distributed uniformly over all disks, instead of sorting them on a single check disk. This has two advantages. First, several write requests potentially be processed in parallel, since the bottleneck of a unique check is removed. Second, read requests have a higher degree of parallelism. This level usually has the best performance.

297) A RAID level 6 uses P+Q redundancy.  Recovery from the failure of a single disk is usually not sufficient in very large disk arrays. First, a second disk might fail before replacement and second the probability of a second disk failing is not negligible. A RAID level 6 system uses Reed-Solomon codes to be able to recover from two simultaneous disk failures.

298) A RAID level 10+0 is a stripe of RAID1+0. In this array of RAID1+0, the RAID0 is implemented in software while RAID1+0 is implemented in hardware.

299) A RAID1+0, RAID3+0, RAID5+0, RAID6+0 and RAID10+0 are referred to as nested RAIDs and hybrid RAIDs and their usage diminishes with the growing and now established popularity of Network Attached Storage and cluster based partitioned approach. Blade servers with consolidated storage and centralized management have facilitated cutting VM slices.

300) Disks compete not only with other disks but also with other forms of storage such as Solid-State Drives. 

Today we continue our discussion on storage engineering


291) A RAID level 0 uses data striping to increase maximum bandwidth available. No redundant information is maintained

292) A RAID level 1 uses two copies of the data. This type of redundancy is often called mirroring There can be combinations of level 0 and 1 where like in level 1, read requests can be scheduled to both the disk and its mirror image and bandwidth for contiguous blocks is improved from the aggregation of all the disks.

293) A RAID level 2 uses a striping unit as a single bit.  The number of check disks grows logarithmically with the number of data disks.

294) A RAID level 3 uses a bit inter-leaved parity where it keeps more redundant information than is necessary. Instead of using several disks to store hamming code that informs which disk has failed, we rely on that information from the disk controller and use a single check disk with parity information which is the lowest overhead possible.

295) A RAID level 4 uses a block inter-leaved parity with a striping unit of a disk block. Block-level striping has the advantage that read requests the size of a disk block can be served entirely by the disk where the requested block resides.  The write of a single block  still requires a read-modify-write cycle, but only one data disk  and the check disk are involved and the difference between the old data block and the new data block is noted

296) A RAID level 5 uses a block inter-leaved distributed parity. The parity blocks are distributed uniformly over all disks,  instead of sorting them on a single check disk. This has two advantages. First, several write requests  potentially be processed in parallel, since the bottleneck of a unique check is removed. Second, read requests have a higher degree of parallelism. This level usually has the best performance.

Today we continue discussing the best practice from storage engineering:

286) File Systems may implement byte range locking to enable concurrent access. Typically, they are not supported by File mapping operation. Poor use of file locks can result in performance issues or deadlock.

287) Disks are potential bottlenecks for system performance and storage system reliability. If the disk fails, the data is lost. A disk array is used to increase performance and reliability through data striping and redundancy.

288) Instead of having a single copy of data, redundant information is maintained and carefully organized so that in the case of a disk failure, it can be used to reconstruct the contents of the failed disk. These redundant array of independent disk organizations are referred as RAID levels and each level represents a tradeoff between reliability and performance.

289) In Data Striping, the data is segmented into equal-size partitions that are distributed over multiple disks. The size of the partition is called the striping unit. The partitions are usually distributed using a round robin mechanism.

290) In Redundancy, if the mean time to failure of a single disk is about a few years, it is smaller for a disk array. Hence check disks and parity schemes are involved to improve reliability. The check disk contains information that can be used to recover from failure of any one disk in the array. This group of data disks and check disks together constitute reliability groups.

#codingexercise
List <string> getDiskGroups (List <String> disks) {
return disks.stream ().map (x -> getGroup (x)).distinct ().collect (Collectors.toList ());
}