Archival using streams discussion is continued:
Traditionally, blob storage has developed benefits for archival:
1) Backups tend to be binary. Blobs can be large and binary
2) Backups and archival platforms can take data over protocols and blobs can be transferred natively or via protocols
3) Cold storage class of object storage is well suited for archival platforms
Stream stores allows one steam to be converted to another and do away with class.
There cold, warm and hot regions of the stream perform the equivalent of class.
Their treatment can also be based on policies just like the use of storage class.
The rules for a storage class need not be mere regex translation of outbound destination to another site-specific address. We are dealing with stream transformations and conversions to another stream. The cold warm and hot regions need not exist in the same stream all the time. They can be written to the own independent streams before being processed. Also the processing policy can be different for each and written in the form of a program We are not just putting the streams on steroids, we are also making it smarter by allowing the administrator to customize the policies. These policies can be authored in the form of expressions and statements much like a program with lots of if then conditions ordered by their execution sequence. The source of truth remains unchanged while data is copied to streams where they can be better suited for all extract transform and load operations on streams. The
data transformation routines that can be offloaded to a compute outside the storage, if necessary, to transform the data prior to archival. The idea here is to package a common technique across data sources that handle their own archival preparations across data streams. All in all it becomes an archival management system rather than remain a store.
Traditionally, blob storage has developed benefits for archival:
1) Backups tend to be binary. Blobs can be large and binary
2) Backups and archival platforms can take data over protocols and blobs can be transferred natively or via protocols
3) Cold storage class of object storage is well suited for archival platforms
Stream stores allows one steam to be converted to another and do away with class.
There cold, warm and hot regions of the stream perform the equivalent of class.
Their treatment can also be based on policies just like the use of storage class.
The rules for a storage class need not be mere regex translation of outbound destination to another site-specific address. We are dealing with stream transformations and conversions to another stream. The cold warm and hot regions need not exist in the same stream all the time. They can be written to the own independent streams before being processed. Also the processing policy can be different for each and written in the form of a program We are not just putting the streams on steroids, we are also making it smarter by allowing the administrator to customize the policies. These policies can be authored in the form of expressions and statements much like a program with lots of if then conditions ordered by their execution sequence. The source of truth remains unchanged while data is copied to streams where they can be better suited for all extract transform and load operations on streams. The
data transformation routines that can be offloaded to a compute outside the storage, if necessary, to transform the data prior to archival. The idea here is to package a common technique across data sources that handle their own archival preparations across data streams. All in all it becomes an archival management system rather than remain a store.
No comments:
Post a Comment