Monday, June 29, 2020

StreamManager enhancements

The ability to append a stream to an existing stream is the same as copy operation to an empty stream. Therefore, comparisons between various usages of append, copy and stitching are left out of this discussion. 


There are a number of applications where one strategy might work better than another. This is left to the discretion of the callers with the minimal required from the stream store described here.  


File management functionality of archiving is another capability that is commonly used for export or import. Stream store has metadata file as well as data both of which can be persisted on tier 2 and imported as exported. 


This functionality is not native to the stream store. Yet high level primitives on stream manager interface alleviates the concern from out of band access and possible tampering. It also makes programmability easier to work with. 


Until now, stream store has largely relied on open, close, read and write functionalities primarily as it tries to become at par with the demands of the stream processing language but these are mostly for saving and analysis purposes leaving much of the data transfer in the hands of the administrator.  


The data import or export is certainly something that belongs outside the analytical applications for the most part but that does not leave it completely out of automations and particularly those that server data pipelines. Modern applications are increasingly targeting continuous data and streams serve very well in data pipelines with a little bit more enhancement to the programmability. 


The stream manager is also the right interface to add these capabilities required at the stream store level from these automation workflows. Most of the existing methods on that interface target streams and the operations taken on the stream as a whole and these automation workflows are served by equivalent commands on the same stream. The new capabilities can be added cleanly without any issues with the earlier releases. 


One of the advantages of providing this capability is that the applications can close the stream manager when they are done with its use allowing proper finalization in all cases. The stream manager interface already comes with the close method to facilitate this.  

No comments:

Post a Comment