Sunday, July 5, 2020

Stream Store Notifications: 


A stream store is elastic, unbounded and continuous data storage. Events arrive in a stream to the stream store and are serialized by time. A partitioned collection of events that span a start time t0 and end time t1 is called a segment. There can be more than one segment between t0 and t1 to allow parallel read and write. The multiple segments are spread out over a key space determined by the use of routing keys used to write the events. Segments do not overlap in both horizontal and vertical directions and need not align on the same time boundaries. Each segment can be started and sealed independently so that no more events are appended to it. The number of segments for a stream depends on the scaling policy specified at the time of stream creation. A scaling policy can be fixed, data-based or event-based to accommodate varying size and count of segments according to anticipated load. 


This logical representation of a stream is hard for a user to visualize when the data is being read while it is arriving by the order of millions from a device.  The user has the guarantee that the events will be written in order of their arrival and none of the events will be lost during write and subsequent read but the user does not know the progress made in terms of reading the segments or if all the data has been read. If the user knew the progress, the readers could be scaled accordingly to keep up or complete reading at a faster rate. 


The stream store publishes notifications for segments that come to the aid of the user in this regard. This article describes these notifications and their usage with potential improvements in the future.   


The notifications are usually based on their type. For example, a well-known stream store publishes a SegmentNotification and an EndOfDataNotification when all the streams managed by a reader group have been read. The notifications are posted to the reader group so that the user can take action associated with the entire reader group 

No comments:

Post a Comment