Reader Group Notifications

Reader Group Notification. 

The stream store publishes notifications for segments that come to the aid of the user in determining whether to scale up or down the readers in a reader group. 

The notifications are usually based on their type. For example, a well-known stream store publishes a SegmentNotification and an EndOfDataNotification when all the streams managed by a reader group have been read. The notifications are posted to the reader group so that the user can take action associated with the entire reader group. Over a period of time, these internal streams that store revisions to notification become useful for introspection queries. 

Notifications go to end users and applications. The analysis they draw from this data would be similar to what the introspection query would demonstrate. The persistence of events and notifications outbound to subscribers helps with being able to replay the notifications and the query.

The notifications are purely a client-side concept because a periodic polling agent that watches for the number of segments and the number of readers can raise this notification. 

There may be questions on why such information on segments, streams and scopes need to come from notifications as opposed to metrics which are suited for charts and graphs via existing time-series database and reporting stack. The answer is quite simple. The stream store is a time-series store and should promote its own stack with the beautiful stream processing language and store that it supports. All the notifications are events and those events are also as continuous as the data that generates them. They can be persisted in the stream store itself.

When the events are persisted in the stream itself, the publisher-subscriber interface then becomes similar to writer-reader that the store already supports. The stack that analyzes and reports the data can read directly from the stream. Should this stream be internal, a publisher-subscriber interface would be helpful. The ability to keep the notification stream internal enables the store to cleanup as necessary. Persistence of events also helps with offline validation and introspection for assistance with product support.

The writes are done by the synchronizer with the help of a revisioned-stream client and it can synchronize the revisions made from across the processes. The synchronizer and therefore its client can be one per reader even as it synchronizes the state change per reader.  The call out here is that there are three stages to smooth out the notification process. First, is the detection of every state change such as the changes in the number of readers and the number of segments from all sources. For example, segment changes can come from both segment sealing and auto-scaling.  The changes in the readers can be detected if the reader group is kept notified. Second, each and every change should be serialized to the revisioned-stream. If the appends to the stream are skipped because the revision turns to be not as new as the most recent revision, then all the detection will still lead to lossy serialization. Third, the revisioned streams should be polled and read by notifier frequently so that all the events are sent to the notification system. If any of the events are skipped and the application is not informed, the notifications will not be accurate in their sequence. Each notification and the sequence of notifications are both useful to the application.