Wednesday, July 15, 2020

Reader Group Notifications continued

The following section describes the set of leads that can be drawn for investigations into notifications. First, the proper methods for determining the data with which the notifications are made, needs to be called correctly. The number of segments is a valid datapoint. It can come from many sources. The streamManager is authoritative but the code that works with the segments might be at a lower level to make use of the streamManager. The appropriate methods to open the stream for finding the number of segments may be sufficient. If that is a costly operation, then the segments can be retrieved from say the readergroup but those secondary data structures must also be kept in sync with the source of truth. 

Second the methods need to be called as frequently as the notifications that are generated otherwise the changes will not be detected. The code that generates the notifications is usually offline as compared to the online usage of the stream. This requires the data to be fetched as required for generating these notifications. 

Lastly, the notification system should not introduce unnecessary delay in sending the notifications. If there is a delay or a timeout, it becomes the same as not receiving the notification.

The ReaderGroupState is initialized at the time of the creation of reader group. It could additionally be initialized/updated at the time of reader creation. Both the reader Group and reader require the state synchronizer to be initialized so the synchronizer can be initialized with the new group state. This is not implemented in the Pravega source code yet. 
During the initialization of the ReaderGroupState the segments are fetched from the controller and corresponds to the segments at the current time. The controller also provides the ability to get all segments between start and end tailcuts. However those segments are a range not a map as desired.
The number of Readers comes from the reader group. These data will remain the same on cross thread calls and invocations from different levels of the stack. The readerGroup state is expected to be accurate for the purpose of sending notifications but it can still be inaccurate if the state is initialed once and only updated subsequently without refreshing the state via initialization each time.

No comments:

Post a Comment