Tuesday, July 14, 2020

Reader Group notifications continued.

One of the best ways to detect all reader group segment changes is to check the number of segments at the time of admission of readers into the reader group.  

This would involve: 

Map<SegmentWithRange, Long> segments = ReaderGroupImpl.getSegmentsForStreams(controller, config); 

synchronizer.initialize(new ReaderGroupState.ReaderGroupStateInit(config, segments, getEndSegmentsForStreams(config))); 

Where the segments are read from the ReaderGroup which knows the segment map and the distribution of readers rather than getting the segments from the state. 

The following section describes the set of leads that can be drawn for investigations into notifications. First, the proper methods for determining the data with which the notifications are made, needs to be called correctly. The number of segments is a valid datapoint. It can come from many sources. The streamManager is authoritative but the code that works with the segments might be at a lower level to make use of the streamManager. The appropriate methods to open the stream for finding the number of segments may be sufficient. If that is a costly operation, then the segments can be retrieved from say the readergroup but those secondary data structures must also be kept in sync with the source of truth. 

Second the methods need to be called as frequently as the notifications that are generated otherwise the changes will not be detected. The code that generates the notifications is usually offline as compared to the online usage of the stream. This requires the data to be fetched as required for generating these notifications. 

Lastly, the notification system should not introduce unnecessary delay in sending the notifications. If there is a delay or a timeout, it becomes the same as not receiving the notification. 

No comments:

Post a Comment