The benefits of Async stream readers in cache for stream storage:
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments
Another approach is to have a continuous reader read the stream from beginning to end in a loop and have the cache decide which of the segments it needs to read. This approach requires the cache to discard segments that are not necessary for the clients. The cache can fork as many readers as necessary to replenish the window of segments it needs. The cache can decide which windows if segments it needs based on overlapping intervals of segments requested by clients. These segments are no longer bound to clients and they are managed exclusively by the cache.
The skip level access continues to serve the readers in this case although it doesn’t need to read the streams directly. The readers present the segments and the cache then uses its statistics and algorithm to prefetch the segments that will serve the clients best. These cache entries can be evicted by any algorithm that suits the cache. Storage is not a concern for the cache because we have distributed ring cache with consistency check points
The maximum number of segments in a window is directly dependent on the cache. The segment size may even be a constant in which case the number of segments depends on how many streams and how many windows per stream the cache wants to accommodate. This window size is not the same as a continuously sliding window used to analyze the stream which is usually one per stream.
When there is only one reader to replenish the segments for the cache, the clients are freed from performing the same task over and over again on the actual stream. The segments that the client need will most likely be found in the cache. If the segments are not found, the cache can fetch it from the stream on demand. Otherwise the scheduled reading from the stream by the cache will replenish it. The cache itself alleviates the load from the stream store.
The cache can also fork multiple readers at fixed duration offsets from the start of the first. Multiple readers at fixed periodic intervals from the start of the first will continue to read the steam segments and reduce the latency in replenishing the segments that the cache needs. When the cache advertises it needs segment number n, the readers that have not yet read n will read and make it available. With multiple readers, the chance that one of them is closest to segment n is higher.
There are ways in which the cache can become smart about loading segments using these readers. For example, instead of waiting for the nearest reader to the segment to read it into the cache, the cache can have a dedicated reader fetch the segment or a set of segments via skip-level access.
There are also techniques to make read and writes to go together. Typically reads and writes are not in the same stream and the cache behaves as a write-through, hence the clients can interact directly with the cache rather than the store. When a segment is not found in the cache, it is also efficiently handled with skip level access.
When readers read a stream from beginning to end, they become easy to scale. This is simple strategy and works well for parallelizing computations. However, it introduces latency as streams can be quite long.
A cache served to provide the stream segments do that clients don’t have to go all the way to the store. It could employ a group of readers that could replenish the segments that are in most demand. There is always the option for clients to reach the store if the cache does not have what it needs. Generally s cache will bring in the segment on behalf of the client if it goes not have it.
The techniques for providing stream segments do not matter to the client and the cache can use any algorithm. The cache also provides the benefits of alleviating load from the stream store without any additional constraints. In fact the cache will also use the same stream reader as the client and with the only difference that there will be fewer stream readers on the stream store than before.
We have not compared this cache layer with a message queue server but there are interesting problems common to both. For example, we have a multiple consumer single producer pattern in the periodic reads from the stream storage. The message queue server or broker enables this kind of publisher-subscriber pattern with retries and dead letter queue. In addition, it journals the messages for review later. Messaging protocols are taken up a notch in performance with the use of a message queue broker and their reliance on sockets with steroids. This leaves the interaction between the caches and the storage to be handled elegantly with well-known messaging framework. The message broker inherently comes with a scheduler to perform repeated tasks across publishers. Hence it is easy for the message queue server to perform as an orchestrator between the cache and the storage, leaving the cache to focus exclusively on the cache strategy suitable to the workloads. Journaling of messages also helps with diagnosis and replay and probably there is no better store for these messages than the object storage itself. Since the broker operates in a cluster mode, it can scale to as many caches as available. Moreover, the journaling is not necessarily available with all the messaging protocols which counts as one of the advantages of using a message broker. Aside from the queues dedicated to handle the backup of objects from cache to storage, the message broker is also uniquely positioned to provide differentiated treatment to the queues. This introduction of quality of service levels expands the ability of the solution to meet varying and extreme workloads The message queue server is not only a nice to have feature but also a necessity and a convenience when we have a distributed cache to work with the stream storage.
The number of workers for the cache or the store does not matter and they can scale.
Please refer to the discussion on caching context for stream managers
https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp
and sample implementation with: https://github.com/ravibeta/JavaSamples/tree/master/logging
The cache essentially projects portions of the stream so that the entire stream does not have to scanned from begin to end repeatedly. This mode of operation is different from finding popular stream segments across stream readers.
The cache may use a queue to hold contiguous segments from a stream.
The choice of queue is extremely important for peak throughput. The use of a lock free data structure as opposed to an ArrayBlockingQueue can do away with lock contention. The queues work satisfactorily until they are full. Garbage free Async stream readers have the best response time.
The benefits here are similar to LogAppenders except that they are on write path whole these are in read path.
Asynchronous read together with lock free access and skip level access boost performance. The cache may encounter significant size for each segment and a writer that transfers a segment over the network may take time in the order of hundreds of milliseconds. Instead, having a continuous background import for adjustable window of steam segments tremendously improves the load on the stream store while responding to seek requests on segments
Another approach is to have a continuous reader read the stream from beginning to end in a loop and have the cache decide which of the segments it needs to read. This approach requires the cache to discard segments that are not necessary for the clients. The cache can fork as many readers as necessary to replenish the window of segments it needs. The cache can decide which windows if segments it needs based on overlapping intervals of segments requested by clients. These segments are no longer bound to clients and they are managed exclusively by the cache.
The skip level access continues to serve the readers in this case although it doesn’t need to read the streams directly. The readers present the segments and the cache then uses its statistics and algorithm to prefetch the segments that will serve the clients best. These cache entries can be evicted by any algorithm that suits the cache. Storage is not a concern for the cache because we have distributed ring cache with consistency check points
The maximum number of segments in a window is directly dependent on the cache. The segment size may even be a constant in which case the number of segments depends on how many streams and how many windows per stream the cache wants to accommodate. This window size is not the same as a continuously sliding window used to analyze the stream which is usually one per stream.
When there is only one reader to replenish the segments for the cache, the clients are freed from performing the same task over and over again on the actual stream. The segments that the client need will most likely be found in the cache. If the segments are not found, the cache can fetch it from the stream on demand. Otherwise the scheduled reading from the stream by the cache will replenish it. The cache itself alleviates the load from the stream store.
The cache can also fork multiple readers at fixed duration offsets from the start of the first. Multiple readers at fixed periodic intervals from the start of the first will continue to read the steam segments and reduce the latency in replenishing the segments that the cache needs. When the cache advertises it needs segment number n, the readers that have not yet read n will read and make it available. With multiple readers, the chance that one of them is closest to segment n is higher.
There are ways in which the cache can become smart about loading segments using these readers. For example, instead of waiting for the nearest reader to the segment to read it into the cache, the cache can have a dedicated reader fetch the segment or a set of segments via skip-level access.
There are also techniques to make read and writes to go together. Typically reads and writes are not in the same stream and the cache behaves as a write-through, hence the clients can interact directly with the cache rather than the store. When a segment is not found in the cache, it is also efficiently handled with skip level access.
When readers read a stream from beginning to end, they become easy to scale. This is simple strategy and works well for parallelizing computations. However, it introduces latency as streams can be quite long.
A cache served to provide the stream segments do that clients don’t have to go all the way to the store. It could employ a group of readers that could replenish the segments that are in most demand. There is always the option for clients to reach the store if the cache does not have what it needs. Generally s cache will bring in the segment on behalf of the client if it goes not have it.
The techniques for providing stream segments do not matter to the client and the cache can use any algorithm. The cache also provides the benefits of alleviating load from the stream store without any additional constraints. In fact the cache will also use the same stream reader as the client and with the only difference that there will be fewer stream readers on the stream store than before.
We have not compared this cache layer with a message queue server but there are interesting problems common to both. For example, we have a multiple consumer single producer pattern in the periodic reads from the stream storage. The message queue server or broker enables this kind of publisher-subscriber pattern with retries and dead letter queue. In addition, it journals the messages for review later. Messaging protocols are taken up a notch in performance with the use of a message queue broker and their reliance on sockets with steroids. This leaves the interaction between the caches and the storage to be handled elegantly with well-known messaging framework. The message broker inherently comes with a scheduler to perform repeated tasks across publishers. Hence it is easy for the message queue server to perform as an orchestrator between the cache and the storage, leaving the cache to focus exclusively on the cache strategy suitable to the workloads. Journaling of messages also helps with diagnosis and replay and probably there is no better store for these messages than the object storage itself. Since the broker operates in a cluster mode, it can scale to as many caches as available. Moreover, the journaling is not necessarily available with all the messaging protocols which counts as one of the advantages of using a message broker. Aside from the queues dedicated to handle the backup of objects from cache to storage, the message broker is also uniquely positioned to provide differentiated treatment to the queues. This introduction of quality of service levels expands the ability of the solution to meet varying and extreme workloads The message queue server is not only a nice to have feature but also a necessity and a convenience when we have a distributed cache to work with the stream storage.
The number of workers for the cache or the store does not matter and they can scale.
Please refer to the discussion on caching context for stream managers
https://1drv.ms/w/s!Ashlm-Nw-wnWvBXkU9jz_Z2EXWLp
and sample implementation with: https://github.com/ravibeta/JavaSamples/tree/master/logging
No comments:
Post a Comment