This is a continuation of the earlier posts starting with this one: http://ravinote.blogspot.com/2020/09/best-practice-from-networking.html
The tasks that run long need to indicate progress in some way so that the scheduler knows that this task is still active and not stuck.
When the tasks must sort the results, the sorting order might change as the listing changes. It is helpful to refresh the listing before sorting.
If the listing is large, it is not easy to refresh without taking a cost on the overall query time. In such cases, it helps to have progressive listing where the changes are made to one end of the listing while the other end remains as is. As the listings are added to the tail, the stats from unchanged can be reused for the new entries.
If the listing is distributed, it helps to have a map-reduce on the listing.
If the entries vary widely affecting the overall results at a high rate, it is easier to take on the changes on the compute side but allow the storage of the listings to be progressive. This way tasks can communicate the changes to their respective sortings to the scheduler which can then adjust the overall sort order
If the listing is a stream, processing on a stream works the same was as cursor on a database adjusting the rankings gathered so far for each entry as they are encountered.
No comments:
Post a Comment