Application troubleshooting continued

Tolerating late events and out of order events

The runtime and the store both process events in the stream but consistency can best be enforced in the store. The runtime has no control over incoming events and these events can arrive late or out of order. It is easy to tell that the events are late or out of order by noting the time. All the events are supposed to have time which gives them order. The events on disk are serialized with segment start, epoch and offsets so they can be read without these symptoms. Every event read from a stream also informs the position from where that event was read. This enables a reader to retry reading that event or use it as the start offset to resume reading after a pause. The runtime assigns timestamps but does not know whether the time is advancing or whether there is no data when the event doesn’t come and similarly if there is another event coming with the same timestamp. Watermarks help with measuring progress in event time. As a processor advances its timestamp, it introduces a watermark for the downstream operators to process. In the case of distributed systems where an operator might get inputs from more than one streams, the watermark on the outgoing stream is determined from the minimum of the watermarks from the invoking streams. As the input streams update their event times, so does the operator. As long as the import and export are done by the same tool or operator, the order of the events can be maintained with the help of sequence number or timestamp either annotated to the events or giving a unique name to each event that indicates order.

The Pravega stream store has an interface for EventRead which gives us the position and EventPointer record. Their representations have segments, epoch and offset but they are not meant to be parsed to get these fields. When taken as an opaque representation of the position, it does not have any significance in transferring events across Pravega instances. That said, it is still possible to transfer events between instances in the same order as they are read with the help of some form of sequence notation, timestamp or watermark.