conflation

 #conflation 

Streams can be conflated into an uber stream. This is helpful in devOps cases where streams are created by different application but need to be merged to a stream that can be truncated or appended by a global policy and useful for downstream systems.   

There can be thousands of streams to be merged and the incoming data rate for these streams could be different. The events in the stream can be compared based on timestamp and the earlier event can follow another.  If the events are distributed by key space, each key at a point of time spanning the key space can be considered to be in an independent stream but will retain the naming convention using the routing key. Since the routing key differentiates the events, the order of events with same timestamp does not matter.  

The controller is the right place to include an api to merge the streams since that is the interface to create the streams. The streams can be referenced by their names that are qualified by their scopes.  

The interface can also support a copy stream method that allows a stream to be copied. In this case, the copy saves the client from having to do so one event at a time. Instead, the entire collection of events, their sequence and offsets can be copied along with the metadata and with renamings. 

Streams can also be conflated to an external store. For example, we can overlay a stream store on top of object storage and have all the streams written to the same bucket. This is elaborated in https://1drv.ms/w/s!Ashlm-Nw-wnWvj_oHPVynmrD-SVI?e=k8EaG8