Introspection store discussion addendum

 The set of policies that can determine the injection of fields into the introspection stream can be maintained in a module just like the analytical application itself. The only difference is that the Flink application is executed higher up the stack in the analysis wing of the platform while the execution of policies is handled within the stream store. The stream store can host a lightweight runtime for the evaluation of rules that can be written in program order. 

Another alternative is for the rules to be simpler and involve simple recognizable predicates for boolean evaluation in strict program order. In such cases, the need for an expression tree is obviated. A set of nested if conditions can be flattened into several sequential single-level if conditions after they include repeated conditions from the outer levels. This is easier to pipeline once the statements are singular and isolated from the other statements and they occur in a declaration.

The reconfiguration of the stream store to annotate or inject fields or events into the stream in the introspection store should be done at the startup or without the requiring the store to go offline visibly.  If the store does need to shutdown prior to reconfiguration, the shutdown and startup can be done on a rolling basis through the nodes of the cluster hosting the store so that the client demands can be continued to be met.

Once all the nodes of the cluster have gone through a rolling restart, the new policies can then kick in. This will involve the events to be grouped or tagged differently. This works even for immutable events by way of introducing delimiter events that bound the set of immutable events implying that they are grouped together. Thus, events with delimiters can form groups and these groups can represent hierarchical levels.

There are ways to provide different delimiters so that a level could mean something different from another while these different delimiters perform the same way. The idea is that a delimiter can have any key/value or any number of them and they can be administrator specified via policies. 

The use of delimiters does not stop with policies. Administrator can choose to set a delimiter without policy evaluation or the system components can use their own delimiters. The inlining of delimiters provides a way to use the same stream while the annotations in a dedicated stream that use the sequence number of the events in the original stream require their own maintenance. The dedicated stream also requires adjustments to its events when the original stream is truncated so that the beginning and end events change and the sequence numbers also change.