Annotations

 #annotations 

Any key value can be used as annotations which helps to differentiate events. There are internal annotations and external annotations. The external ones come from customers and if it can be applied to any event, stream or scope, then they serve to improve the query operators used with the events.  

The internal annotations are stored in table segments which is designed to hold different key values. The data is stored independent from metadata using BTree data structures. This allows the data to be stored with or without any metadata. The metadata stored as key-values can be arbitrarily large and cannot pollute the data.  

The data structure for internal and external metadata is the same. Just the access is different. There is no access allowed to internal metadata because it must remain a source of truth. The external metadata can change but it helps to version the change.  

The metadata is injected using the same field extraction technique discussed as above. In this case, the annotations come from the user.  

The user provided metadata can be applied to different containers such as stream, scope and pipelines. There are several applications that can query this metadata such as reporting stacks, troubleshooting, and diagnostics, stamping and providing environmental information. Most of these applications are read-only stacks so they can quickly scale to any spatial or temporal elasticity.