Support for small to large footprint introspection database and query

 Distributed Collection agents

As with any store not just introspection store, data can come from different sources for the destination as the store. Collection agents for each type of source make it convenient for the data to be transferred to the store  

The collection agents do not themselves need to be monitored. The data they send can be lossy but it should arrive at the store. The store is considered a singleton local instance. It may not even be in the same system as the one it serves. The rest of the store may be global and shared but the data transfer from collection agent does not have to go directly to the global shared storage. If it helps to have the introspection store serve as the same local destination for all collection agents, the introspection store can be kept outside the global storage. In either case the streams are managed centrally by the stream store and the storage refers to tier 2 persistence.

Distributed Destinations:

Depending on the mode of deployment, the collection agents can be either lean or bulky. In the latter case, they come configured with their own storage so that all the events are batched under the resource restriction of the site where the agent is deployed. Those batched events can then be periodically pushed to the introspection store. This is rather useful when certain components of the system don’t even share the same cluster or host on which the streamstore is deployed. The collection agents are usually as close to the data source as possible so the design to keep them going regardless of whether the rest of the system is reachable or not is prudent given that certain sites might even be dark. Under such circumstance, the ability to propagate events collected remotely for introspection of data collection agents will be very helpful for administrators to use as and when they like.

Encryption

The operational data from any system are like logs and metrics. They don’t necessarily have any sensitive data as long as the producers are not publishing any data, there is very little effort required to secure the introspection store. But it is also important to ensure that the introspection data remains secure even when it is exported from the system. Key-certificates can help with encrypting the data so that they can be secured at rest and in transit when accessed offline to the system. In all the above examples, we have referred to the introspection store as local to the system. But if the introspection store were to be distributed and the data was propagating across the network, it would be helpful to secure the transit.  Even exports can be encrypted which can be incremental or full depending on the rsync algorithm setting to publish the changes to the store. The export of the data occurs in this case only when the system is online but frequently exporting the data whenever possible help keep consolidate data in a dedicated instance so that introspection store from all instances can then be collected as if it were coming to a network operations center.