It is customary to have specific log queries against the introspection log store within the cluster because they identify some of the common root cause analysis from product Support and Sustaining Engineering. Some of these queries have been mentioned in the attached document:  

https://1drv.ms/w/s!Ashlm-Nw-wnWtCFt6NmEo6HhZmWm 

It would be even more flexible to have support for more than one kind of log stores. For example, we could have a cluster internal logstore and a cluster external logstore. The external logstore could be a time-series store, a database or a web-accessible object store with each participating in different workflows with different usages. 

These log queries then become as useful as log specific d-scripts. The stores themselves may help with visualizations for these and other queries.  

Provisioning of stores for logs external to clusters is dependent on the service broker. This may be done during the cluster setup or when the logs become available. The logs usually have nearly a five year retention period although only the last 90 days are most heavily used in queries. The maintenance of the store is not a concern as long as it is hosted in a public cloud. 

The reliance on Kubernetes cluster only log storage does not compete with log services external to the cluster.  The cluster can be self-sufficient for limited logs while the cluster external services can provide a durable storage along with all the best practices of storage engineering. The cluster only log storage can leverage high availability and load balancing indigenous to the cluster while the cluster external services can divert log reading loads from the cluster itself. The cluster can reduce points of failure and facilitate capture of console sessions and other diagnostics actions taken on the cluster while the services external to the service cannot change the source of truth. The cluster specific log collection allows the ability to specify fluentd rules while the services outside the cluster have to rely on classification.