Logging Sidecar:
AcmeApplication is hosted on Kubernetes and logging throughout the
product can take advantage of the rich features of the runtime and its
plugins. This document justifies the need to support logging including
mechanisms to do it regardless of whether pods, jobs, batches or
containers are being recycled.
AcmeApplication integrated analytics over storage. As such, all the
deployers and the running instances of components log to a file which
is automatically rolled after predefined limits. This works well when
the file is accessible at all times and can be used in queries such as
log stream queries.
However, virtually every K8s resource can have varying lifetime and
therefore the default logging framework applied logging at a cluster
scope. AcmeApplication does not yet take advantage of an application
block pattern for all the applications because the applications are
fundamentally different and proprietary in their logging and format.
The default Kubernetes framework can tap into the files regardless of
origin and send the logs to a common sink that is singleton
cluster-wide. This is yet to be turned on. Metrics, InfluxDB and
Grafana are some of the other applications that have their own logs
and there is nothing unifying the logs and their namespaces into a
common store for analysis
Some products such as log indexes can be configured for rich analytics
via their search language that has been acknowledged in the industry
as best for machine generated data. The integration of a log index
store with AcmeApplication is a business decision and not a
technological decision. Popular use cases for logging and queries over
logs involve significant cost reduction in production support,
troubleshooting and sustaining engineering.
Whether an index store collects the logs from all the applications via
the connectors or whether a syslog drain is setup for consolidation is
merely a choice depending on the use case. The technologies used may
vary from use case to use case but in this document, we suggest a
system that is architecturally sound
It has not yet been proven to be helpful to have stream queries over
logs as much as regular interactive queries over the user interface on
all the logs from different applications that is stored in designated
indexes. Consequently, there are various levels and areas of emphasis
in the stack to support rich analytics over logging.
At the Kubernetes system level, among the popular options, we could
prioritize fluentD logging sidecar as a way to collect logs from
across the pods. This will be beneficial to the applications as it
alleviates the concerns from them and at the same time provides a
consistent model across the applications.
At the store level, we recommend that the logs have their own storage
whether it is the file storage or an index store. This will allow
uniform aging, archiving and rolling of logs based on timeline and
will work across all origins
At the analytics level, we can outsource the reporting stack, charts,
graphs and dashboards to the application that is best to analyze
machine data.
Although StorageApplication, SmartAnalytics, metrics, charts and
graphs can be used to store, search and render results, it will take a
longer round-trip time between the user query and the results.
Therefore, dogfooding of the StorageApplication store for logging will
not be helpful at this time. But perhaps, this could be considered in
subsequent versions.