Diagnosing keycloak connection refused error from a server hosted in the Kubernetes container
The following is a trail of steps down the path of an investigation that has proved time consuming and hard to resolve. Before we begin some introduction to the terminology may help explain the problem. These include the following:
A keycloak server is an identity management framework that has allows Independent identity providers to be configured
A Kubernetes framework is an orchestration framework that hosts application such as Keycloak and alleviates the automation efforts of servicing and scaling the resources used.
A container is a lightweight host for an I stance if the application and includes the operating system which is isolated for that application
A connection refused error comes from the keycloak server usually due to some configuration error
The diagnosis of the exact configuration error and location requires thorough investigation of all the aspects of the deployment pertaining to connectivity because this internal error could have ripple symptoms all the way to external boundary
The dns entry for the keycloak server should allow reaching the server by its name rather than by the cluster master ip address.
The IDENTITY_SERVER_URL must be configured properly with the scheme, address, port, path and qualifier.
The ingress resource for the service must respond be properly configured to translate the external connection with the internal connection to the backend. An improper connection will promptly result in “default backend 404” error. In the case of Keycloak the external facing host and port for the ingress maybe set to keycloak.myrealm.cluster.local and the port is 80 for http and 443 for SSL. The backend is usually available at port 8080 for Keycloak server.
The application server itself has a lot of settings that can be configured with application.properties, configuration.xml, and command line options. In addition settings can also be defined via passing them to the keycloak via proprietary files that have a “.cli” extension.
The command line can include a 0.0.0.0 option to bind the application to all available network interface as opposed to the localhost ip address of 127.0.0.1
The jboss.bind.address.management can also include this address instead of localhost. These settings do not need to be in conflict.
The security context of the container does not need to be elevated for connect to work and they can allow the keycloak server to run as jboss user
The application.properties can include a role setting such as follows:
keycloak.securityConstraints[0].authRoles[0]=* keycloak.securityConstraints[0].securityCollections[0].patterns[0]=/*
The trouble encountered during diagnosis include settings that don’t get applied, settings getting reset, typo in settings or multiple sources for the same settings.

Auditing:
Audit events are some of the system generated events that are widely produced due to compliance from almost all software products. These include system software, orchestrator frameworks and user applications. As a source of data for both storage and analysis, audit events are interesting use-case. Charts and graphs for reporting as well as queries for diagnostics and troubleshooting are extremely helpful. It is therefore very popular and applied in a variety of usages.
The stream storage is appropriate for use with audit events. In terms of scalability, consistency, durability and performance, this storage handles not just the size but also the count for the events in the stream.

Audit serves to detect unwanted access and maintain compliance with regulatory agencies. Most storage services enable auditing by each and every component in the control path. This is very much like the logging for components. In addition, the application exposes a way to retrieve the audits.
The best way for a higher-level storage product to enforce spatial locality of the data is to store it directly on the raw device. However, this uses up the disk partitions and the interfaces are OS specific. Instead developments like RAID and SAN have made the virtual device more appealing. Consequently, storage product now accesses a single file and place these blocks directly in the file. The file is essentially treated as a linear array of disk-resident pagesh
These higher-level storage products will try to postpone or reorder writes and this may conflict with OS read-ahead and write behind approach. The write ahead logging is required by these products to provide durability and correctness

Example of streaming queries
    DataStream<StockPrice> socketStockStream = env
            .socketTextStream("localhost", 9999)
            .map(new MapFunction<String, StockPrice>() {
                private String[] tokens;

                @Override
                public StockPrice map(String value) throws Exception {
                    tokens = value.split(",");
                    return new StockPrice(tokens[0],
                        Double.parseDouble(tokens[1]));
                }
            });
    //Merge all stock streams together
    DataStream<StockPrice> stockStream = socketStockStream
        .merge(env.addSource(new StockSource("TCKR", 10)););

WindowedDataStream<StockPrice> windowedStream = stockStream
    .window(Time.of(10, TimeUnit.SECONDS))
    .every(Time.of(5, TimeUnit.SECONDS));

DataStream<StockPrice> maxByStock = windowedStream.groupBy("symbol")
    .maxBy("price").flatten();

 The window method call as a data-driven example could be:
DataStream<String> priceWarnings = stockStream.groupBy("symbol")
    .window(Delta.of(0.05, new DeltaFunction<StockPrice>() {
        @Override
        public double getDelta(StockPrice oldDataPoint, StockPrice newDataPoint) {
            return Math.abs(oldDataPoint.price - newDataPoint.price);
        }
    }, DEFAULT_STOCK_PRICE))
.mapWindow(new SendWarning()).flatten();

Even a stream from social media can be used for correlations:
DataStream<Double> rollingCorrelation = tweetsAndWarning
    .window(Time.of(30, TimeUnit.SECONDS))
    .mapWindow(new WindowCorrelation());

The application stack for stream analysis can independently scale the analysis and storage tiers to their own clusters. Clusters in this case is not just for high availability but a form of distributed processing for scale out purposes. Many traditional desktop centric applications are invested way high on scale up techniques when scale out processing when workloads have become smarter to narrow the gap between peak traffic and regular traffic.

Kubernetes provides a security relevant chronological set of records documenting the sequence of activities that have affected the system by individual users and actions take on their behalf by the system.
This feature is enabled with the following steps:
1) Audit policy – This is the main audit policy. We don’t specify it as a file with –audit-policy-file, rather we define the following:
a. Log. Metadata – this logs requests metadata but not the request or response itself. We define policies for all of Kubernetes core group resources.
audit/audit-policy.yaml:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: RequestResponse
    resources:
    - group: ""
    resources: ["pods"]
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: ""
      resources: ["endpoints", "services"]

  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*"
    - "/version"

  - level: Request
    resources:
    - group: ""
      resources: ["configmaps"]
    namespaces: ["kube-system"]

  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets", "configmaps"]

  - level: Request
    resources:
    - group: ""
    - group: "extensions"

  - level: Metadata
    omitStages:
      - "RequestReceived"

b. Audit Backends:  There are also auditing solutions like Falco that make it easy but here we define a Log backend and have the audit events route to the SRS gateway. This includes –audit-log-path for /var/log, --audit-log-maxage for 3 months , --audit-log-maxbackup for a number of 20 log files to be retained, and –audit-log-maxsize of 5MB each.
c. We have additional options specified to truncate logs with –audit-log-truncate-enabled
d. We use fluentd to collect and distribute audit events from log file
We install fluentd in the kube-apiserver node
We create a config file for fluentd
We start fluentd with
fluentd -c /etc/fluentd/config -vv
We start kube-apiserver with the following options:
--audit-policy-file=/etc/kubernetes/audit-policy.yaml –audit-log-path=/var/log/kube-audit and –audit-log-format=json
We check audit for different namespaces in /var/log/audit-*.log

Container Monitoring Solution:
We describe a Log Analytics workspace that helps with monitoring containers hosted on an orchestration framework. This article is intended to be generic so that the salient features of a container monitoring solution are called out rather than the specifics of a container. This article therefore avoids discussion of other overlapping production level considerations such as specific security context capabilities where certain specific actions such as binding to all network interfaces are permitted without giving full root access. The purpose of this article is to focus exclusively on a container monitoring solution without any restrictions.
Public clouds have made system operations monitoring a breeze with their proprietary and even altruistic solutions that work across containers from one another.  They begin with a Log Analytics agent and integrate with their externally offered cloud services. The techniques for gathering the logs by these agents are also dependent on the operating system flavor where the orchestrator engine is hosted. These hosts are even given specific global identifiers to enable their lookup in the corresponding global inventory.
As with all agents and services on a container, a secret or an account is required to control the accesses to resources for their activities. This role based access control, namespace and global naming conventions is a prerequisite for any agent.
The agent has to run continuously forwarding data with little or no disruption. Some orchestrators facilitate this with the help of concepts similar to a Daemonset that run endlessly. The deployment is verified to be working correctly when a standard command produces an output same as a pre-defined output. Verification of monitoring capabilities becomes part of the installation feature.
The log analytics agent needs to have a controller or gateway to send the report. It can work autonomously with little or no context provided from the controller but the data needs to make its way to a store or service. This is facilitated by binding the agent to its destination.
Almost all activities of any resource in the orchestrator framework can be logged. These include the core system resources which may spew the data via switches in the auditing framework. The option to log and consolidate logs from applications framework and user modules are often tied to the location and publications of logs from them.
Specific queries govern the monitoring for the log analytics. These are well-prepared and part of a standard portfolio of monitoring charts that help with the analysis of the health of the containers.
Specific logging implementation is discussed in an earlier article as outlined here

Logging in Kubernetes:
The support for logging in Kubernetes starts with a Kubernetes Daemonset that will monitor container logs and forward them to a log indexer. Any log forwarder can be used as the image for the container in the Daemonset yaml specification. Some environment variables and shell commands might be necessary to set and run on the container. There will also be a persistent volume typically mounted at /var/log. The mounted volumes must be verified to be available under the corresponding hostPaths. 
FluentD collector agent is a way to collect logs from across the pods. The logging sidecar model will be beneficial to the applications as it alleviates the concerns from them and at the same time provides a consistent model across the applications. At the store level, it is better that logs have their own storage and index whether it is the file storage or an index store. This will allow uniform aging, archiving and rolling of logs based on timeline and will work across all origins 
Logging can also be considered a service to provision external to the cluster. This is easy to do with a variety of log products that provide service like functionality. As long as there is no data loss, most log system users are tolerant to latency. This makes it easier for Logging to be implemented with merely a Kubernetes service broker and alleviating all concerns for logging from the cluster. 

Logging using Elasticsearch, Kibana and FluentD  helps end to end usage of logs from applications hosted on Kubernetes.

This section briefly discusses the setup specific to Elasticsearch, Kibana and Fluentd.  

The elasticsearch resource is specified as a single node deployment in yaml and with the container image as docker.elastic.co/elasticsearch/elasticsearch:6.5.4. The corresponding elasticsearch service is defined with a container port as 9200. 

The Kibana is specified as a deployment with image as docker.elastic.co/kibana/kibana:6.5.4 service with port 5601.  

FluentD is a logging agent. It is specified as a daemonset but first we need a fluentd-rbac.yaml where we define a serviceaccount, a  cluster role and a ClusterRoleBinding for fluentd so that it can access pods and namespaces with get, list and watch. The specification for fluentd as a Kubernetes DaemonSet with names as fluentd-logging which requires a container image as fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch and a volume mount of /var/log. The environment variables define FLUENT_ELASTICSEARCH_HOST, FLUENT_ELASTICSEARCH_PORT (9200), FLUENT_ELASTICSEARCH_SCHEME and FLUENT_ELASTICSEARCH_UID as 0. The pods should show the fluentd as running. 

Today we continue discussing the summary of the book "stream processing with Apache Flink". Flink is an Apache software which is well-known to unify stream and batch processing
It features low latency, high throughput, rich programmable APIs, flexible windows and exactly once processing. As opposed to Apache Samza which features state over stream with low level stream apis, Apache Spark Streaming which performs stream processing on top of batch processing, Apache Storm which features true streaming but with low level APIs, Apache Flink may be considered a good analytical frameworks suitable to handle all workloads such as event logs, historic data, and ETL transformations.
At its core, Apache Flink stack supports a DataStream APIs with stream optimizer over an Flink runtime. The data stream sources could even include message brokers and their connectors.
The DataStream APIs support a policy based flexible windowing semantic where time, count, delta etc can be for windowing. The operators on the streams involve joins, crosses, and support iterations. The stream transformations are supported with CoMap and CoReduce functions.
This is illustrated with the help of financial data where multiple inputs are read for say merging the stock data and simple sum/count may be used as aggregations on the windows or the windows can be data driven or a time-based tumbling window, and a streaming join can be performed on multiple data streams.
For example,
val socketStockStream = env.socketTextStream("localhost", 9999)
                                         .map( x => { val split = x.split(",")
                                         StockPrice(split(0), split(1).toDouble) })
val stockStream =  socketStockStream.merge(env.addSource(generateStock("TCKR")(10) _))
val windowedStream = stockStream.window(Time.of(10, SECONDS)).every(Time.of(5, SECONDS))
val maxByStock = windowedStream.groupBy("symbol").maxBy("price")

The window method call as a data-driven example could be:
.window(Delta.of(0.05, priceChange, defaultPrice))
Even a stream from social media can be used for correlations:
val rollingCorrelation = tweetsAndWarning.window(Time.of(30, seconds)).mapWindow(computeCorrelation _)
Reference: https://flink.apache.org/news/2015/02/09/streaming-example.html

An essay on a message broker for Kubernetes:

Kubernetes is an open-source cluster manager that is developed by Google and donated to Cloud Native Compute Foundation. Kubernetes provides automation for deployment scaling management of containerized applications. It does with the help of activities on a cluster pertaining to state convergence, service addresses, health monitoring, service discovery and DNS resolution.

One of the central themes of any automation is the ability to run the same activities over and over again for different state and resources. Much automation chose the option to make all the resources declared in succinct text format and allowing the operations to reconcile these resources to a given state. Kubernetes leverages these concepts to the point where all the needs for the applications as described as resources. Then the application can scale out by repeating this across the cluster members.

Clusters comprise of pods that have bare essentials to run the containers for their respective workloads from the applications they host. These pods are typically shallow, light weight and facilitate scale-out with the help of statefulsets and replicasets.

This worked very well to containerize and isolate applications so that they can be hosted anywhere and still perform well for large workloads with little or no disruption. However, established cloud and on- premise integration was essential for bridging the old with the new.

A service broker helped bridge the gap by representing a gateway to all external services. These services conformed to a Open Service Broker API standard that let external services be represented as internal resources so that they could be instantiated and commissioned as per cluster resources.

The point of all this integration was to enable Kubernetes to expand its reach and audience. This is witnessed by the popularity of Kubernetes where the participants to the conferences on Kubernetes increased by orders of magnitude in a few years.

Storage on the other hand has always been preferred local because as the data makes its way to storage tier, it tends to gravitate and become dense and stationary, aging over time. Pods in Kubernetes shared data by relying on file storage that was mounted as local file shares in each and every pod. If the storage were to be web accessible, a storage service was provisioned via the service broker.

Yet the Kubernetes ecosystem also benefits with the same notions of compute, storage and networking that has facilitated the world outside and is equally receptive to them as applications to be hosted on its pods. A message broker, for instance, makes the case for tapping networking to streamline the data access patterns suitable for dedicated storage.

A message broker is not a new innovation for Kubernetes but the ability to designate a class of workloads specific to message broker for the purposes of removing reliance on file-system and connecting various storage tiers is an improvement to this ecosystem. Up to this point, the jargon for Kubernetes involved general purpose definitions but some special purpose designations can improve the automation capabilities of Kubernetes that expand the backend dependencies of this framework. This essay is therefore a window to a new world of possibilities where products participate not just via a

service broker sending traffic that is not necessarily regulated, conforming, dedicated or even channeled. A message broker can bring class level quality of service and various capabilities specific to networking that improves the utilization of existing storage tier products.

Events are not relational and find many applications that are similar to the use cases of NoSql storage. Specifically, events conform to append only stream storage due to the sequential nature of the events. Events are also processed in windows making a stream processor such as Flink extremely suitable for events. Stream processors benefit from stream storage and such a storage can be overlaid on any Tier-2 storage.

Event storage gained popularity because a lot of IoT devices started producing them. Read and writes were very different from conventional data because they were time-based sequential and progressive. Although stream storage is best for events, any time-series database could also work. However, they are not web-accessible unless they are in an object store. Their need for storage is not very different from applications requiring object storage that facilitate store and access. However as object storage makes inwards into vectorized execution, the data transfers become increasingly fragmented and continuous. At this junction it is important to facilitate data transfer between objects and Event and it is in this space that Events and object store find suitability. Search, browse and query operations are facilitated in a web service using a web-accessible store. 

File-systems have long been the destination to store artifacts on disk and while file-system has evolved to stretch over clusters and not just remote servers, it remains inadequate as a blob storage. Data writers have to self-organize and interpret their files while frequently relying on the metadata stored separate from the files.  Files also tend to become binaries with proprietary interpretations. Files can only be bundled in an archive and there is no object-oriented design over data. If the storage were to support organizational units in terms of objects without requiring hierarchical declarations and supporting is-a or has-a relationships, it tends to become more usable than files.  

Since Event storage overlays on Tier 2 storage on top of blocks, files, streams and blobs, it is already transferring data to object storage. However, the reverse is not that frequent although objects in a storage class can continue to be serialized to Event in a continuous manner. It is also symbiotic to audience on both storage. 

On the Kubernetes side, the collector is one of the logging drivers. The JSON logging driver, the journald logging driver, fluentd are some of the examples. FluentD gives the ability to customize each input channel in terms of format and content which subsequently helps with search queries.  

There are many connectors to choose from when it comes to picking a connector for a specific log index store. In this section, we describe a particular connector for stream storage because this is helpful to directly send the data to stream storage. 

The logstash connector can send data to a stream.  A plugin for logstash is required only when the stream needs to be bound. In this case, a clientconfiguration is instantiated and bound to the stream controller uri along with the credentials such as username, password if the stream connections are not open for all. The streammanager is provided the clientconfiguration along with the parameters to set up the stream. For example, the parameters might include a routingkey and a stream configuration with a scaling policy that can include the minimum number of segments. The routingkey is only to locate the stream. The stream configuration describes the behaviour of the stream. With this configuration provided to the stream manager, a stream can be created.  

Then a clientFactory can instantiate a reader and writer on the stream. With the writer an encoded data can then be serialized as event into the event stream. All subsequent data writes are essentially with the stream writer.  

This plugin for connector will usually multiplex data into the event writer. As long as the plugin is bound to the connector, it acts like a funnel. 

Logstash can handle all types of logging data. At some point, metrics and log entries may look similar for data and this makes logstash extend its inputs to collect data from both sources. The output of logstash is the input to the stream. The plugin for a particular stream is therefore required on the output side of the logstash.  

Logstash may be considered external to the Kubernetes cluster but it can also be deployed local to the cluster

Logstash could be considered heavy for the purposes of sending logs to a store. Any log forwarder can simply forward the logs to a store listening on a tcp port. This is the equivalent of the Unix command “tail –f logfile ¦ nc host port”. The store can even be the same stream storage that is within the cluster.