Logging as a side car model:
There are various levels and areas of emphasis in the stack to support rich analytics over logging. At the Kubernetes system level, among the popular options, we could prioritize fluentD logging sidecar as a way to collect logs from across the pods. This will be beneficial to the applications as it alleviates the concerns from them and at the same time provides a consistent model across the applications. At the store level, we recommend that the logs have their own storage whether it is the file storage or an index store. This will allow uniform aging, archiving and rolling of logs based on timeline and will work across all origins
At the analytics level, we can outsource the reporting stack, charts, graphs and dashboards to the application that is best to analyze machine data. Although StorageApplication, SmartAnalytics, metrics, charts and graphs can be used to store, search and render results, it will take a longer round-trip time between the user query and the results.

There is usually only one sidecar  per service in a sidecar proxy deployment. A sidecar can be in the same pod as the one hosting the service. The number of instances of sidecars can scale as appropriate. A smaller resource profile is necessary for allowing the explosion of sidecar instances.  If the target of a sidecar is available as a daemonset, there will only be one sidecar per host rather than one sidecar per pod.
#codingexercise
Task getNext(List<Task> tasks) {
For (Task task :tasks) {
If (task.Completed == false) {
Return task;
}
}
Return null;
} 

Logging as a service broker implementation:
Kubernetes service brokers allow the provisioning of services outside the cluster. This enables services to run anywhere and work independently from the cluster. They can even be deployed in the public and private cloud where they can scale across Kubernetes clusters. This is a helpful addition to any Kubernetes cluster.
Almost all provisioning of resources translates into well-known operations of create, update, delete, get and list on the resources. Therefore, the resource operations are aligned with the usage of resources along most workflows. This makes it handy for use of Kubernetes clusters for many applications and services.
The trend in cloud computing has shifted from service-oriented architecture towards microservices. This has helped the independent provisioning of resources, deployment and scaling of services and overhaul or reworking of services. Service broker makes no claim about how the services need to evolve - whether to form a structural composite pattern or a behavior maintaining pattern. This minimal enforcement has worked mutually beneficial for both the services as well as the Kubernetes cluster.
Logging can also be considered a service to provision external to the cluster. This is easy to do with a variety of log products that provide service like functionality. As long as there is no data loss, most log system users are tolerant to latency. This makes it easier for Logging to be implemented with merely a Kubernetes service broker and alleviating all concerns for logging from the cluster.

Today we continue discussion logging in Kubernetes.  We start with a Kubernetes Daemonset that will monitor container logs and forward them to a log indexer. Any log forwarder can be used as the image for the container in the Daemonset yaml specification. Some environment variables and shell commands might be necessary to set and run on the container. There will also be a persistent volume typically mounted at /var/log. The mounted volumes must be verified to be available under the corresponding hostPaths.
 Journald is used to collect logs from those components that do not run inside a container. For example, kubelet and the controller runtime which is usually Docker will write to journald when the host has systemd enabled. Otherwise, they write to .log files in /var/log directory. Klog is the logging library used by such system components.
One log indexer is sufficient for a three node cluster with thirty containers generating 1000 messages/second each even when the message size can be a mix of small (say 256 byte) and large (1KB).
Timber is an example of a log product. The use of this product typically entails logstash, elasticsearch, and Kibana where the elasticsearch is for api access and the kibana is for web user interface. Any busybox container image can be used to produce logs which we can use as test data for our logging configuration.
The logrotate tool rotates the log once the side exceeds a given threshold.
The typical strategies for pushing logs include the following:
1) use a node-level logging agent that runs on every node. For example, Stackdriver logging on Google cloud platform and ElasticSearch on conventional Kubernetes clusters.
2) include a dedicated sidecar container for logging in an application pod.
3) Push logs directly in the backend from within an application.
Between the options 1 and 2, the latter is preferable for performance reasons. It is not intrusive and i collects the logs with fluentd which provides a rich language to annotate or transform log sources. Also, option 2 can scale independently without impact to the rest of the cluster.

Today we are going to discuss log indexer deployment to container framework.  We start with a Kubernetes Daemonset that will monitor container logs and forward them to a log indexer. Any log forwarder can be used as the image for the container in the Daemonset yaml specification. Some environment variables and shell commands might be necessary to set and run on the container. There will also be a persistent volume typically mounted at /var/log. The mounted volumes must be verified to be available under the corresponding hostPaths.
In order for logging to be helpful, it is better to have the log sources differentiated. For example, the tools to deploy and monitor the app must be different from the tool to deploy and maintain the container cluster. Although forwarders and indexers can differentiate log stream, it is better to do that at the cluster level.  There are also plugins available from different log indexer product companies which support Docker logging.
The forwarder is also specific to the log product company. We need it only to forward logs. It can be run as a a Daemonset or directly on the Kubernetes nodes. The forwarder is not only a proprietary tool, it is a convenience for deployers to move lots of data reliably and securely using log product maker guidelines. Json driver and journald can be used for integration with Kubernetes. Journald is used to collect logs from those components that do not run inside a container. For example, kubelet and the controller runtime which is usually Docker will write to journald when the host has systemd enabled. Otherwise, they write to .log files in /var/log directory. Klog is the logging library used by such system components.
At this point, it should be important to mention that the collector does not only collect logs.
The log product event collector needs all of the following:
1) logging
2) Metadata/objects
3) Metrics
One indexer is sufficient for a three node cluster with thirty containers generating 1000 messages/second each even when the message size can be a mix of small (say 256 byte) and large (1KB).
Timber is another example of a log product. The use of this product typically entails logstash, elasticsearch, and Kibana where the elasticsearch is for api access and the kibana is for web user interface. Any busybox container image can be used to produce logs which we can use as test data for our logging configuration.
The logrotate tool rotates the log once the side exceeds a given threshold.
The typical strategies for pushing logs include the following:
1) use a node-level logging agent that runs on every node. For example, Stackdriver logging on Google cloud platform and ElasticSearch on conventional Kubernetes clusters.
2) include a dedicated sidecar container for logging in an application pod.
3) Push logs directly in the backend from within an application. 

A warm welcome for Kubernetes Service brokers.

Kubernetes service brokers allow the provisioning of services outside the cluster. This enables services to run anywhere and work independently from the cluster. They can even be deployed in the public and private cloud where they can scale across Kubernetes clusters. This is a helpful addition to any Kubernetes cluster.

The service broker architecture also enforces consistency across provisioning of resources. This is done with the help of a standard set of APIs for all resources and their corresponding services that implement the service broker.  Resources are the way in which Kubernetes recognizes each and every entity in the cluster. The framework of Kubernetes then takes this inventory of resources and reconciles the state of the cluster to match the definition of the resources. Anything hosted on the Kubernetes clusters can be described with the help of a custom resource.

Service brokers facilitate the resources provisioned by external services to be understood by Kubernetes as custom resources within the cluster.  This bridging of external resource provisioning with in cluster representation of resources has provided significant benefits to users for the use of these resources.

Almost all provisioning of resources translates into well-known operations of create, update, delete, get and list on the resources. Therefore the resource operations are aligned with the usage of resources along most workflows. This makes it handy for use of Kubernetes clusters for many applications and services.

The trend in cloud computing has shifted from service oriented architecture towards microservices. This has helped the independent provisioning of resources, deployment and scaling of services and overhaul or reworking of services. Service broker makes no claim about how the services need to evolve - whether to form a structural composite pattern or a behavior maintaining pattern. This minimal enforcement has worked mutually beneficial for both the services as well as the Kubernetes cluster.

Yesterday we were referring to the design of Kubernetes Service brokers. Today we look at the implementation for one of them.
A typical service broker will implement the  ServiceInstanceService and ServiceInstanceBindingService Methods that correspond to the ServiceInstance and ServiceBinding definititons in the Kubernetes framework.
The ServiceInstanceService implementation will determine
1) whether to accept a given service instance request for create, update, delete, get or list operations.
2) The classes are separate for each resource they represent.
3) They are composed under a composite service that represents a facade to the composite OSBA service. The OSBA service therefore allows service oriented architecture for the implementations of the service broker.
4) The service is looked up by the service broker with a match to the service definition in the request. The method accepting the request in the composite service implementation can determine if it should handle the request.
5) validations for accepting requests are performed by the service broker.
6)  when a service instance is created, it tries to populate the resource representation and persist the resource. Sometimes persisting the resource to be created or updated requires interacting with other services. These may need to be done in a transaction scope and support good exception handling, diagnosability and transparency.

A service instance binding is looked up for a service instance with the help of the following supported lookup techniques:
ClusterServiceClassExternalName and ClusterServicePlanExternalName
ClusterServiceClassExternalID and ClusterServicePlanExternalID
ClusterServiceClassName and ClusterServicePlanName
ServiceClassExternalName and ServicePlanExternalName
ServiceClassExternalID and ServicePlanExternalID
ServiceClassName and ServicePlanName

We continue with our discussion of Keycloak service broker on Kubernetes.

Service brokers are independent and they are not connected except by passing parameters.
https://github.com/kubernetes-sigs/service-catalog/blob/master/docs/parameters.md

Advantages of service catalog

Enable services to be hosted outside cluster
Adhere to OSBA API
Allow services to be independent and scalable and define their own resources.
The sevice catalog allows services to own and describe their policies

Disadvantages of service catalog
They are not for relations or dependencies between services
They cannot handle sync
The infrastructure cannot look into the resources
There is no querying or mapping of resources to services other than what is declared
Annotations and tags are not supported in a way that service broker mapping for resources can be looked up programmatically.

The upshot is that service broker seems to bring on the complexities of service-oriented architecture where composition is a pillar of organization.

We continue with our discussion on sample calls using Kubernetes admin rest API.
Sample calls:
curl -i -k -u admin:password 'http://localhost:8080/auth/realms/master/protocol/openid-connect/token' -X POST -d 'grant_type=password' -d 'client_id=admin-cli' -d 'username=admin' -d 'password=password'
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Tue, 23 Jul 2019 18:45:19 GMT
Content-Type: application/json
Content-Length: 1921
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-store
Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; HttpOnly
Pragma: no-cache

{"access_token":"eyJhbGc_<trimmed>_K9G1ApFj01Q","expires_in":60,"refresh_expires_in":1800,"refresh_token":"eyJhb_<trimmed>_94","token_type":"bearer","not-before-policy":0,"session_state":"91be6030-ecf2-4744-b9db-6d70caf75a6d","scope":"profile email"}

curl -i -k 'http://localhost:8080/auth/admin/realms/master/roles/' -H 'Authorization: Bearer eyJhbGc_<trimmed>_K9G1ApFj01Q'

#
# curl -i -k 'http://localhost:8080/auth/admin/realms/master/roles/' -H 'Authorization: Bearer eyJhbGc_<trimmed>_K9G1ApFj01Q'
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Tue, 23 Jul 2019 19:17:14 GMT
Content-Type: application/json
Content-Length: 804
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-cache

[{"id":"d73bf354-fb63-4a2e-8e61-a91cae1d9302","name":"uma_authorization","description":"${role_uma_authorization}","composite":false,"clientRole":false,"containerId":"master"},{"id":"b731700b-d023-4cf0-83b6-cb858337c331","name":"user","composite":false,"clientRole":false,"containerId":"master"},{"id":"6abfcdf0-24bf-4fb5-bddc-1f6781c7a158","name":"admin","composite":true,"clientRole":false,"containerId":"master"},{"id":"2154a5ad-55db-4950-acdd-1932cfb27ce7","name":"project1-member","description":"Provides access to resources for namespace 'master-system'.","composite":false,"clientRole":false,"containerId":"master"},{"id":"1a45fb30-cd43-4afe-8348-644ccecfb303","name":"offline_access","description":"${role_offline-access}","composite":false,"clientRole":false,"containerId":"master"}] #


curl -i -k 'http://localhost:8080/auth/admin/realms/master/clients/' -H 'Authorization: Bearer eyJhbGc_<trimmed>_K9G1ApFj01Q'
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Tue, 23 Jul 2019 19:39:59 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-cache

[{"id":"1f913347-16eb-4914-901d-63df062c7f17","clientId":"realm-management","name":"${client_realm-management}","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":true,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":false,"nodeReRegistrationTimeout":0,"protocolMappers":[{"id":"7606ca45-f717-4166-9da2-5da6af7e6834","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"368ed8fc-7f93-4236-b29d-463383c46e49","clientId":"account","name":"${client_account}","baseUrl":"/auth/realms/master/account","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","defaultRoles":["view-profile","manage-account"],"redirectUris":["/auth/realms/master/account/*"],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":false,"nodeReRegistrationTimeout":0,"protocolMappers":[{"id":"8a5319db-0351-4500-a42e-caa382047771","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"d8f4b92c-b2dc-499c-96a5-01b15f3fd721","clientId":"project1-repo","name":"project1-repo","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":true,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"c6d09f4a-ad6d-11e9-b875-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50991","bindingID":"c6d2d31a-ad6d-11e9-b875-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"363d2abb-1532-48ee-816a-eea23a397790","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","project1-repo","role_list","profile","roles","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"e4bc688e-af1a-46f9-a629-a81b24ef01c3","clientId":"master-cli","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":true,"serviceAccountsEnabled":false,"publicClient":true,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"saml.assertion.signature":"false","saml.force.post.binding":"false","saml.multivalued.roles":"false","saml.encrypt":"false","saml.server.signature":"false","saml.server.signature.keyinfo.ext":"false","exclude.session.state.from.auth.response":"false","saml_force_name_id_format":"false","saml.client.signature":"false","tls.client.certificate.bound.access.tokens":"false","saml.authnstatement":"false","display.on.consent.screen":"false","saml.onetimeuse.condition":"false"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"0a07cf37-9b99-4427-9d3d-272d21c1d14c","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"1a807aa4-c255-4784-97fe-54e1dc866f3f","clientId":"project1-flink","name":"project1-flink","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":["*"],"webOrigins":["+"],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":true,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"c6c1d4b4-ad6d-11e9-b875-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50988","bindingID":"c6c2615a-ad6d-11e9-b875-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"86bba63f-e4ec-40c0-b1e6-4d78fd6d3f99","name":"Client ID","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientId","id.token.claim":"true","access.token.claim":"true","claim.name":"clientId","jsonType.label":"String"}},{"id":"dfceafff-ae50-4dd4-94d0-62123ba83e26","name":"Client IP Address","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientAddress","id.token.claim":"true","access.token.claim":"true","claim.name":"clientAddress","jsonType.label":"String"}},{"id":"4567f93e-67b7-4925-9b61-99d1012222ed","name":"Client Host","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientHost","id.token.claim":"true","access.token.claim":"true","claim.name":"clientHost","jsonType.label":"String"}},{"id":"e912b822-c63d-4499-8452-37c7b3bac5a1","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","project1-flink","profile","roles","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"375d646f-f83e-44b9-9c81-9026fea20f70","clientId":"broker","name":"${client_broker}","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":false,"nodeReRegistrationTimeout":0,"protocolMappers":[{"id":"d761c7d1-fc2f-42a8-be93-efcd69134a04","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"2b96d311-8a72-4425-9762-a5832ba814d1","clientId":"security-admin-console","name":"${client_security-admin-console}","baseUrl":"/auth/admin/master/console/index.html","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":["/auth/admin/master/console/*"],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":true,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":false,"nodeReRegistrationTimeout":0,"protocolMappers":[{"id":"329747cf-d72f-41ca-b2ac-b685561aaddc","name":"locale","protocol":"openid-connect","protocolMapper":"oidc-usermodel-attribute-mapper","consentRequired":false,"config":{"userinfo.token.claim":"true","user.attribute":"locale","id.token.claim":"true","access.token.claim":"true","claim.name":"locale","jsonType.label":"String"}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"dae45177-8406-4ba5-be8d-f81541e7ee97","clientId":"master-ui","name":"DellEMC Stream Processing Platform UI","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":["http://frog-man.lab-oxygen.demo.masterbeta.com/*","http://localhost:9000/*"],"webOrigins":["http://frog-man.lab-oxygen.demo.masterbeta.com","http://localhost:9000"],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":true,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":true,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"de8d629f-a8ea-11e9-ae64-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50989","bindingID":"de80aa90-a8ea-11e9-ae64-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"f662d042-fe40-4b16-92ee-ab53655516cd","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","profile","roles","master-ui","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"f043e6d3-acec-4e79-9f15-2e4ded24718a","clientId":"project1-pravega","name":"project1-pravega","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":true,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"c6bb43b9-ad6d-11e9-b875-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50988","bindingID":"c6c0a12c-ad6d-11e9-b875-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"30d0b1ff-7eb4-40d9-a8c2-dcacb5dae4d7","name":"Client IP Address","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientAddress","id.token.claim":"true","access.token.claim":"true","claim.name":"clientAddress","jsonType.label":"String"}},{"id":"1ed5020d-6cb4-4fc8-9bbf-deeac91a7fb8","name":"Client Host","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientHost","id.token.claim":"true","access.token.claim":"true","claim.name":"clientHost","jsonType.label":"String"}},{"id":"ab51bc73-709b-48db-8226-4fe292d39001","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}},{"id":"85c47dc2-6b02-49dd-b9c2-9ad15780a665","name":"Client ID","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientId","id.token.claim":"true","access.token.claim":"true","claim.name":"clientId","jsonType.label":"String"}}],"defaultClientScopes":["web-origins","role_list","profile","roles","project1-pravega","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"8f665d0f-bad9-4567-bdf0-7d96ecdf04cb","clientId":"kcsa-manager","name":"kcsa-manager","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":true,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"010a6558-0217-4ea9-91f2-18e6bd4518ba","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}},{"id":"635412eb-602b-4701-b4ae-2f17a83dc312","name":"Client IP Address","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientAddress","id.token.claim":"true","access.token.claim":"true","claim.name":"clientAddress","jsonType.label":"String"}},{"id":"7abf0cd9-6157-4f77-932a-a30ebeabd652","name":"Client ID","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientId","id.token.claim":"true","access.token.claim":"true","claim.name":"clientId","jsonType.label":"String"}},{"id":"b6dd8329-2328-4355-90ca-753dafe4f7d1","name":"Client Host","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientHost","id.token.claim":"true","access.token.claim":"true","claim.name":"clientHost","jsonType.label":"String"}}],"defaultClientScopes":["web-origins","role_list","profile","roles","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"8306ff5e-e770-40e7-a361-ab5cfc5d4157","clientId":"grafana","name":"grafana","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":["http://grafana.frog-man.lab-oxygen.demo.masterbeta.com/*"],"webOrigins":["http://grafana.frog-man.lab-oxygen.demo.masterbeta.com"],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":true,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"eee21f41-a8ea-11e9-ae64-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50991","bindingID":"eed44cba-a8ea-11e9-ae64-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"5228794f-9ff3-44ee-b6a3-9112880b9b5b","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","grafana","profile","roles","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"671fddf5-7bd7-452c-8300-c97337cd9d1d","clientId":"admin-cli","name":"${client_admin-cli}","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":true,"serviceAccountsEnabled":false,"publicClient":true,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":false,"nodeReRegistrationTimeout":0,"protocolMappers":[{"id":"93e826bb-1c1c-4d5c-bf19-177ec06996b3","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","role_list","roles","profile","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}},{"id":"38adb0ea-d81f-4cc5-b9c3-4786ad46fbea","clientId":"pravega-controller","name":"pravega-controller","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":false,"serviceAccountsEnabled":true,"authorizationServicesEnabled":true,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"eef102ef-a8ea-11e9-ae64-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50988","bindingID":"eee1947c-a8ea-11e9-ae64-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"5cef8eda-193b-48a5-9299-62c805b6b42e","name":"Client IP Address","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientAddress","id.token.claim":"true","access.token.claim":"true","claim.name":"clientAddress","jsonType.label":"String"}},{"id":"aaf6e57b-18b8-455b-a9fb-40340a5b84bb","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}},{"id":"17947fed-238a-4399-a146-0c37adead31c","name":"Client ID","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientId","id.token.claim":"true","access.token.claim":"true","claim.name":"clientId","jsonType.label":"String"}},{"id":"5dab1482-a4c0-4047-886c-7047bff176a0","name":"Client Host","protocol":"openid-connect","protocolMapper":"oidc-usersessionmodel-note-mapper","consentRequired":false,"config":{"user.session.note":"clientHost","id.token.claim":"true","access.token.claim":"true","claim.name":"clientHost","jsonType.label":"String"}}],"defaultClientScopes":["web-origins","role_list","profile","roles","pravega-controller","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}}]



# curl -i -k 'http://localhost:8080/auth/admin/realms/master/clients/d8f4b92c-b2dc-499c-96a5-01b15f3fd721/roles/' -H 'Authorization: Bearer eyJhbGc_<trimmed>_K9G1ApFj01Q'
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Tue, 23 Jul 2019 19:55:35 GMT
Content-Type: application/json
Content-Length: 2
Connection: keep-alive
Cache-Control: no-cache

# curl -i -k 'http://localhost:8080/auth/admin/realms/master/clients/d8f4b92c-b2dc-499c-96a5-01b15f3fd721' -H 'Authorization: Bearer eyJhbGc_<trimmed>_K9G1ApFj01Q'
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Tue, 23 Jul 2019 19:55:55 GMT
Content-Type: application/json
Content-Length: 1151
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-cache

{"id":"d8f4b92c-b2dc-499c-96a5-01b15f3fd721","clientId":"project1-repo","name":"project1-repo","surrogateAuthRequired":false,"enabled":true,"clientAuthenticatorType":"client-secret","redirectUris":[],"webOrigins":[],"notBefore":0,"bearerOnly":false,"consentRequired":false,"standardFlowEnabled":false,"implicitFlowEnabled":false,"directAccessGrantsEnabled":true,"serviceAccountsEnabled":false,"publicClient":false,"frontchannelLogout":false,"protocol":"openid-connect","attributes":{"serviceInstanceID":"c6d09f4a-ad6d-11e9-b875-02505600310c","planID":"42b1c543-7207-408c-a2ec-9c34182f50991","bindingID":"c6d2d31a-ad6d-11e9-b875-02505600310c"},"authenticationFlowBindingOverrides":{},"fullScopeAllowed":true,"nodeReRegistrationTimeout":-1,"protocolMappers":[{"id":"363d2abb-1532-48ee-816a-eea23a397790","name":"docker-v2-allow-all-mapper","protocol":"docker-v2","protocolMapper":"docker-v2-allow-all-mapper","consentRequired":false,"config":{}}],"defaultClientScopes":["web-origins","project1-repo","role_list","profile","roles","email"],"optionalClientScopes":["address","phone","offline_access"],"access":{"view":true,"configure":true,"manage":true}}

#codingexercise

Count the number of nodes in a circular linked list
Integer count (Node start) {
Integer count = 0;
If (start == null) return count;
count +=1;
If (start.next == start) return count;
Node cur = start;
while (cur.next != start) {
    count += 1;
    cur = cur.next;
}
return count;
}

Find if two strings are interleaved in a third string

        static bool isInterleaved(String A, String B, String C)

        {

            int ia = 0;

            int ib = 0;

            for (int i = 0; i < C.Length; i++)

            {

                if (ia < A.Length && C[i] == A[ia])

                {

                    ia++;

                }

                else if (ib < B.Length && C[i] == B[ib])

                {

                    ib++;

                }

                else

                return false;

               

            }

            if (ia != A.Length || ib != B.Length)

                return false;

            return true;

        }

Today we describe the authentication for the admin rest API  for Keycloak referenced earlier. The API is helpful for Kubernetes cluster security where users are identified with Keycloak. We assume a deployed instance available at http://localhost:8080/auth.

The admin API takes a token from '/realms/master/protocol/openid-connect/token' A token can be requested using password grant and using default admin credentials from keycloak open source.
To clear login failures for all users and release temporarily disabled users, we use:

DELETE /{realm}/attack-detection/brute-force/users

To get the status of a username in brute force detection, we have

GET /{realm}/attack-detection/brute-force/users/{userId}

To get all roles for the realm or client, we have

GET /{realm}/clients/{id}/{roles}

to get a  role by name we have

GET /{realm}/clients/{id}/roles/{role-name}

To update a role by role name, we have

PUT /{realm}/clients/{id}/roles/{role-name}

Add a composite to the role, we have

POST /{realm}/clients/{id}/roles/{role-name}/composites

Add a client-level role to the user role mappings, we have

POST /{realm}/groups/{id}/role-mappings/clients/{client}

To get a list of all users

GET /{realm}/users

to get the representation of a user, we have

GET /{realm}/users/{id}

To revoke consents and offline tokens for particular client from users

DELETE /{realm}/users/{id}/consents/{client}

To get all admin-events for a realm, we have

GET /{realm}/admin-events

to get the client registration policy providers with configProperties properly filled, we have

GET /{realm}/client-registration-policy/providers

to add the client-roles to the user role mapping, we have

POST /{realm}/groups/{id}/role-mappings/clients/{client}

Note that the Kubernetes namespaces are not part of the keycloak role representation. Keycloak may or not be hosted on Kubernetes. To use kubectl for enumerating serviceinstance and service bindings, we need to use the proper namespace.

#codingexercise

Count the number of nodes in a circular linked list
Integer count (Node start) {
Integer count = 0;
If (start == null) return count;
count +=1;
If (start.next == start) return count;
Node cur = start;
while (cur.next != start) {
    count += 1;
    cur = cur.next;
}
return count;
}

Today we enumerate some of the apis from the admin rest api reference for Keycloak. This is helpful for Kubernetes cluster security where users are identified with Keycloak. We assume a deployed instance available at http://localhost:8080/auth.
To clear login failures for all users and release temporarily disabled users, we use:
DELETE /{realm}/attack-detection/brute-force/users
To get the status of a username in brute force detection, we have
GET /{realm}/attack-detection/brute-force/users/{userId}
To get all roles for the realm or client, we have
GET /{realm}/clients/{id}/{roles}
to get a  role by name we have
GET /{realm}/clients/{id}/roles/{role-name}
To update a role by role name, we have
PUT /{realm}/clients/{id}/roles/{role-name}
Add a composite to the role, we have
POST /{realm}/clients/{id}/roles/{role-name}/composites
Add a client-level role to the user role mappings, we have
POST /{realm}/groups/{id}/role-mappings/clients/{client}
To get a list of all users
GET /{realm}/users
to get the representation of a user, we have
GET /{realm}/users/{id}
To revoke consents and offline tokens for particular client from users
DELETE /{realm}/users/{id}/consents/{client}
To get all admin-events for a realm, we have
GET /{realm}/admin-events
to get the client registration policy providers with configProperties properly filled, we have
GET /{realm}/client-registration-policy/providers
to add the client-roles to the user role mapping, we have
POST /{realm}/groups/{id}/role-mappings/clients/{client}
Note that the Kubernetes namespaces are not part of the keycloak role representation. Keycloak may or not be hosted on Kubernetes. To use kubectl for enumerating serviceinstance and service bindings, we need to use the proper namespace.

We were enumerating security pod context policies yesterday. These include the following:
1) a privileged - determines if any container in a pod can be allowed access to devices on the host. A privileged container gives access to all devices on the host.
2) hostPID/hostIPC - controls whether the pod containers can share the host Process ID and IPC namespaces which makes it accessible from outside the container.
3) hostNetwork/hostPort - controls whether the pod may share the node network namespace and have access to the loopback device, listening on localhost or snooping on network activity.
4) allowed HostPaths - specifies a whitelist of host paths where a pathPrefix allows host paths to begin with that prefix only and a readOnly field to indicate no write access.
5) allowed flexVolume - specifies a whitelist of flexvolume drivers when the volume is a flexvolume type. A flexvolume allows vendor specific operations to third party storage backend providers
6) fsGroup - allows groups on volumes with the runAs directive to specify fsGroup ID.
7) readOnlyRootFileSystem - specifies the container to run with no writeable layer.
8) runAsUser and runAsGroup - specifies which user id or group the containers are run with. a non-root enforces least privilege policy
9) Privilege escalation - these options allow the privilege escalation container option.
10) Capabilities - specify linux  'capabilities' which are per thread attributes to specify the permissions available in categories under privileged accounts. A whitelist of capabilities are specified with this list.
11) SELinux - short form for Security Enhanced Linux provides support for the enforcement of different access control policies. Directives using RunAs specify different seLinuxOptions
12) AllowedProcMountType - specifies a whitelist of proc mount types. Most container runtimes mask certain paths in /proc to avoid divulging special devices or information
13) apparmor and seccomp - are annotations for profiles that the containers can run with.
14) forbiddensysctls excludes specific sysctls which can be a combination of safe and unsafe syctls.

The security of a container depends on the following three items:
1) Role and usage of service accounts
2) Role based access controls
3) Defining the security context of a pod.
Kubectl service account creation is covered earlier. Role based access control helps manage proliferation of user accounts to access secrets. The securing of pods helps with the enforcement of policies such as least privilege policy.
Let us look at some of the security context to apply to pods.
The policies are enumerated as:
1) privileged: this governs the containers to be run as privileged.
2) hostPID: usage of host namespaces
3) hostNetwork: usage of host networking
4) volumes: usage of volume types
5) allowedHostPaths: usage of host paths
6) allowedFlexTypes: usage of flex volumes
7) fsgroup: allocating an FSGroup that owns the pods volume.
8) readonlyrootfilesystem: requires the use of readonly root filesystem.
9) runasuser: the user ID of the container.
10) allowPrivilegeEscalation:  restricting escalation to root privileges 

When the pods are in trouble, it can be recovered with a liveness probe. The logs for the pods indicate if there were some errors. However, the mitigation to restart the pods cannot always be based on the detection of errors from the logs. This is where a liveness probe helps because it will restart the pod automatically when the probe fails. There are three different types of probes 1) a probe can be a command 2) it can be an HTTP request to a path served by a web server or 3) it can be a generic TCP probe. All types of probes target what is running within the containers.
The traffic flow to a pod can be controlled using a readiness probe. Even if the pods are up and running, we only want to send traffic to them when they are ready to serve requests. The readiness probe also has three different types just like the liveness probe and they can be used one for the other. However, they serve different purposes and should be maintained separately.
The liveness and readiness probes are defined in the containers section of the pod specification. They are denoted by livenessProbe and readinessProbe in the pod deployment yaml specification. 
The kubelet on each worker node uses a livenessProbe to overcome ramp-up issues or deadlocks. A service that load-balances a set of pods uses the readinessProbe to determine if a pod is ready and hence should receive traffic. A livenessProbe uses a restartPolicy and so does the readinessProbe when they are used interchangeably but for different purposes.

Today we continue with our discussion on Kubernetes user accounts and refresh tokens from our earlier post. The refresh token is retrieved from the identity provider's authorization url. Kubectl refreshes the ID token with the help of a refresh token. Kubernetes never uses the refresh token. It is meant to be a secret for the user.  A refresh token is generated only once and it can only be passed between the user and the identity provider. This makes it more secure than long lived bearer tokens. It is also opaque so there is no personally identifiable information divulged.  The Kubernetes dashboard uses id token and refresh token. It does not have a login system so it requires an existing token. The dashboard has therefore required the use of a reverse proxy which will inject the id_token on each request. The same reverse proxy then refreshes the token as needed.  This certainly alleviates the user authentication from the Kubernetes dashboard so much so that it can now be directly included with the user interface of the applications hosted on the Kubernetes system. Most of the panels in the dashboard are read-only so this is very helpful to all users  

The identity provider serves two purposes it honors the Open ID connect way of providing identities. As part of that identity, it will need to support discovery which is required to make calls. Second it is required to support the generation of tokens and to inject them into kube configuration. A variety of identity providers can support both these functionalities 

#codingexercise
Method to classify synonyms:

def classify_synonyms(): 

    words = [{'cat': ['animal', 'feline']}, {'dog':['animal', 'lupus']}, {'dolphin':['fish', 'pisces']}, {'spider':['insect','arachnid']}] 

    groups = [] 

    for item in words: 

        if item not in groups: 

             merged = False 

             for key in groups: 

                 group = next(iter(key)) 

                 for value in iter(item.values()): 

                  if group in value: 

                    index = groups.index(key) 

                    old = iter(groups[index].values()) 

                    new = iter(item.keys()) 

                    merged = [] 

                    for v in old: 

                        merged += v 

                    merged += new 

                    groups[index] = {group:merged} 

                    merged = True 

             if not merged: 

                k = next(iter(item.keys())) 

                v = next(iter(item.values())) 

                groups.append({v[0] : [k]}) 

    print(groups) 

classify_synonyms() 

#[{'animal': ['cat', 'dog']}, {'fish': ['dolphin']}, {'insect': ['spider']}] 

The above method merely classifies the input to the first level of grouping. It does not factor in multiple matches between synonyms, selection of the best match in the synonym, unavailability of synonyms, unrelated words, and unrelated synonyms. The purpose is just to show that given a criterion for the selection of a group, the words can be merged. The output of the first level could then be taken as the input of the second level. The second level can then be merged and so on until a dendrogram appears.  

Given this dendrogram, it is possible to take edge distance as the distance metric for semantic similarity.  

Since we do this hierarchical classification only for the finite number of input words in a text, we can take it to be a bounded cost of O(nlogn) assuming fixed upper cost for each merge. 

def nlevel(id, group_dict=df.GroupID, _cache={0:0}): 

    if id in _cache: 

        return _cache[id] 

  

    return 1+nlevel(group_dict[id],group_dict) 

  

df['nLevel'] = df.ID.map(nlevel) 

  

print df[['nLevel','ID','Group']] 

It is also possible to apply page rank given that synonyms are connected by edges. 

def pagerank (u, constant): 

sum = constant 

For node-v in adjacencies (u): 

     Sum += Pagerank_for_node_v (node-v) / number_of_links (node-v) 

Return sum