Ingress control on Applications requiring Keystore and truststore on Kubernetes clusters. 

Keys and certificates are used to secure data by using the public key to encrypt and the private key to decrypt. A certificate is used as a stamp of authority. It can include the public key. The certificate then becomes usable to secure the ends of a channel such as https.  Applications tend to require a key and a certificate in their configuration. Sometimes they require keystores and truststores as alternative formats. Keystores is a combination of key and certificate. It is made available in the form of a file with pfx extension or p12 extension. The truststores is merely a collection of certificates to be trusted. It could include a certificate chain if the certificates are signed. The generation of these bundles is specified in the RFC and performed with command line tools such as openssl and keytool. Not all languages have libraries to generate these bundles  and therefore the logic to generate the bundle is not part of many software development kits (SDK). 

Kubernetes is a system for managing the containerized applications. It facilitates deployment and scaling. As part of deployment, applications have to set up accounts, passwords and other secrets. These secrets are necessary to be made available as files and environment variables for the deployment to go through. 

The infrastructure provides a standard way of keeping these secrets. The secret is specified declaratively and then subsequently made available to the application. It can also be specified dynamically by the application code. 

The secret is kept in the form of base64 encoded data. The secret can be written out a file on mounted volume for use by the application. In such case, it is preferable to mark the volume as read only. The secret can be accessible from any pod in the system 

Kubernetes (aka K8s) applications are deployed with the help of an SDK called the operator-sdk. A Kubernetes operator is a controller that takes its resource definitions for an application and reconciles the container configuration with the definitions. For example, an upgrade would be defined by the source and destination version numbers and associated locations to permit the controller to take the necessary actions. Definitions are for resources specific to the applications and they are called custom resource definitions. They are written in Yaml – a language for definitions and maintained in a folder called deploy under the root project folder. The other folder is for the logic in the controller and written in Go language. The operator logic has code for apis and controller. As with all logic, there is also code for the entry point invocation. Moreover, there is a Dockerfile created by packaging the operator into a Docker image that deploys the operator and an associated account to run it. Writing the operator is facilitated with the help of an operator software development kit which is available as a command line tool that generates the scaffolds necessary for the api and the controller. 

Kubernetes takes keys, certificates, keystores and truststores as secrets. For example, we can specify:  

kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}  

An ingress controller controls the traffic into the Kubernetes cluster. Typically, this is done with the help of nginx. An Ingress-nginx controller can be configured to use the certificate.  

Nginx provides the option to specify a –default-ssl-certificate.  The default certificate is used as a catch-all for all traffic in the server. Nginx also provides a –enable-ssl-passthrough feature This bypasses the nginx and the controller instead pipes it forward and backward between the client and backend. If the virtual domain cannot be resolved, it passes to the default backend.  

Therefore the two steps required to secure the ingress controller include the following: 

1. Use a library such as cert-manager to generate keys and certificates. 

2. Use the generated key and certificates as Kubernetes secrets and generate the keystore and truststore whose location is specified in the SSL configuration of the application. 

The makings of PKCS12 

Keys and certificates are used to secure data by using the public key to encrypt and the private key to decrypt. A certificate is used as a stamp of authority. It can include the public key. 

Keys and certificates are often exchanged in PEM encoding. It is a format for a container that seems to have originated from Privacy Enhanced Mail which has long been abandoned. The container format proved useful. Keys and certificates in PEM encoding are easy to recognize with their distinctive BEGIN and END literals. 

PKCS12 is a more recent container format and one that can store one or more cryptography artifacts in what are called bags. It is made available in the form of a file with pfx extension or p12 extension. Although it can store keys, certificates, root and chains or any combinations of these, it is typically understood as a type of keystore which is a format widely accepted by applications written in Java to denote a combination of a  key and certificate. Together, this keystore can secure mutual authentication. We will go into mutual authentication a little later but first let us discuss this format a little bit more. It has been surprisingly fragile unless the packing and unpacking are done by the same tool which goes by the name keytool.  Usually, the keytool creates the keystore and the openssl tool unpacks or verifies it. 

A keystore also has an older and perhaps now deprecated format. We don’t need to go over that since the pkcs12 is more recent. By that same reasoning, it is yet to infuse into the libraries and sdks in all languages for applications that generally don’t want the inconvenience of knowing anything beyond a key and a certificate. 

PKCS12 format is a representation of an authenticated safe with one or more safe bags. There are many types of safe bag: some are used to store keys transparently, others are used as bags to shroud keys, some store certificates, others store certificate revocation lists and generally bags that can store any miscellaneous or future extensions. The format itself was meant as a simple collection of bags but unlike archive formats, there is no inbuilt verification to the count, size and consistency of bags or their contents. This makes handling very complicated at times involving nested switch case. OpenSSL also does a poor job of making all fields as optional which often breaks when fields don’t align or tags are not recognized and bags or contents cannot be skipped often resulting in errors like: 

"error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag  

error:0D07803A: asn1 encoding routines: ASN1_ITEM_EX_D2I: nested asn1 error"  

We mentioned mutual authentication earlier. It is a method where the server and the client present different certificates. In this sequence of message exchanges for mutual authentication between the server and the client, the server initiates the messages. First, the server sends hello message.  Next it sends the certificate, followed by a request to get the client’s certificate and lastly the server-side hello done message. The client responds first with its certificate. Then it sends the session key with the client key exchange message. Then it sends the certificate verify message and changes the cipher spec. Lastly it sends the client-side finished message. The server closes the mutual authentication with the cipher changed message and the server-side finished message.  

Algorithms used for creating the keys are called digital signature algorithms. There are two kinds of encryption algorithms - RSA and ECDSA. In both cases, a message signed with the public key can only be opened with the help of the corresponding private key. RSA has historically been more popular with ECDSA gaining support only recently. They are usually compared in terms of bits to denote their security level. Bits is the number of steps taken before an attacker can compromise the security. A 2048-bit RSA public key has a security level of 112 bits. ECDSA needs only 224-bit sized public keys to provide the same security level which provides efficiency for storage. Signing and verification of the signature constitute the two most costly steps performed. The input size plays into this cost for embedded devices.  
Before we compare keystore and truststore with the applications of these algorithm types, let us make a few definitions clear for our discussions:
a server-cert: is the certificate that the server gives the client
a server-key: is the private key associated with the server-cert
a server-ca-cert: is the certificate of the Certificate Authority that signed the server-cert
a client-cert: is the certificate that the client presents to the server
a client-key: is the private key that corresponds to the client-cert
a client-ca-cert: is the certificate that the CA used to sign the client-cert.
The keystore has a private key and the certificate is usually of the same type as the algorithm used with the private key. A keystore can be jks or pkcs12. The former is older and deprecated while the latter is used. pkcs12 had a notable bug  A pkcs12 file that contains only the client-ca-cert does not import into a java keystore. It was called the pkcs12 loading bug. This results in a broken pipe exception on the client side and a no trusted certificate found on the server side. It happens because the server has sent a certificate request but does not include any domain names with which the server can look them up. 
#codingexercise

Node GetSecondLargestInBST (Node root){ 

Node largest = GetRightmost (root); 

Return  GetPredecessor(largest); 

} 

Algorithms used for creating the keys are called digital signature algorithms. There are two kinds of encryption algorithms - RSA and ECDSA. In both cases, a message signed with the public key can only be opened with the help of the corresponding private key. RSA has historically been more popular with ECDSA gaining support only recently. They are usually compared in terms of bits to denote their security level. Bits is the number of steps taken before an attacker can compromise the security. A 2048-bit RSA public key has a security level of 112 bits. ECDSA needs only 224-bit sized public keys to provide the same security level which provides efficiency for storage. Signing and verification of the signature constitute the two most costly steps performed. The input size plays into this cost for embedded devices. 

Kubernetes takes keys, certificates, keystores and truststores as secrets. For example, we can specify: 

kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE} 

An ingress controller controls the traffic into the Kubernetes cluster. Typically, this is done with the help of nginx. An Ingress-nginx controller can be configured to use the certificate. 

Nginx provides the option to specify a –default-ssl-certificate.  The default certificate is used as a catch-all for all traffic in the server. Nginx also provides a –enable-ssl-passthrough feature This bypasses the nginx and the controller instead pipes it forward and backward between the client and backend. If the virtual domain cannot be resolved, it passes to the default backend. 

Strict transport requirement on nginx enforces a redirect for all http traffic to https. Certificate management is necessary for this purpose. What used to be kube-lego and is now cert-manager automatically renews expired certificates.