We continue with our discussion on backup of Kubernetes resources

 is also a difference in the results of the Velero tool versus custom configuration generated using the scripts above. For example, there is no knowledge of the product and the logic pertaining to the reconcilation of the operator states in built into the output of the tool. The custom configuration on the other hand, leverages the product specific knowledge to make the export and import of user resources all the more efficient, streamlined and conformant with the product.

The above chart now provides the convenience of using helm with Kubernetes custom resources with either one resource encompassing all other required K8s resources at user namespace scope or use several discrete resources at the charts level while allowing transaction like behavior with create and delete at the overall chart level. The benefit is that resources are now grouped by user and automated for creation and deletion.
The groupings for the resources can be based on selectors. This makes the chart combine resources dynamically. Since the resource have annotations, selectors based on match can be efficiently used to group resources. The charts make it convenient to create and delete these groups of resources all at once.
The custom resource is not a dynamic selection of resources. It is a resource in itself. A custom resource may have definition to include other resources and it will make it easy to create and delete them with the help of a single resource.

The Velero tool is designed to take backups from cluster. It requires S3 storage which comes with a cloud provider such as AWS.
The install command is
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.0.0 \
    --bucket velerobucket \
    --backup-location-config region=us-east-2 \
    --snapshot-location-config region=us-east-2 \
    --secret-file /root/aws-iam-creds-csv-local \
    --log_dir /tmp/velero
And the server part can be created with helm charts. However, the backups were yet to be created so I do not have that handy.

On the other hand, I have created scripts and charts to make it easy to create and delete K8s resources.
This chart now provides the convenience of using helm with Kubernetes custom resources with either one resource encompassing all other required K8s resources at user namespace scope or use several discrete resources at the charts level while allowing transaction like behavior with create and delete at the overall chart level. The benefit is that resources are now grouped by user and automated for creation and deletion.

We continue with our discussion on backup of Kubernetes resources
 is also a difference in the results of the Velero tool versus custom configuration generated using the scripts above. For example, there is no knowledge of the product and the logic pertaining to the reconcilation of the operator states in built into the output of the tool. The custom configuration on the other hand, leverages the product specific knowledge to make the export and import of user resources all the more efficient, streamlined and conformant with the product.
The above is particularly true for custom resources and their definitions. The custom resources have a two fold utility:
1) they are a bigger scope than the native Kubernetes resources and translate export and import to simpler instructions and
2) they provide the opportunity to offload all maintenance to the reconciliation logic in-built into their corresponding operators which may even have their own assembling and disassembling in terms of native Kubernetes resources.
The overall suggestion is that scope and actions can become more granular to help with their export-import usability.
One of the challenges in registering resources is passing ip addresses for pod, host and cluster regardless of the technique used to export and import. These are dynamic values that are obtained as the import proceeds and are not available beforehand. Although it is easy to write a query to retrieve the ip address, even those queries have parameters such as pod names which do not necessarily have a pattern. This chains yet another query to retrieve the parameter. If this was limited to a few levels, it would have been easy to repeat. However not all resources are like pods, so the parameters for each type of resource have their own logic.
Similarly, another criteria is the determination of uid for the resource itself or its parent or that of the cluster. Again the determination of this parameter varies by resource whose uid is needed or the determination of the owner that may require a lookup table.
Certain values for ip and uid can be “None” but they can always be the case.
Also, the charts deploy hard-coded definitions and resources. They invoke scripts only during certain events. Each definition and its corresponding resource can be provisioned with the given values beforehand as long as we are talking about flat native K8s resources but the same resources in user namespaces may become sophisticated and complex in scope with hierarchy which requires those dependencies to be followed for each registration and deregistration.
Sample resources and definition files have been generated with the shell script shown above and they have been repeatedly modified and their import attempted to be automated to come to the enumerations above. It is very easy to tweak the scripts for a given user namespace after a few trials and use the script as a template for creating and populating say namespaces. The use of schema or auxiliary data structures to store and each every resource types and their import logic and order seems on overkill as a general purpose solution

Lies, sweet lies:
Last Friday and today I came across as raving at times, no pun intended. I was ready to find composure again in any sort of activity. Luckily, I found a book from the local library that I had brought home sometime back. It is titled “Fearless at work” by Michael Carroll. He is a Manhattanite who spent years as an executive prior to finding relevant teachings from Buddhism his calling. I presumed this book would be much of the sage and ancient wisdom that washes away in the hustle and bustle of modern day life. I was pleasantly surprised to read anecdotes one after another that held such profound connection for me.
This book teaches us to be free, confident and skillful. On any day, those words would have sounded too simple to afford any time for reflection. His writing and his references however made a tremendous difference in how I have come to realize that there is boundless abilities in being fearless.
Take the example of a Taoist story of a boy named Simha He was walking along the edge of the jungle on his way home when a tiger spotted him. The boy was clever and being mindful of where he was decided to make it to a quarry nearby. He slowed turned towards the path of safety and dashed towards the quarry. As he climbed up the quarry with the help of a vine, the tiger came to the base only a little too late. The boy’s relief was however shortlived There came another tiger at the top of the quarry ready to make its meal. As the boy hung in limbo, he couldn’t help notice a pair of mice playing in the nearby growth nibbling at the very vine that was supporting his weight. As the boy weighed his options, he saw a bright strawberry just within reach if he could swing close enough. In the midst of a desperate dismal circumstance, Simha delights in a simple gesture which is the tasting of the berry. Michael teaches that to be human is to confront our circumstance but always with the possibility of delight. He reminds us that the outcome of this story is not as important as the slogan “No delight; no courage”.
 This is essentially one of the five primary slogans Michael mentions at the outset of his book and walks us through each of these. These five are 1 Face the fierce facts of life 2 No delight; no courage 3 Recognize fear 4. Discover the jewel of fearless abundance and 5. Command gracefully. He knows we will forget teachings even if they are in the form of slogans. So he suggests that we write it down on index cards or better yet attach our own experiences to some of the slogan. These five are the primary. There are thirty-eight slogans in all. He doesn’t mind if we randomly pick a slogan from a set of cards and put it against our calendars, so we become more familiar with them.
He makes it clear this book is not about solving fear nor is it a roadmap for a fear free life. Instead he draws on the Buddhist observation that “fear does not exist”. It is merely our experience and we can overcome it with the practice of mindfulness-awareness meditation.
Among his countless stories and varied sources including the military examples of “shih”, he lays out plain and simple narratives that we immediately get instead of lengthy and numbing discussions. Among his teachings, one of my favorites is about lies. When we have absolutely nothing, we can take delight in lies and pretense if it helps us feel confident to confront the world. In this regard, I recall his example of “the wish fulfilling gem from the mystical kingdom of Shambhala” The gem has the magical ability to manifest whatever the possessor wishes. The abundance of fearlessness is like this wish fulfilling gem. One day when a worker was feeling anxious all day about a possible layoff when called by his manager, his wife called him and said “you have won a lottery of 345 million dollars”. He was more fearless and poised even in the face of bad news and would come out appearing confident and even shaking hands with the manager.
This book has been a delight and a profound learning in what even nothing can offer so much possibilities to be fearless.

We continue with the discussion of the script to export and import Kubernetes resources.
By changing the namespace, the same files can be imported again.
The export order should be resources first followed by definitions while the import order should be definitions followed by resources.
The above script only prints the commands and does not execute them.
There are two modifications that may be required to the commands.
First, the ip addresses for host, cluster and the service endpoints will need to be modified when the resources are exported from one namespace on a cluster to another. These addresses are easy to find as we make incremental progress towards the definitions and resources.
Second, the uid will need to be specified for some resources because they will pertain to existing resources in fields such as ownerReferences. This is a way to indicate that the resource will be cleaned up when the owner gets cleaned up. The ownerReferences is a way of chaining the cleanup and gives us an opportunity to do proper cleanup with a broader scope.
These two changes will help enable the resources and definitions to be recreated in the destination cluster and namespace. The commands also provide a point of reference between this method and the resulting changes provided by the Velero tool. It might be interesting to note that the latter involves specifications that may not be as lightweight and customized as the scripts above. At the same time, the tool works across deployments in general. The scripts above will work for any Kubernetes framework.

The following script rounds up the retrieval of definitions
#!/bin/bash
# retrieves definitions and resources for a user namespace
COMMAND=$1
if [ -z "$COMMAND" ]
then
  echo "Command does not exist"
  exit 1
fi

NAMESPACE=$2
if [ -z  "$NAMESPACE" ]
then
  echo "Namespace does not exist"
  exit 1
fi


# Pack
if [[ $1 == "GET" ]]; then
   kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n $NAMESPACE -o yaml > definitions.yaml
   kubectl get all -n $NAMESPACE -o yaml > resources.yaml
fi
#Unpack
if [[ $1 == "PUT" ]]; then
   kubectl create ns $NAMESPACE
   if [ ! -e "definitions.yaml" ]; then
       echo "definitions does not exist"
       exit $E_NOFILE
   fi
   sed -i.bak '/creationTimestamp/d'  definitions.yaml
   sed -i.bak '/deletionTimestamp/d' definitions.yaml
   sed -i.bak '/uid:/d' definitions.yaml
   echo kubectl create -f definitions.yaml --validate=false

   if [ ! -e "resources.yaml" ]; then
       echo "resources does not exist"
       exit $E_NOFILE
   fi
   sed -i.bak '/creationTimestamp/d' resources.yaml
   sed -i.bak '/deletionTimestamp/d' resources.yaml
   sed -i.bak '/uid:/d' resources.yaml
   sed -i.bak '/uid:/d' resources.yaml
   echo kubectl create -f resources.yaml --validate=false
fi



By changing the namespace, the same files can be imported again.
The export order should be resources first followed by definitions while the import order should be definitions followed by resources.

The dependencies between the exported resources is not obvious from their metadata since each is independently exported. In order to help with their export, their annotations could be enhanced to include a category or a level.
The order of restoring the Kubernetes resources at another point is also not known. When databases were backed up and restored, the order of creation was known from their schema. However, in this case the Kubernetes resources are independently exported and imported. Even if it is a custom resource, all that needs to be imported first is their custom resource definition.
A Custom resource application can include other definitions so there is definitely a cascading order of creation involved. This order is already determined in the charts that were used to deploy the product. The export order can be any order but the import order has to be the same as what the charts had described at deployment time.
The order of export and import do not have to be reverse of each other but it certainly helps to verify the order offline.  The attributes such as creationTimestamp, deletionTimestamp, uid and even finalizers can be removed prior to import.
A command like “kubect get all -n <user-namespace> -o yaml > user-namespace-resources.yaml” will infact determine the order appropriately which can then be directly supplied to the “kubectl create -f user-namespace-resources.yaml” as long as interfering attributes have been removed.
These won’t export custom resource definitions or the resources that can be fetched with “kubectl api-resources" nor will it setup the persistent volumes and external services that are represented via service brokers within the Kubernetes clusters. Those are assumed to be ready before the import begins.