Ideas for a graceful shutdown of an application hosted on Kubernetes orchestration framework (K8s) continued...

Even a script launched with dumb-init allows the message propagation to the programs as shown below:

#!/usr/bin/dumb-init /bin/sh

aBackgroundProcess &  # launch a process in the background

aForegroundProcess      # launch another process in the foreground

And the change to the Dockerfile is minimal as shown below:

ENTRYPOINT [“/usr/local/bin/dumb-init”, “--”]

CMD [“path/to/file”]

Note that the json syntax above is necessary and the path to the file should be a full literal otherwise the expanded “sh -c” does not propagate the message.

The terminationGracePeriod, if specified, must be greater than the duration of wait within any preStop hook execution.

8. Eighth, there are special capabilities with statefulset which include the following:

a. They can be used to create replicas when pods are being deployed. The pods are created sequentially in order from 0 to N-1 When the pods are deleted, they are terminated in the reverse order

b. They can be used to created ordered and graceful scaling. All of the predecessors are ensured to be ready and running prior to scaling

c. Before a pod is terminated, all of its successors must be completely shut down.

The above set of guarantees is referred to as the “OrderedReady” pod management.

There is also parallel pod management which does not chain the pods.

Statefulset can also be used to perform rolling updates. This is one case where healthy pods may be terminated. Kubernetes slowly terminates old pods while spinning up new ones. If a node is drained, Kubernetes terminates all the pods on that node. If a node runs out of resources, pods may be terminated to free some resource. While we discussed SIGTERM and preStop hook, we have not discussed an appropriate limit for the terminationGracePeriodSeconds  on the pod spec. This is typically set to 30 or 60 seconds but it merely has to be greater than the duration of running all the chained handlers for the termination messages

Please note that the use of “lifecycle: command: ” scripts in postStart and preStop. These should ideally not use “/bin/sh -c” because they don’t pass messages. It is preferable to either use dumb-init or actual executable that handles ^C event.