Restarting Pods on Config Changes

Housekeeping

See the Getting Started documentation for installing and setting up:

• microk8s

• UCF tools

  • Accessing NGC

  • Setting up repositories

Introduction

In this tutorial, we will first demonstrate the problem mentioned in previous tutorial config changes not causing pods to restart and show how to add dependencies on configmaps so that any changes in configmaps cause pods to restart.

This tutorial is a continuation of the tutorial Partial Redeployment of App.

Inspect current microservices and resources

$ microk8s kubectl get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2    1/1     Running   0          44m
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk   1/1     Running   0          2m16s

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   44m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           44m
deployment.apps/http-server-http-server-deployment   1/1     1            1           44m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       44m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9   1         1         1       44m

Let’s also inspect the configmap contents where the workingDir parameter was added.

$ microk8s kubectl get cm -l "app.kubernetes.io/instance=server-client"
NAME                      DATA   AGE
http-server-workload-cm   0      49m
curl-client-workload-cm   0      49m
http-server-configs-cm    2      49m
http-server-scripts-cm    1      49m
curl-client-scripts-cm    1      49m
curl-client-configs-cm    1      49m

$ microk8s kubectl get cm http-server-configs-cm -o yaml
apiVersion: v1
data:
  config.json: |
    {
      "workingDir": "/usr/local"
    }
  config.yaml: |
    SampleConfig:
      sampleValue: 0
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: server-client
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-03-14T06:05:18Z"
  labels:
    app.kubernetes.io/instance: server-client
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: http-server
    app.kubernetes.io/version: 0.0.4
    generated_with: helm_builder
    hb_version: 0.7.1
    helm.sh/chart: http-server-0.0.4
    msb_version: 2.0.0
  name: http-server-configs-cm
  namespace: default
  resourceVersion: "954500"
  uid: 8bdae84f-84b0-411b-a7aa-337cab82bdc2

As we can see config.json contains the current path of /usr/local.

Updating the parameters in the application

Update the app.yaml file to the following:

specVersion: 2.0.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.4

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server
  parameters:
    workingDir: /var
    fileToCreate: someotherfile.txt

connections:
    client/http: http-server/http

The parameter workingDir has been modified and the app version has been updated.

Building the Application and Partial Redeployment

Rebuild and redeploy the app using:

$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5

Inspecting the Microservices and Application

$ microk8s kubectl get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2    1/1     Running   0          56m
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk   1/1     Running   0          15m

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   57m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           56m
deployment.apps/http-server-http-server-deployment   1/1     1            1           56m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       56m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9   1         1         1       56m
replicaset.apps/http-server-http-server-deployment-7df656c66f   0         0         0       53m

We can also get logs of the curl-client container:

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   936  100   936    0     0   457k      0 --:--:-- --:--:-- --:--:--  457k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="bin/">bin/</a></li>
<li><a href="cuda/">cuda@</a></li>
<li><a href="cuda-11/">cuda-11@</a></li>
<li><a href="cuda-11.4/">cuda-11.4/</a></li>
<li><a href="cuda-11.6/">cuda-11.6/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="games/">games/</a></li>
<li><a href="include/">include/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="man/">man@</a></li>
<li><a href="mpi/">mpi@</a></li>
<li><a href="nvm/">nvm/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="share/">share/</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="src/">src/</a></li>
<li><a href="ucx/">ucx@</a></li>
</ul>
<hr>
</body>
</html>

Let’s also get the contents of the http-server configmap:

$ microk8s kubectl get cm http-server-configs-cm -o yaml
apiVersion: v1
data:
  config.json: |
    {
      "workingDir": "/var"
    }
  config.yaml: |
    SampleConfig:
      sampleValue: 0
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: server-client
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-03-14T06:05:18Z"
  labels:
    app.kubernetes.io/instance: server-client
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: http-server
    app.kubernetes.io/version: 0.0.4
    generated_with: helm_builder
    hb_version: 0.7.1
    helm.sh/chart: http-server-0.0.4
    msb_version: 2.0.0
  name: http-server-configs-cm
  namespace: default
  resourceVersion: "963732"
  uid: 8bdae84f-84b0-411b-a7aa-337cab82bdc2

We can see that the existing pod of http-server continues to run and return directory contents of old path /usr/local even though the config reflects the updated path of /var.

Adding Dependency on configmaps

Lets update the manifest of http-server:

type: msapplication
specVersion: 2.0.0
name: ucf.svc.http-server
chartName: http-server
description: http server
version: 0.0.5
tags: []
keywords: []
publish: false
ingress-endpoints:
  - name: http
    description: REST API endpoint
    protocol: TCP
    scheme: http
    mandatory: False
    data-flow: in-out

params:
  workingDir: "/localvol"
  #> type: string
  #> description: Working directory for listing contents
  #> flags: mandatory
  fileToCreate: somefile.txt
  #> type: string
  #> description: Name of the file to create in the working directory
  #> flags: mandatory
---
spec:
  - name: http-server-deployment
    type: ucf.k8s.app.deployment
    parameters:
      apptype: stateless

  - name: http-server-container
    type: ucf.k8s.container
    parameters:
      image:
        repository: nvcr.io/nvidia/pytorch
        tag: 22.04-py3
      command: [sh, -c]
      args: [
            "WORKING_DIR=$(cat /opt/configs/config.json | jq -r '.workingDir') && cd $WORKING_DIR && echo $PWD && touch $(FILE_TO_CREATE) && ls && python -m http.server 8080
            "]
      env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: FILE_TO_CREATE
        value: $params.fileToCreate
      ports:
        - containerPort: 8080
          name: http
      volumeMounts:
        - name: localvol
          mountPath: /localvol

  - name: svc
    type: ucf.k8s.service
    parameters:
      ports:
      - port: 8080
        protocol: TCP
        name: http

  - name: localvol
    type: ucf.k8s.volume
    parameters:
      persistentVolumeClaim:
        claimName: local-path-pvc

  - name: cm-dependencies
    type: ucf.appspec.restartPodOnConfigChanges
    parameters:
      # Add dependency on all configmaps detected in the microservice
      addAll: true

      # Add dependency on default configmaps for scripts, configs and workload configs
      # addDefault: true

      # Add dependency on individual configmaps using the configmap names
      # configMaps:
      # - http-server-configs-cm

There are two changes in the manifest:

• A new component cm-dependencies of type ucf.appspec.restartPodOnConfigChanges has been added with addAll set to true

• Microservice version has been updated

Update the app.yaml file to the following, only the http-server version has been updated in the dependencies:

specVersion: 2.0.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.5

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server
  parameters:
    workingDir: /var
    fileToCreate: someotherfile.txt

connections:
    client/http: http-server/http

Rebuild and Redeploy the Microservice and Application

Rebuild and redeploy the microservice and app using:

$ cd $HOME/ucf_tutorial/services
$ ucf_ms_builder_cli service build -d http-server/ -f
$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5

Inspect the Redeployed Application

$ microk8s kubectl get all
NAME                                                      READY   STATUS        RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2    1/1     Running       0          78m
pod/http-server-http-server-deployment-d5764c767-vx6rp    1/1     Running       0          17s
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk   1/1     Terminating   0          37m

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   78m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           78m
deployment.apps/http-server-http-server-deployment   1/1     1            1           78m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       78m
replicaset.apps/http-server-http-server-deployment-7df656c66f   0         0         0       75m
replicaset.apps/http-server-http-server-deployment-d5764c767    1         1         1       17s

As we can see the existing http-server pod has been terminated and new pod launched. Checking the logs of the curl-client pod and the new http-server pod confirms that the new path /var took effect:

$ microk8s kubectl logs --tail -1 -l "app=http-server-http-server-deployment,microservice_version=0.0.5"
/var
backups
cache
lib
local
lock
log
mail
opt
run
someotherfile.txt
spool
tmp

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   743  100   743    0     0   362k      0 --:--:-- --:--:-- --:--:--  362k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="backups/">backups/</a></li>
<li><a href="cache/">cache/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="local/">local/</a></li>
<li><a href="lock/">lock@</a></li>
<li><a href="log/">log/</a></li>
<li><a href="mail/">mail/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="run/">run@</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="spool/">spool/</a></li>
<li><a href="tmp/">tmp/</a></li>
</ul>
<hr>
</body>
</html>

Update the Config Param and Rebuild and Redeploy the application

Update the app.yaml file to the following, only the workingDir parameter has been updated to /tmp:

specVersion: 2.0.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.5

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server
  parameters:
    workingDir: /tmp
    fileToCreate: someotherfile.txt

connections:
    client/http: http-server/http

Rebuild and Redeploy the Application

Rebuild and redeploy the microservice and app using:

$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5

Inspect the Redeployed Application

$ microk8s kubectl get all
NAME                                                     READY   STATUS        RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2   1/1     Running       0          93m
pod/http-server-http-server-deployment-c88597df9-wr89f   1/1     Running       0          21s
pod/http-server-http-server-deployment-d5764c767-vx6rp   1/1     Terminating   0          15m

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   93m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           93m
deployment.apps/http-server-http-server-deployment   1/1     1            1           93m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       93m
replicaset.apps/http-server-http-server-deployment-7df656c66f   0         0         0       90m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9   0         0         0       93m
replicaset.apps/http-server-http-server-deployment-c88597df9    1         1         1       21s
replicaset.apps/http-server-http-server-deployment-d5764c767    0         0         0       15m

As we can see the existing http-server pod has been terminated and new pod launched. Checking the logs of the curl-client pod and the new http-server pod confirms that the new path /tmp took effect:

$ microk8s kubectl logs http-server-http-server-deployment-c88597df9-wr89f
/tmp
core-js-banners
someotherfile.txt
v8-compile-cache-0
yarn--1650154850728-0.1942747438338328
yarn--1650154852391-0.6050681633516224
yarn--1650154853881-0.6683114706726603
yarn--1650154920633-0.4394541481795069
yarn--1650154922130-0.5297801175307726

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100   989  100   989    0     0   482k      0 --:--:-- --:--:-- --:--:--  482k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="core-js-banners">core-js-banners</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="v8-compile-cache-0/">v8-compile-cache-0/</a></li>
<li><a href="yarn--1650154850728-0.1942747438338328/">yarn--1650154850728-0.1942747438338328/</a></li>
<li><a href="yarn--1650154852391-0.6050681633516224/">yarn--1650154852391-0.6050681633516224/</a></li>
<li><a href="yarn--1650154853881-0.6683114706726603/">yarn--1650154853881-0.6683114706726603/</a></li>
<li><a href="yarn--1650154920633-0.4394541481795069/">yarn--1650154920633-0.4394541481795069/</a></li>
<li><a href="yarn--1650154922130-0.5297801175307726/">yarn--1650154922130-0.5297801175307726/</a></li>
</ul>
<hr>
</body>
</html>

This confirms that the change in configmap contents caused the pod to restart with the help of ucf.appspec.restartPodOnConfigChanges component in the microservice manifest.

Stopping and Cleaning up Microservices and Applications

Finally, to stop and clean up the application we can run:

$ microk8s helm3 uninstall server-client

Some Notes

• Only those configmaps that are part of the microservice helm chart can be added as dependencies.