Restarting Pods on Config Changes
Housekeeping
See the Getting Started documentation for installing and setting up:
microk8s
UCS Tools
Accessing NGC
Setting up repositories
Introduction
In this tutorial, we will first demonstrate the problem mentioned in previous tutorial config changes not causing pods to restart and show how to add dependencies on configmaps so that any changes in configmaps cause pods to restart.
This tutorial is a continuation of the tutorial Partial Redeployment of App.
Inspect current microservices and resources
$ microk8s kubectl get all
NAME READY STATUS RESTARTS AGE
pod/curl-client-curl-client-deployment-569849964-shcc2 1/1 Running 0 44m
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk 1/1 Running 0 2m16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 49d
service/http-server-http-server-deployment-svc ClusterIP 10.152.183.245 <none> 8080/TCP 44m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/curl-client-curl-client-deployment 1/1 1 1 44m
deployment.apps/http-server-http-server-deployment 1/1 1 1 44m
NAME DESIRED CURRENT READY AGE
replicaset.apps/curl-client-curl-client-deployment-569849964 1 1 1 44m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9 1 1 1 44m
Let’s also inspect the configmap contents where the workingDir parameter was added.
$ microk8s kubectl get cm -l "app.kubernetes.io/instance=server-client"
NAME DATA AGE
http-server-workload-cm 0 49m
curl-client-workload-cm 0 49m
http-server-configs-cm 2 49m
http-server-scripts-cm 1 49m
curl-client-scripts-cm 1 49m
curl-client-configs-cm 1 49m
$ microk8s kubectl get cm http-server-configs-cm -o yaml
apiVersion: v1
data:
config.json: |
{
"workingDir": "/usr/local"
}
config.yaml: |
SampleConfig:
sampleValue: 0
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: server-client
meta.helm.sh/release-namespace: default
creationTimestamp: "2023-03-14T06:05:18Z"
labels:
app.kubernetes.io/instance: server-client
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: http-server
app.kubernetes.io/version: 0.0.4
generated_with: helm_builder
hb_version: 0.7.1
helm.sh/chart: http-server-0.0.4
msb_version: 2.5.0
name: http-server-configs-cm
namespace: default
resourceVersion: "954500"
uid: 8bdae84f-84b0-411b-a7aa-337cab82bdc2
As we can see config.json contains the current path of /usr/local.
Updating the parameters in the application
Update the app.yaml file to the following:
specVersion: 2.5.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application
dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.4
components:
- name: client
type: ucf.svc.curl-client
- name: http-server
type: ucf.svc.http-server
parameters:
workingDir: /var
fileToCreate: someotherfile.txt
connections:
client/http: http-server/http
The parameter workingDir has been modified and the app version has been updated.
Building the Application and Partial Redeployment
Rebuild and redeploy the app using:
$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5
Inspecting the Microservices and Application
$ microk8s kubectl get all
NAME READY STATUS RESTARTS AGE
pod/curl-client-curl-client-deployment-569849964-shcc2 1/1 Running 0 56m
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk 1/1 Running 0 15m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 49d
service/http-server-http-server-deployment-svc ClusterIP 10.152.183.245 <none> 8080/TCP 57m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/curl-client-curl-client-deployment 1/1 1 1 56m
deployment.apps/http-server-http-server-deployment 1/1 1 1 56m
NAME DESIRED CURRENT READY AGE
replicaset.apps/curl-client-curl-client-deployment-569849964 1 1 1 56m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9 1 1 1 56m
replicaset.apps/http-server-http-server-deployment-7df656c66f 0 0 0 53m
We can also get logs of the curl-client container:
$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 936 100 936 0 0 457k 0 --:--:-- --:--:-- --:--:-- 457k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="bin/">bin/</a></li>
<li><a href="cuda/">cuda@</a></li>
<li><a href="cuda-11/">cuda-11@</a></li>
<li><a href="cuda-11.4/">cuda-11.4/</a></li>
<li><a href="cuda-11.6/">cuda-11.6/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="games/">games/</a></li>
<li><a href="include/">include/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="man/">man@</a></li>
<li><a href="mpi/">mpi@</a></li>
<li><a href="nvm/">nvm/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="share/">share/</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="src/">src/</a></li>
<li><a href="ucx/">ucx@</a></li>
</ul>
<hr>
</body>
</html>
Let’s also get the contents of the http-server configmap:
$ microk8s kubectl get cm http-server-configs-cm -o yaml
apiVersion: v1
data:
config.json: |
{
"workingDir": "/var"
}
config.yaml: |
SampleConfig:
sampleValue: 0
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: server-client
meta.helm.sh/release-namespace: default
creationTimestamp: "2023-03-14T06:05:18Z"
labels:
app.kubernetes.io/instance: server-client
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: http-server
app.kubernetes.io/version: 0.0.4
generated_with: helm_builder
hb_version: 0.7.1
helm.sh/chart: http-server-0.0.4
msb_version: 2.5.0
name: http-server-configs-cm
namespace: default
resourceVersion: "963732"
uid: 8bdae84f-84b0-411b-a7aa-337cab82bdc2
We can see that the existing pod of http-server continues to run and return directory contents of old path /usr/local even though the config
reflects the updated path of /var.
Adding Dependency on configmaps
Lets update the manifest of http-server:
type: msapplication
specVersion: 2.5.0
name: ucf.svc.http-server
chartName: http-server
description: http server
version: 0.0.5
tags: []
keywords: []
publish: false
ingress-endpoints:
- name: http
description: REST API endpoint
protocol: TCP
scheme: http
mandatory: False
data-flow: in-out
params:
workingDir: "/localvol"
#> type: string
#> description: Working directory for listing contents
#> flags: mandatory
fileToCreate: somefile.txt
#> type: string
#> description: Name of the file to create in the working directory
#> flags: mandatory
---
spec:
- name: http-server-deployment
type: ucf.k8s.app.deployment
parameters:
apptype: stateless
- name: http-server-container
type: ucf.k8s.container
parameters:
image:
repository: nvcr.io/nvidia/pytorch
tag: 22.04-py3
command: [sh, -c]
args: [
"WORKING_DIR=$(cat /opt/configs/config.json | jq -r '.workingDir') && cd $WORKING_DIR && echo $PWD && touch $(FILE_TO_CREATE) && ls && python -m http.server 8080
"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: FILE_TO_CREATE
value: $params.fileToCreate
ports:
- containerPort: 8080
name: http
volumeMounts:
- name: localvol
mountPath: /localvol
- name: svc
type: ucf.k8s.service
parameters:
ports:
- port: 8080
protocol: TCP
name: http
- name: localvol
type: ucf.k8s.volume
parameters:
persistentVolumeClaim:
claimName: local-path-pvc
- name: cm-dependencies
type: ucf.appspec.restartPodOnConfigChanges
parameters:
# Add dependency on all configmaps detected in the microservice
addAll: true
# Add dependency on default configmaps for scripts, configs and workload configs
# addDefault: true
# Add dependency on individual configmaps using the configmap names
# configMaps:
# - http-server-configs-cm
There are two changes in the manifest:
A new component
cm-dependenciesof typeucf.appspec.restartPodOnConfigChangeshas been added withaddAllset totrueMicroservice version has been updated
Update the app.yaml file to the following, only the http-server version has been updated in the dependencies:
specVersion: 2.5.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application
dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.5
components:
- name: client
type: ucf.svc.curl-client
- name: http-server
type: ucf.svc.http-server
parameters:
workingDir: /var
fileToCreate: someotherfile.txt
connections:
client/http: http-server/http
Rebuild and Redeploy the Microservice and Application
Rebuild and redeploy the microservice and app using:
$ cd $HOME/ucf_tutorial/services
$ ucf_ms_builder_cli service build -d http-server/ -f
$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5
Inspect the Redeployed Application
$ microk8s kubectl get all
NAME READY STATUS RESTARTS AGE
pod/curl-client-curl-client-deployment-569849964-shcc2 1/1 Running 0 78m
pod/http-server-http-server-deployment-d5764c767-vx6rp 1/1 Running 0 17s
pod/http-server-http-server-deployment-5cfdc4bfc9-24pkk 1/1 Terminating 0 37m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 49d
service/http-server-http-server-deployment-svc ClusterIP 10.152.183.245 <none> 8080/TCP 78m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/curl-client-curl-client-deployment 1/1 1 1 78m
deployment.apps/http-server-http-server-deployment 1/1 1 1 78m
NAME DESIRED CURRENT READY AGE
replicaset.apps/curl-client-curl-client-deployment-569849964 1 1 1 78m
replicaset.apps/http-server-http-server-deployment-7df656c66f 0 0 0 75m
replicaset.apps/http-server-http-server-deployment-d5764c767 1 1 1 17s
As we can see the existing http-server pod has been terminated and new pod launched. Checking the logs of the curl-client pod and the new
http-server pod confirms that the new path /var took effect:
$ microk8s kubectl logs --tail -1 -l "app=http-server-http-server-deployment,microservice_version=0.0.5"
/var
backups
cache
lib
local
lock
log
mail
opt
run
someotherfile.txt
spool
tmp
$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 743 100 743 0 0 362k 0 --:--:-- --:--:-- --:--:-- 362k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="backups/">backups/</a></li>
<li><a href="cache/">cache/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="local/">local/</a></li>
<li><a href="lock/">lock@</a></li>
<li><a href="log/">log/</a></li>
<li><a href="mail/">mail/</a></li>
<li><a href="opt/">opt/</a></li>
<li><a href="run/">run@</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="spool/">spool/</a></li>
<li><a href="tmp/">tmp/</a></li>
</ul>
<hr>
</body>
</html>
Update the Config Param and Rebuild and Redeploy the application
Update the app.yaml file to the following, only the workingDir parameter has been updated to /tmp:
specVersion: 2.5.0
version: 0.0.5
doc: README.md
name: server-client-app
description: Server Client Application
dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.5
components:
- name: client
type: ucf.svc.curl-client
- name: http-server
type: ucf.svc.http-server
parameters:
workingDir: /tmp
fileToCreate: someotherfile.txt
connections:
client/http: http-server/http
Rebuild and Redeploy the Application
Rebuild and redeploy the microservice and app using:
$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f
$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.5
Inspect the Redeployed Application
$ microk8s kubectl get all
NAME READY STATUS RESTARTS AGE
pod/curl-client-curl-client-deployment-569849964-shcc2 1/1 Running 0 93m
pod/http-server-http-server-deployment-c88597df9-wr89f 1/1 Running 0 21s
pod/http-server-http-server-deployment-d5764c767-vx6rp 1/1 Terminating 0 15m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 49d
service/http-server-http-server-deployment-svc ClusterIP 10.152.183.245 <none> 8080/TCP 93m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/curl-client-curl-client-deployment 1/1 1 1 93m
deployment.apps/http-server-http-server-deployment 1/1 1 1 93m
NAME DESIRED CURRENT READY AGE
replicaset.apps/curl-client-curl-client-deployment-569849964 1 1 1 93m
replicaset.apps/http-server-http-server-deployment-7df656c66f 0 0 0 90m
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9 0 0 0 93m
replicaset.apps/http-server-http-server-deployment-c88597df9 1 1 1 21s
replicaset.apps/http-server-http-server-deployment-d5764c767 0 0 0 15m
As we can see the existing http-server pod has been terminated and new pod launched. Checking the logs of the curl-client pod and the new
http-server pod confirms that the new path /tmp took effect:
$ microk8s kubectl logs http-server-http-server-deployment-c88597df9-wr89f
/tmp
core-js-banners
someotherfile.txt
v8-compile-cache-0
yarn--1650154850728-0.1942747438338328
yarn--1650154852391-0.6050681633516224
yarn--1650154853881-0.6683114706726603
yarn--1650154920633-0.4394541481795069
yarn--1650154922130-0.5297801175307726
$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 989 100 989 0 0 482k 0 --:--:-- --:--:-- --:--:-- 482k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="core-js-banners">core-js-banners</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="v8-compile-cache-0/">v8-compile-cache-0/</a></li>
<li><a href="yarn--1650154850728-0.1942747438338328/">yarn--1650154850728-0.1942747438338328/</a></li>
<li><a href="yarn--1650154852391-0.6050681633516224/">yarn--1650154852391-0.6050681633516224/</a></li>
<li><a href="yarn--1650154853881-0.6683114706726603/">yarn--1650154853881-0.6683114706726603/</a></li>
<li><a href="yarn--1650154920633-0.4394541481795069/">yarn--1650154920633-0.4394541481795069/</a></li>
<li><a href="yarn--1650154922130-0.5297801175307726/">yarn--1650154922130-0.5297801175307726/</a></li>
</ul>
<hr>
</body>
</html>
This confirms that the change in configmap contents caused the pod to restart with the help of ucf.appspec.restartPodOnConfigChanges component
in the microservice manifest.
Stopping and Cleaning up Microservices and Applications
Finally, to stop and clean up the application we can run:
$ microk8s helm3 uninstall server-client
Some Notes
Only those configmaps that are part of the microservice helm chart can be added as dependencies.