Partial Redeployment of App#

Housekeeping#

See the Getting Started documentation for installing and setting up:

  • microk8s

  • UCS Tools

    • Accessing NGC

    • Setting up repositories

Introduction#

In this tutorial, we will show how to partially redeploy only the updated resources in the application using helm upgrade command.

Some other points of interest in this tutorial are:

  • Adding parameters to a microservice and setting them from the application

  • Using the configs directory

This tutorial is a continuation of the tutorial Sharing Storage between Pod Replicas.

Updating the microservices and application#

Create a file called config.json in configs directory of http-server with following contents:

{
          "workingDir": "$params.workingDir"
}

Files added under the configs directory get automatically added as a configmap and mounted to /opt/configs directory in the microservice containers. Here we have added a field workingDir set to a param workingDir. The $params placeholder will automatically be replaced with the actual value during application build.

Update the microservice manifest of the HTTP server microservice http-server to the following:

type: msapplication
specVersion: 2.5.0
name: ucf.svc.http-server
chartName: http-server
description: http server
version: 0.0.4
tags: []
keywords: []
publish: false
ingress-endpoints:
  - name: http
    description: REST API endpoint
    protocol: TCP
    scheme: http
    mandatory: False
    data-flow: in-out

params:
  workingDir: "/localvol"
  #> type: string
  #> description: Working directory for listing contents
  #> flags: mandatory
  fileToCreate: somefile.txt
  #> type: string
  #> description: Name of the file to create in the working directory
  #> flags: mandatory
---
spec:
  - name: http-server-deployment
    type: ucf.k8s.app.deployment
    parameters:
      apptype: stateless

  - name: http-server-container
    type: ucf.k8s.container
    parameters:
      image:
        repository: nvcr.io/nvidia/pytorch
        tag: 22.04-py3
      command: [sh, -c]
      args: [
            "WORKING_DIR=$(cat /opt/configs/config.json | jq -r '.workingDir') && cd $WORKING_DIR && echo $PWD && touch $(FILE_TO_CREATE) && ls && python -m http.server 8080
            "]
      env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: FILE_TO_CREATE
        value: $params.fileToCreate
      ports:
        - containerPort: 8080
          name: http
      volumeMounts:
        - name: localvol
          mountPath: /localvol

  - name: svc
    type: ucf.k8s.service
    parameters:
      ports:
      - port: 8080
        protocol: TCP
        name: http

  - name: localvol
    type: ucf.k8s.volume
    parameters:
      persistentVolumeClaim:
        claimName: local-path-pvc

There are three changes to the manifest file:

  • Two parameters named workingDir and fileToCeate have been added along with some annotations (type, description, flags.)

  • A new environment variable FILE_TO_CREATE has been added with value set to $params.fileToCreate placeholder.

  • The container arguments have been updated to read workingDir from the config.json file, change into the directory, create a file with the name passed as microservice parameter fileToCreate and list the contents of the directory.

Update the app.yaml file to the following:

specVersion: 2.5.0
version: 0.0.4
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.4

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server
  parameters:
    workingDir: /usr/local
    fileToCreate: myfile.txt

connections:
    client/http: http-server/http

The parameters workingDir and fileToCreate have been set on the http-server and the version of the http-server and the app has been updated.

Building Microservices and Applications and Deploying them#

Follow the steps mentioned below but remember to update the version of the http-server and curl-client under dependencies section in app.yaml.

Inspecting and Debugging Microservices and Application#

Let’s verify that the application was deployed successfully:

$ microk8s kubectl get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2    1/1     Running   0          17s
pod/http-server-http-server-deployment-5cfdc4bfc9-cmfbq   1/1     Running   0          17s

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   21s

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           19s
deployment.apps/http-server-http-server-deployment   1/1     1            1           19s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       18s
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9   1         1         1       18s

We can check the http-server container logs to see that current working directory was /usr/local and a file named myfile.txt was created as specified in the application.

$ microk8s kubectl logs --tail -1 -l "app=http-server-http-server-deployment"
/usr/local
bin
cuda
cuda-11
cuda-11.4
cuda-11.6
etc
games
include
lib
man
mpi
myfile.txt
nvm
sbin
share
src
ucx

We can also get logs to verify if the file was created in the mounted volume:

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   936  100   936    0     0   457k      0 --:--:-- --:--:-- --:--:--  457k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="bin/">bin/</a></li>
<li><a href="cuda/">cuda@</a></li>
<li><a href="cuda-11/">cuda-11@</a></li>
<li><a href="cuda-11.4/">cuda-11.4/</a></li>
<li><a href="cuda-11.6/">cuda-11.6/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="games/">games/</a></li>
<li><a href="include/">include/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="man/">man@</a></li>
<li><a href="mpi/">mpi@</a></li>
<li><a href="myfile.txt">myfile.txt</a></li>
<li><a href="nvm/">nvm/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="share/">share/</a></li>
<li><a href="src/">src/</a></li>
<li><a href="ucx/">ucx@</a></li>
</ul>
<hr>
</body>
</html>

As we can see myfile.txt and some other files inside /usr/local directory show up in the client, meaning that both the parameter values set by the application took effect.

Redeploying the application#

Let’s update the parameter value of fileToCreate in the application:

specVersion: 2.5.0
version: 0.0.4
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.4

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server
  parameters:
    workingDir: /usr/local
    fileToCreate: someotherfile.txt

connections:
    client/http: http-server/http

Rebuild the app using:

$ cd $HOME/ucf_tutorial/apps
$ ucf_app_builder_cli app build server-client-app/ -f

To redeploy only the updated resources of the application (partial redeployment), use the helm upgrade command:

$ microk8s helm upgrade server-client server-client-app/server-client-app-0.0.4

Lets reinspect the kubernetes resources and pods:

$ microk8s kubectl get all
NAME                                                      READY   STATUS        RESTARTS   AGE
pod/curl-client-curl-client-deployment-569849964-shcc2    1/1     Running       0          3m40s
pod/http-server-http-server-deployment-7df656c66f-mbl2z   1/1     Running       0          9s
pod/http-server-http-server-deployment-5cfdc4bfc9-cmfbq   1/1     Terminating   0          3m40s

NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.245   <none>        8080/TCP   3m44s

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           3m42s
deployment.apps/http-server-http-server-deployment   1/1     1            1           3m42s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-569849964    1         1         1       3m41s
replicaset.apps/http-server-http-server-deployment-7df656c66f   1         1         1       9s
replicaset.apps/http-server-http-server-deployment-5cfdc4bfc9   0         0         0       3m41s

It can be seen that the change in the parameter value of http-server microservice caused it’s existing pod to terminate and launch a new pod with update value, while other resources which were not changed (e.g curl-client microservice pods) are not affected.

We can check the container logs of the new pod of http-server to see that a file named someotherfile.txt was created as specified in the updated application.

$ microk8s kubectl logs --tail -1 -l "app=http-server-http-server-deployment,microservice_version=0.0.4"
/usr/local
bin
cuda
cuda-11
cuda-11.4
cuda-11.6
etc
games
include
lib
man
mpi
nvm
sbin
share
someotherfile.txt
src
ucx

Lets check the latest logs of the curl-client pod:

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   922  100   922    0     0   450k      0 --:--:-- --:--:-- --:--:--  450k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="bin/">bin/</a></li>
<li><a href="cuda/">cuda@</a></li>
<li><a href="cuda-11/">cuda-11@</a></li>
<li><a href="cuda-11.4/">cuda-11.4/</a></li>
<li><a href="cuda-11.6/">cuda-11.6/</a></li>
<li><a href="etc/">etc/</a></li>
<li><a href="games/">games/</a></li>
<li><a href="include/">include/</a></li>
<li><a href="lib/">lib/</a></li>
<li><a href="man/">man@</a></li>
<li><a href="mpi/">mpi@</a></li>
<li><a href="nvm/">nvm/</a></li>
<li><a href="sbin/">sbin/</a></li>
<li><a href="share/">share/</a></li>
<li><a href="someotherfile.txt">someotherfile.txt</a></li>
<li><a href="src/">src/</a></li>
<li><a href="ucx/">ucx@</a></li>
</ul>
<hr>
</body>
</html>

As we can see that the output lists the updated file name someotherfile.txt.

Stopping and Cleaning up Microservices and Applications#

Finally, to stop and clean up the application we can run:

$ microk8s helm3 uninstall server-client

Note

Do not clean up the application if you are continuing to the next tutorial Restarting Pods on Config Changes.

Some Notes#

  • Partial redeployment works even when microservice version is updated in the application without changing any other parameters. Resources belonging to only those microservices whose version is changed should be redeployed, other are not affected.

  • It also works when some microservices are added / removed / replaced with other microservices. To try this:

    • Build and deploy the Text Chatbot

    • Wait for the Text Chatbot application to run fully and interact with it

    • Build the Speech Chatbot

    • Deploy the Speech Chatbot application using helm upgrade command

    • Unchanged microservices Riva Speech Skills and BotMaker Dialog Manager should remain unaffected and pods continue to run without interruption.

    • Resources related to BotMaker Text Web App should be terminated while resources related to BotMaker Speech Controller and BotMaker Speech Web App be created.

    • Wait for the Speech Chatbot application to run fully and interact with it

  • Changing parameters that cause only configmap contents to be updated do not cause the microservice pods to restart. So even though the helm upgrade command succeeded, the existing pods can continue to run and use stale config files. This is addressed in the next tutorial Restarting Pods on Config Changes.