Sharing Storage between Pod Replicas

Housekeeping

See the Getting Started documentation for installing and setting up:

microk8s
UCS Tools
- Accessing NGC
- Setting up repositories

Introduction

In this tutorial, we will show how to share storage between Pod Replicas of a microservice.

This tutorial is a continuation of the tutorial Adding Storage.

Updating the microservices and application

No special changes are required in the manifest for the sharing storage to work. However, we will make some changes in the manifest to demonstrate that the storage is accessible by each of the pod replicas.

Update the microservice manifest of the HTTP server microservice http-server to the following:

type: msapplication
specVersion: 2.5.0
name: ucf.svc.http-server
chartName: http-server
description: http server
version: 0.0.3
tags: []
keywords: []
publish: false
ingress-endpoints:
  - name: http
    description: REST API endpoint
    protocol: TCP
    scheme: http
    mandatory: False
    data-flow: in-out
---
spec:
  - name: http-server-deployment
    type: ucf.k8s.app.deployment
    parameters:
      apptype: stateless

  - name: http-server-container
    type: ucf.k8s.container
    parameters:
      image:
        repository: nvcr.io/nvidia/pytorch
        tag: 22.04-py3
      command: [sh, -c]
      args: [
            "cd /localvol && echo $PWD && touch $POD_NAME.txt && ls && python -m http.server 8080
            "]
      env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      ports:
        - containerPort: 8080
          name: http
      volumeMounts:
        - name: localvol
          mountPath: /localvol

  - name: svc
    type: ucf.k8s.service
    parameters:
      ports:
      - port: 8080
        protocol: TCP
        name: http

  - name: localvol
    type: ucf.k8s.volume
    parameters:
      persistentVolumeClaim:
        claimName: local-path-pvc

There are two changes to the manifest file:

Current pod name is exposed as an environent variable POD_NAME inside the container
The container args have been updated to create an empty file <pod-name>.txt

Update the microservice manifest of the Curl Client microservice curl-client to the following:

type: msapplication
specVersion: 2.5.0
name: ucf.svc.curl-client
chartName: curl-client
description: Curl Client
version: 0.0.2
tags: []
keywords: []
publish: false

egress-endpoints:
  - name: http
    description: REST API endpoint
    protocol: TCP
    scheme: http
    mandatory: False
    data-flow: in-out

---
spec:
  - name: curl-client-deployment
    type: ucf.k8s.app.deployment
    parameters:
      apptype: stateless

  - name: curl-client-container
    type: ucf.k8s.container
    parameters:
      image:
        repository: nvcr.io/nvidia/pytorch
        tag: 22.04-py3
      command: [sh, -c]
      args: [
            "while true; do sleep 10 && curl $egress.http.address:$egress.http.port; done
            "]

There is a single change to the manifest file:

The container arguments have been updated to periodically call the HTTP api every 10 seconds

In the app.yaml file to the following:

specVersion: 2.5.0
version: 0.0.3
doc: README.md
name: server-client-app
description: Server Client Application

dependencies:
- ucf.svc.curl-client:0.0.2
- ucf.svc.http-server:0.0.3

components:
- name: client
  type: ucf.svc.curl-client
- name: http-server
  type: ucf.svc.http-server

connections:
    client/http: http-server/http

The version of the http-server, curl-client and the app has been updated.

Building Microservices and Applications and Deploying them

Follow the steps mentioned below but remember to update the version of the http-server and curl-client under dependencies section in app.yaml.

Connecting Microservices - Building Microservices and Applications to build the services and app using CLI
Connecting Microservices - Creating and Building Application using Studio to create and build the app visually using Studio
Connecting Microservices - Deploying and Running Microservices and Application - to deploy the app

Inspecting and Debugging Microservices and Application

Let’s verify that the application was deployed successfully:

$ microk8s kubectl get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/curl-client-curl-client-deployment-77d5f5465d-jfw47   1/1     Running   0          11s
pod/http-server-http-server-deployment-6bb7754dfc-kw2dh   1/1     Running   0          11s

NAME                                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1    <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.80   <none>        8080/TCP   11s

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           11s
deployment.apps/http-server-http-server-deployment   1/1     1            1           11s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-77d5f5465d   1         1         1       11s
replicaset.apps/http-server-http-server-deployment-6bb7754dfc   1         1         1       11s

We can check the http-server container to see that a file named <pod-name>.txt was created in the current working directory - in this case the /localvol folder.

$ microk8s kubectl logs --tail -1 -l "app=http-server-http-server-deployment"
/localvol
http-server-http-server-deployment-6bb7754dfc-kw2dh.txt
somefile.txt

We can also get logs to verify if the file was created in the mounted volume:

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   481  100   481    0     0   156k      0 --:--:-- --:--:-- --:--:--  156k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="http-server-http-server-deployment-6bb7754dfc-kw2dh.txt">http-server-http-server-deployment-6bb7754dfc-kw2dh.txt</a></li>
<li><a href="somefile.txt">somefile.txt</a></li>
</ul>
<hr>
</body>
</html>

As we can see http-server-http-server-deployment-6bb7754dfc-kw2dh.txt shows up in the client, meaning that the file was indeed created in the mounted file path.

Now lets manually scale up the pod replicas to 3:

$ microk8s kubectl scale deploy/http-server-http-server-deployment --replicas=3

Lets confirm that there are 2 new pod replicas:

$ microk8s kubectl get all
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/curl-client-curl-client-deployment-77d5f5465d-jfw47   1/1     Running   0          11m
pod/http-server-http-server-deployment-6bb7754dfc-kw2dh   1/1     Running   0          11m
pod/http-server-http-server-deployment-6bb7754dfc-q68wq   1/1     Running   0          19s
pod/http-server-http-server-deployment-6bb7754dfc-xdb6p   1/1     Running   0          19s

NAME                                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes                               ClusterIP   10.152.183.1    <none>        443/TCP    49d
service/http-server-http-server-deployment-svc   ClusterIP   10.152.183.80   <none>        8080/TCP   11m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/curl-client-curl-client-deployment   1/1     1            1           11m
deployment.apps/http-server-http-server-deployment   3/3     3            3           11m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/curl-client-curl-client-deployment-77d5f5465d   1         1         1       11m
replicaset.apps/http-server-http-server-deployment-6bb7754dfc   3         3         3       11m

Lets check the logs of the curl-client pod. We must wait for all the http-server pods to go into the RUNNING state plus an additoional 10 seconds for the curl client pod to execute the curl command.

$ microk8s kubectl logs --tail -1 -l "app=curl-client-curl-client-deployment"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   751  100   751    0     0   183k      0 --:--:-- --:--:-- --:--:--  183k
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="http-server-http-server-deployment-6bb7754dfc-kw2dh.txt">http-server-http-server-deployment-6bb7754dfc-kw2dh.txt</a></li>
<li><a href="http-server-http-server-deployment-6bb7754dfc-q68wq.txt">http-server-http-server-deployment-6bb7754dfc-q68wq.txt</a></li>
<li><a href="http-server-http-server-deployment-6bb7754dfc-xdb6p.txt">http-server-http-server-deployment-6bb7754dfc-xdb6p.txt</a></li>
<li><a href="somefile.txt">somefile.txt</a></li>
</ul>
<hr>
</body>
</html>

As we can see that there is a file corresponding to each pod replica of http-server. This confirms that the same storage (PVC) was mounted in all the pod replicas.

Stopping and Cleaning up Microservices and Applications

Finally, to stop and clean up the application we can run:

$ microk8s helm3 uninstall server-client

Some Notes

The microservice must handle read/write access contention to the shared storage itself. Kubernetes does not provide any in-built mechanism.
Persistent Volumes created by the Local Path Provisioner can only be shared by pods running on the same node
- Pod replicas mounting PVC with mdx-local-path storageClass will never get scheduled on different node because of nodeAffinity of the backing Persistent Volume
- This can lead to pod replicas not getting scheduled at all if the node where the PV has been created has insufficent resources to launch more pods.
To share storage between pod replicas scheduled on different nodes use a different PV provisioner that allows PVs to be mounted across nodes.
- One such example is Kubernetes NFS Subdir External Provisioner.