WorkloadRule CRD

Workload is something what is running in the tenant cluster. For example, workload can represent Kubernetes Pod, Openstack VM or something else. Currently the only supported tenant orchestrator is Kubernetes and the only supported resource type is Pod.

Notification which contains Workload objects are transferred by tenant plugins from the tenant to the infrastructure cluster. universe.workload.v1 GRPC API used for this purpose.

Workload object contains following fields:

Copy
Copied!
            

# contains meta information about workload metadata: # unique workload ID, e.g. Pod resource UID for k8s id: aac076f4-a565-4ed0-aac2-3da5698f5a78 # orchestrator type, the only supported orchestrator for now is kubernetes orchestrator: kubernetes # resource type identifier in orchestrator, the only supported resourceType for now is v1/Pod resourceType: v1/Pod # resource name in the tenant cluster resourceName: nginx # resource namespace in the tenant cluster resourceNamespace: default # contains description of the object state state: # name of the tenant node on which this workload is started nodeName: cloud-dev-12 # indicate if workload is in ready state ready: true # orchestrator-specific information which will help to identify resource and describe its state extra: # if Tenant cluster is k8s, extra field will always include # all labels and annotations which are set for resource in the Tenant cluster labels: k8s-app: nginx annotations: k8s.v1.cni.cncf.io/networks-status: |- [{ "name": "default/ovn-primary", "interface": "eth0", "ips": [ "192.0.1.9" ], "mac": "0a:58:c0:00:01:09", "default": true, "dns": {} }]

Pod field to Workload object field mapping

Workload object field

Pod field

metadata.id metadata.uid
metadata.orchestrator, always kubernetes
metadata.resourceType, always v1/Pod
metadata.resourceName metadata.name
metadata.resourceNamespace metadata.namespace
state.nodeName spec.nodeName
state.ready status.conditions - Ready condition
state.extra.labels metadata.labels
state.extra.annotations metadata.annotations

WorkloadRule CR creation in the tenant cluster will trigger creation of a tenant driven workload rule in infrastructure which means the following. WorkloadRule means the following: If Workload which match conditions defined in spec.resourceKind, spec.workloadTerms exist in the tenant cluster, then resource defined in spec.template will be created in the infrastructure cluster.

Copy
Copied!
            

apiVersion: workload.universe.nvidia.com/v1alpha1 kind: WorkloadRule metadata: name: rule1 namespace: universe spec: # select workloads with matching .metadata.resourceType # the only supported type for now is v1/Pod resourceKind: v1/Pod # include terms to select workloads # can include multiple matchExpressions objects, # matchExpressions objects are ORed. workloadTerms: - matchExpressions: # rules inside match expression are ANDed # match expression works same way as expressions # in nodeAffinity config for Pod # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity # key should contain workload field path in JSONPath format - key: .state.nodeName # supported operators are: In, NotIn, Exists, DoesNotExist, Gt and Lt operator: In # values to compare with values: - cloud-dev-12 - cloud-dev-13 - key: .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status'] operator: Exists - key: .state.extra.labels.k8s-app operator: NotIn values: - nginx - key: .metadata.resourceNamespace operator: In values: - default workloadInfoInject: # path in workload object in JSONPath format - workloadKey: .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status'] # asAnnotation is the only supported method for workloadInfoInject for now asAnnotation: # value from .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status'] # will be injected as annotation with secondary-network-status name name: secondary-network-status - workloadKey: .state.nodeName asAnnotation: name: tenant-node-name # no workloadKey or empty workloadKey mean include entire object - asAnnotation: # annotation key # as result entire workload object will be rendered as JSON and included in # annotation with name entire-workload name: entire-workload # defines DPU selection policy, can be SameNode, Any, # SameNode - run Pod on a DPU which installed to the node on which workload is running # Any - run Pod on any DPU # default policy is SameNode # additional nodeSelector and NodeAffinity rules can be defined in template section, # policy from dpuSelectionPolicy will be ANDed with settings from template dpuSelectionPolicy: SameNode # contains template for the single k8s resource template: apiVersion: v1 kind: Pod metadata: # name will be ignored, resource will be created using with name selected by infrastructure cluster name: does-not-matter # namespace will be set automatically to match tenant namespace namespace: does-not-matter spec: # additional node selector rules for the Pod, # this selector will be merged with selector generated by dpuSelectionPolicy and # some internal selectors which help to achieve isolation between tenants nodeSelector: foo: bar containers: - name: nginx image: nginx:1.14.2 env: - name: TENANT_NODE_NAME valueFrom: fieldRef: # inject env variable with downward API from annotation # defined in workloadInfoInject section fieldPath: metadata.annotations['tenant-node-node'] volumeMounts: - name: workload-info mountPath: /workload-info # standard k8s way to mount downwardAPI info as a volume volumes: - name: workload-info downwardAPI: items: # contains entire workload as JSON - path: workload fieldRef: fieldPath: metadata.annotations["entire-workload"] - path: secondary-network-status fieldRef: fieldPath: metadata.annotations["secondary-network-status"] status: # this field is used in the Tenant cluster to reflect synchronization status with iCP # can be "success" or "unknown", success mean that sync loop was able to read info # from the iCP cluster and information in the Tenant cluster is probably up to date. # unknown status mean that sync loop was not able to read info from the iCP cluster # for some time and information in the tenant cluster can be outdated syncResult: success

Previous UVSConfigMap CRD
Next Helm Charts
© Copyright 2023, NVIDIA. Last updated on Feb 7, 2024.