WorkloadRule CRD
Workload is something what is running in the tenant cluster. For example, workload can represent Kubernetes Pod, Openstack VM or something else. Currently the only supported tenant orchestrator is Kubernetes and the only supported resource type is Pod.
Notification which contains Workload objects are transferred by tenant plugins from the tenant to the infrastructure cluster. universe.workload.v1 GRPC API used for this purpose.
Workload object contains following fields:
# contains meta information about workload
metadata:
# unique workload ID, e.g. Pod resource UID for k8s
id: aac076f4-a565-4ed0-aac2-3da5698f5a78
# orchestrator type, the only supported orchestrator for now is kubernetes
orchestrator: kubernetes
# resource type identifier in orchestrator, the only supported resourceType for now is v1/Pod
resourceType: v1/Pod
# resource name in the tenant cluster
resourceName: nginx
# resource namespace in the tenant cluster
resourceNamespace: default
# contains description of the object state
state:
# name of the tenant node on which this workload is started
nodeName: cloud-dev-12
# indicate if workload is in ready state
ready: true
# orchestrator-specific information which will help to identify resource and describe its state
extra:
# if Tenant cluster is k8s, extra field will always include
# all labels and annotations which are set for resource in the Tenant cluster
labels:
k8s-app: nginx
annotations:
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "default/ovn-primary",
"interface": "eth0",
"ips": [
"192.0.1.9"
],
"mac": "0a:58:c0:00:01:09",
"default": true,
"dns": {}
}]
Workload object field |
Pod field |
---|---|
metadata.id | metadata.uid |
metadata.orchestrator, always kubernetes | |
metadata.resourceType, always v1/Pod | |
metadata.resourceName | metadata.name |
metadata.resourceNamespace | metadata.namespace |
state.nodeName | spec.nodeName |
state.ready | status.conditions - Ready condition |
state.extra.labels | metadata.labels |
state.extra.annotations | metadata.annotations |
WorkloadRule CR creation in the tenant cluster will trigger creation of a tenant driven workload rule in infrastructure which means the following. WorkloadRule means the following: If Workload which match conditions defined in spec.resourceKind, spec.workloadTerms exist in the tenant cluster, then resource defined in spec.template will be created in the infrastructure cluster.
apiVersion: workload.universe.nvidia.com/v1alpha1
kind: WorkloadRule
metadata:
name: rule1
namespace: universe
spec:
# select workloads with matching .metadata.resourceType
# the only supported type for now is v1/Pod
resourceKind: v1/Pod
# include terms to select workloads
# can include multiple matchExpressions objects,
# matchExpressions objects are ORed.
workloadTerms:
- matchExpressions:
# rules inside match expression are ANDed
# match expression works same way as expressions
# in nodeAffinity config for Pod
# https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
# key should contain workload field path in JSONPath format
- key: .state.nodeName
# supported operators are: In, NotIn, Exists, DoesNotExist, Gt and Lt
operator: In
# values to compare with
values:
- cloud-dev-12
- cloud-dev-13
- key: .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status']
operator: Exists
- key: .state.extra.labels.k8s-app
operator: NotIn
values:
- nginx
- key: .metadata.resourceNamespace
operator: In
values:
- default
workloadInfoInject:
# path in workload object in JSONPath format
- workloadKey: .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status']
# asAnnotation is the only supported method for workloadInfoInject for now
asAnnotation:
# value from .state.extra.annotations['k8s.v1.cni.cncf.io/networks-status']
# will be injected as annotation with secondary-network-status name
name: secondary-network-status
- workloadKey: .state.nodeName
asAnnotation:
name: tenant-node-name
# no workloadKey or empty workloadKey mean include entire object
- asAnnotation:
# annotation key
# as result entire workload object will be rendered as JSON and included in
# annotation with name entire-workload
name: entire-workload
# defines DPU selection policy, can be SameNode, Any,
# SameNode - run Pod on a DPU which installed to the node on which workload is running
# Any - run Pod on any DPU
# default policy is SameNode
# additional nodeSelector and NodeAffinity rules can be defined in template section,
# policy from dpuSelectionPolicy will be ANDed with settings from template
dpuSelectionPolicy: SameNode
# contains template for the single k8s resource
template:
apiVersion: v1
kind: Pod
metadata:
# name will be ignored, resource will be created using with name selected by infrastructure cluster
name: does-not-matter
# namespace will be set automatically to match tenant namespace
namespace: does-not-matter
spec:
# additional node selector rules for the Pod,
# this selector will be merged with selector generated by dpuSelectionPolicy and
# some internal selectors which help to achieve isolation between tenants
nodeSelector:
foo: bar
containers:
- name: nginx
image: nginx:1.14.2
env:
- name: TENANT_NODE_NAME
valueFrom:
fieldRef:
# inject env variable with downward API from annotation
# defined in workloadInfoInject section
fieldPath: metadata.annotations['tenant-node-node']
volumeMounts:
- name: workload-info
mountPath: /workload-info
# standard k8s way to mount downwardAPI info as a volume
volumes:
- name: workload-info
downwardAPI:
items:
# contains entire workload as JSON
- path: workload
fieldRef:
fieldPath: metadata.annotations["entire-workload"]
- path: secondary-network-status
fieldRef:
fieldPath: metadata.annotations["secondary-network-status"]
status:
# this field is used in the Tenant cluster to reflect synchronization status with iCP
# can be "success" or "unknown", success mean that sync loop was able to read info
# from the iCP cluster and information in the Tenant cluster is probably up to date.
# unknown status mean that sync loop was not able to read info from the iCP cluster
# for some time and information in the tenant cluster can be outdated
syncResult: success