Installation Guide

Nsight Operator installation and configuration guide.

Installation and Configuration

Quick Install

For a stock installation with the defaults, run:

helm install --wait \
    --namespace nsight-operator \
    --create-namespace \
    nsight-operator \
    https://helm.ngc.nvidia.com/nvidia/devtools/charts/nsight-operator-26.3.1.tgz

By default, profiling is enabled in Coordinator Mode for all applications running in Pods labeled with nvidia-nsight-profile: enabled. Continue with the sections below to customize the installation.

Customizing Helm Values

The NVIDIA Nsight Operator can be customized to suit particular needs. Likely, you will want to configure the nsight-injector.nsightToolConfigs and nsight-injector.injectionRules values. A values file can be used for setting these parameters.

Example Values Files

Sample custom_values.yaml. This configuration will enable profiling for any instance of yourawesomeapp found in injection Pods, enabling Python sampling and PyTorch tracing. This configuration uses Coordinator Mode.

# Nsight Systems profiling configuration (under nsight-injector sub-chart)
nsight-injector:
  nsightToolConfigs:
    - name: "default-nsight-tool-config"
      nsightToolArgs: "--python-sampling=true --cuda-graph-trace=node --pytorch=autograd-nvtx"
  injectionRules:
    - name: "has-injection-label"
      nsightToolConfigRef: "default-nsight-tool-config"
      processIncludePatterns:
        - ".*yourawesomeapp.*"
      podObjectSelector:
        matchLabels:
          nvidia-nsight-profile: enabled

helm install -f custom_values.yaml \
    --namespace nsight-operator \
    --create-namespace \
    nsight-operator https://helm.ngc.nvidia.com/nvidia/devtools/charts/nsight-operator-26.3.1.tgz

Sample custom_values_launch.yaml. This configuration switches to Launch Mode: profiling starts automatically when the target process starts and runs for a fixed duration, without any coordinator involvement. Use this mode for unattended captures with a bounded profiling window.

# Nsight Systems profiling configuration in launch mode
nsight-injector:
  nsightToolConfigs:
    - name: "default-nsight-tool-config"
      coordinator: false
      nsightToolArgs: "-t cuda,nvtx,osrt --python-sampling=true --duration=20 --kill=none"
  injectionRules:
    - name: "has-injection-label"
      nsightToolConfigRef: "default-nsight-tool-config"
      processIncludePatterns:
        - ".*yourawesomeapp.*"
      podObjectSelector:
        matchLabels:
          nvidia-nsight-profile: enabled

helm install -f custom_values_launch.yaml \
    --namespace nsight-operator \
    --create-namespace \
    nsight-operator https://helm.ngc.nvidia.com/nvidia/devtools/charts/nsight-operator-26.3.1.tgz

Sample custom_values_extended.yaml. This configuration enables profiling for any instance of yourawesomeapp running in injected Pods, except for those started with the argumenttoskip argument. Profiling is configured to collect data for a maximum duration of 20 seconds. The nsys-output-volume will be mounted to all profiled Pods. A Persistent Volume Claim must be available in the target namespaces for successful operation. Additionally, kernel.perf_event_paranoid will be set to -1 on all nodes where profiling is performed.

# Nsight Systems profiling configuration
nsight-injector:
  nsightToolConfigs:
    - name: "default-nsight-tool-config"
      nsightToolArgs: "--python-sampling=true"
  injectionRules:
    - name: "has-injection-label"
      nsightToolConfigRef: "default-nsight-tool-config"
      additionalVolumes:
        [
          {
            "name": "nsys-output-volume",
            "persistentVolumeClaim": { "claimName": "CSP-managed-disk" },
          },
        ]
      additionalVolumeMounts:
        [{ "name": "nsys-output-volume", "mountPath": "/mnt/nsys/output" }]
      processIncludePatterns:
        - ".*yourawesomeapp.*"
      processExcludePatterns:
        - ".*yourawesomeapp.*argumenttoskip.*"
      podObjectSelector:
        matchLabels:
          nvidia-nsight-profile: enabled

  machineConfig:
    - name: kernel.perf_event_paranoid
      value: -1

helm install -f custom_values_extended.yaml \
    --namespace nsight-operator \
    --create-namespace \
    nsight-operator https://helm.ngc.nvidia.com/nvidia/devtools/charts/nsight-operator-26.3.1.tgz

Sample custom_values_analysis_limits.yaml. This configuration sets the CPU and memory requests and limits for the Nsight Analysis service, which map to the NsightAnalysis resources field. Defining explicit limits is recommended when installing into a namespace that enforces a LimitRange: if the namespace’s default limit is lower than the request, Pod admission fails. Setting both requests and limits avoids that conflict. Note that if a max limit is set below the requested resource amount, the Pod is rejected during admission.

# NsightAnalysis resource requirements
nsight-analysis:
  resources:
    requests:
      cpu: 1000m
      memory: 1Gi
    limits:
      cpu: 20
      memory: 20Gi

helm install -f custom_values_analysis_limits.yaml \
    --namespace nsight-operator \
    --create-namespace \
    nsight-operator https://helm.ngc.nvidia.com/nvidia/devtools/charts/nsight-operator-26.3.1.tgz

Configuration Values

The NVIDIA Nsight Operator helm chart includes the following components:

• Operator-level values: Configure the operator controller and general settings.

• Coordinator values (nsight-coordinator.*): Configure the coordinator deployment for profiling session control.

• Gateway values (nsight-gateway.*): Configure the Envoy gateway for REST API access.

• Cloud Storage values (cloudStorage.*): Configure storage for profiling results.

• OTLP Collector values (nsight-otel-collector.*): Configure OTLP collector for trace mirroring.

• OTLP Proxy values (otlpProxyConfig.*): Configure OTLP proxy injection.

• Injector sub-chart values (nsight-injector.*): Configure injection behavior.

• STUNner TURN gateway values (nsight-tenant-operator.stunner.* and stunner-gateway-operator.*): Configure the bundled STUNner TURN gateway for Nsight Streamer WebRTC relay.

• Analysis Service values (nsight-analysis.*): Configure the analysis service for running recipes.

Operator-level Configuration

Variable	Description	Default
installation.multitenant	Enable multi-tenant mode. When true, the operator controller runs cluster-wide but control-plane components (Coordinator, Gateway, Storage, OTel Collector, Analysis) are provisioned per tenant namespace rather than in the operator namespace. With default values, the operator auto-provisions them the first time a matching Pod is admitted into a namespace; namespace admins can also pre-deploy their own resources.	false
leaderElection.enabled	Enable Kubernetes leader election for the operator controller. Safe and recommended for multi-replica deployments; harmless with a single replica.	true
leaderElection.resourceName	Name of the Lease used for leader election.	nsight-operator.nvidia.com
hostNetwork	Run the operator controller in the host network namespace. Rarely needed.	false
global.nsightCloud.distrolessPythonImage	Distroless Python image used by chart-managed Python components (operator and injector pre-delete Helm hooks, the injector node-config init container, and readiness-waiter init containers). Override this in air-gapped clusters that mirror images to a private registry so that helm install and especially helm uninstall do not hang in ImagePullBackOff. See Uninstalling Nsight Operator.	nvcr.io/nvidia/distroless/python:3.14-v4.0.9
global.nsightCloud.distrolessPythonImagePullPolicy	Pull policy for the shared distroless Python image.	IfNotPresent
global.nsightCloud.distrolessPythonImagePullSecrets	Pull secrets for the shared distroless Python image.	[]
labels	Extra labels added to the operator Deployment metadata.	{}
annotations	Extra annotations added to the operator Deployment metadata.	{}
podLabels	Extra labels added to the operator Deployment pod template metadata.	{}
podAnnotations	Extra annotations added to the operator Deployment pod template metadata.	{}
global.nsightCloud.labels	Extra labels merged into the metadata of all chart-managed resources.	{}
global.nsightCloud.annotations	Extra annotations merged into the metadata of all chart-managed resources.	{}
global.nsightCloud.schedulerConfig	Default nodeSelector, affinity, tolerations, and topologySpreadConstraints inherited by all sub-components (coordinator, gateway, analysis, streamer, OTel collector, tenant operator, cloud UI) when they do not set their own.	(empty)
global.nsightCloud.securityContext.pod	Default Pod-level securityContext inherited by all sub-components. Defaults provide runAsNonRoot: true and the RuntimeDefault seccomp profile.	non-root / seccomp
global.nsightCloud.securityContext.container	Default container-level securityContext inherited by all sub-components. Drops all capabilities and disables privilege escalation.	drop ALL

Per-Component Configuration

Each operator-managed component has its own set of configuration values for resources, scheduling, security contexts, and custom metadata. Every component supports labels, annotations, podLabels, and podAnnotations values for injecting custom metadata into the CRs and the pods the operator creates. See the Custom Resource Definition (CRD) Reference for the full list of CRD fields available per component.

Coordinator Configuration

Variable	Description	Default
nsight-coordinator.enabled	Enable deployment of the NsightCoordinator CR in the operator namespace.	true
nsight-coordinator.labels	Extra labels merged into the NsightCoordinator CR metadata and propagated to operator-created resources.	{}
nsight-coordinator.annotations	Extra annotations merged into the NsightCoordinator CR metadata and propagated to operator-created resources.	{}
nsight-coordinator.podLabels	Extra labels merged into the coordinator pod template metadata.	{}
nsight-coordinator.podAnnotations	Extra annotations merged into the coordinator pod template metadata.	{}

Additional configuration options can be specified through Helm values that map to the default NsightCoordinator CRD created during installation. See the Custom Resource Definition (CRD) Reference section for all available fields and their corresponding Helm value paths.

Gateway Configuration

Variable	Description	Default
nsight-gateway.enabled	Enable deployment of the NsightGateway CR for REST API gateway access to Coordinator and Analysis services.	true
nsight-gateway.labels	Extra labels merged into the NsightGateway CR metadata and propagated to operator-created resources.	{}
nsight-gateway.annotations	Extra annotations merged into the NsightGateway CR metadata and propagated to operator-created resources.	{}
nsight-gateway.podLabels	Extra labels merged into the gateway pod template metadata.	{}
nsight-gateway.podAnnotations	Extra annotations merged into the gateway pod template metadata.	{}

Additional configuration options can be specified through Helm values that map to the default NsightGateway CRD created during installation. See the Custom Resource Definition (CRD) Reference section for all available fields and their corresponding Helm value paths.

Cloud Storage Configuration

Variable	Description	Default
cloudStorage.enabled	Enable cloud storage integration for profiling results.	true
cloudStorage.minio.labels	Extra labels merged into the NsightCloudStorageConfig CR metadata and propagated to operator-created MinIO resources.	{}
cloudStorage.minio.annotations	Extra annotations merged into the NsightCloudStorageConfig CR metadata and propagated to operator-created MinIO resources.	{}
cloudStorage.minio.podLabels	Extra labels merged into the MinIO pod template metadata.	{}
cloudStorage.minio.podAnnotations	Extra annotations merged into the MinIO pod template metadata.	{}

Additional configuration options can be specified through Helm values that map to the default NsightCloudStorageConfig CRD created during installation. See the Custom Resource Definition (CRD) Reference section for all supported fields and their corresponding Helm value paths. Storage configuration Secrets must use storage-config.yaml as the data key.

OTLP Collection Configuration

Variable	Description	Default
nsight-otel-collector.enabled	Enable OTLP collector for trace mirroring.	true
otlpProxyConfig.enabled	Enable OTLP proxy injection for trace mirroring.	true
otlpProxyConfig.proxyImage	Envoy image injected as the OTLP proxy sidecar. Override this when mirroring images to a private registry.	nvcr.io/nvidia/devtools/envoy:v1.37.0
otlpProxyConfig.imagePullSecrets	Image pull secrets added to pods that receive the OTLP proxy sidecar. Set this when otlpProxyConfig.proxyImage points to a private registry.	[]
otlpProxyConfig.resources	Default resource requests and limits for the injected OTLP proxy sidecar. Set this value to null to inject the sidecar without resource requests or limits. Do not use {} as the Helm override: Helm merges empty maps with the chart defaults instead of clearing them.	100m CPU / 50Mi memory requests; 500m CPU / 200Mi memory limits
nsight-otel-collector.labels	Extra labels merged into the NsightOtelCollector CR metadata and propagated to operator-created resources.	{}
nsight-otel-collector.annotations	Extra annotations merged into the NsightOtelCollector CR metadata and propagated to operator-created resources.	{}
nsight-otel-collector.podLabels	Extra labels merged into the OTel collector pod template metadata.	{}
nsight-otel-collector.podAnnotations	Extra annotations merged into the OTel collector pod template metadata.	{}

Additional configuration options can be specified through Helm values that map to the default NsightOtelCollector and OTLPProxyConfig CRDs created during installation. See the Custom Resource Definition (CRD) Reference section for all available fields and their corresponding Helm value paths.

STUNner TURN Gateway Configuration

STUNner install-time settings, such as enabled, installGatewayApiCRDs, protocol, and gatewayAnnotations, live under nsight-tenant-operator.stunner.*. The TURN listener port is configured on nsight-gateway.service.turnPort. The STUNner relay Pod settings, such as resources, tolerations, affinity, securityContext, and containerSecurityContext, live under stunner-gateway-operator.stunnerGatewayOperator.dataplane.spec.*. See STUNner TURN Gateway for examples.

For air-gapped installs with STUNner enabled, also mirror and override the bundled STUNner images: stunner-gateway-operator.stunnerGatewayOperator.deployment.container.manager.image, stunner-gateway-operator.stunnerGatewayOperator.dataplane.spec.image, and stunner-gateway-operator.stunnerAuthService.deployment.container.authService.image. Configure matching imagePullSecrets under the corresponding deployment and dataplane.spec sections.

Analysis Configuration

Variable	Description	Default
nsight-analysis.enabled	Enable analysis service for running recipes.	true
nsight-analysis.labels	Extra labels merged into the NsightAnalysis CR metadata and propagated to operator-created resources.	{}
nsight-analysis.annotations	Extra annotations merged into the NsightAnalysis CR metadata and propagated to operator-created resources.	{}
nsight-analysis.podLabels	Extra labels merged into the analysis pod template metadata.	{}
nsight-analysis.podAnnotations	Extra annotations merged into the analysis pod template metadata.	{}

Additional configuration options can be specified through Helm values that map to the default NsightAnalysis CRD created during installation. See the Custom Resource Definition (CRD) Reference section for all available fields and their corresponding Helm value paths.

Nsight Streamer

Nsight Streamer is deployed separately using the NsightStreamer CRD rather than through Helm chart values. See Nsight Streamer for deployment instructions and the NsightStreamer CRD Reference for all available fields.

Injector Sub-chart Configuration

These values are prefixed with nsight-injector. when using the parent chart:

Variable	Description	Default
nsight-injector.nsightToolConfigs[].nsightToolArgs	The parameters for Nsight Systems used during profiling. See the Nsight Systems User Guide for available parameters. Placeholders within these parameters will be substituted with their actual values during execution.	--python-sampling=true --trace-fork-before-exec=true
nsight-injector.injectionRules[].processIncludePatterns	Regex patterns that specify which processes or commands in the container should be profiled.	[".*"]
nsight-injector.injectionRules[].processExcludePatterns	Regex patterns that specify which processes or commands in the container should NOT be profiled.	[]
nsight-injector.defaultInjectionExcludePatterns	Cluster-wide default regex patterns that are always excluded from injection, even when custom processExcludePatterns are provided. Intended to skip shells, coreutils, Nsight tools, etc. Set to [] to disable.	A preset list including shells (bash, sh, zsh, dash), common utilities, and Nsight tools. See chart values.yaml.
nsight-injector.injectionRules[].additionalVolumes	Additional volumes that will be injected into profiled Pods.
nsight-injector.injectionRules[].additionalVolumeMounts	Volume mounts that will be injected into matching containers.
nsight-injector.nsightToolConfigs[].env	Environment variables injected only into the profiled process (not added to container spec). Each item must have name and value fields.
nsight-injector.nsightToolConfigs[].containerEnv	Environment variables to inject into the target container (added to the Pod spec). Visible to all processes in the container. If a variable is present in both containerEnv and env, the value from env takes precedence for profiled process execution.
nsight-injector.nsightToolConfigs[].coordinator	Enable coordinator mode for on-demand profiling.	true
nsight-injector.nsightToolConfigs[].opentelemetryTrace	Enable OpenTelemetry trace mirroring for this profile.	false
nsight-injector.nsightToolConfigs[].opentelemetryTraceGracePeriodSec	Seconds to keep OpenTelemetry trace mirroring active after a profiling STOP so the async OTLP SDK can flush buffered spans. Only used when opentelemetryTrace is true.	9
nsight-injector.nsightToolConfigs[].name	Unique name used to reference this profile, for example from an injection rule’s nsightToolConfigRef or another profile’s baseNsightToolConfigRef.	default-nsight-tool-config
nsight-injector.injectionRules	List of injection rules rendered into the default NsightOperatorProfileConfig. Set to [] to disable the built-in default rules. This replaces the removed enableDefault flag.	Two label-based rules
nsight-injector.injectionRules[].name	Unique name for the injection rule.	–
nsight-injector.injectionRules[].nsightToolConfigRef	Name of the profile (from nsightToolConfigs) applied to Pods matched by this rule. If empty, defaultNsightToolConfigRef is used when set.	default-nsight-tool-config
nsight-injector.injectionRules[].nsightTags	Tags applied to sessions matched by this rule. The first tag selects the matching NsightCloudStorageConfig and is emitted as the legacy runtime nsightTag for process-hook compatibility. Currently only the first tag is used; additional tags are reserved for future use.	[default]
nsight-injector.injectionRules[].podObjectSelector	Match target Pods by labels using matchLabels and/or matchExpressions.
nsight-injector.injectionRules[].podNamespaceSelector	Match target Pods by their namespace labels using matchLabels and/or matchExpressions.
nsight-injector.machineConfig	Helm-only array of name/value pairs (system configurations) which should be updated before profiling on target nodes (currently, only kernel.perf_event_paranoid is supported). Namespace ProfileConfig CRs cannot override this value. See Requirements for x86_64 and Arm SBSA targets on Linux. To prevent the operator from updating node configurations, set machineConfig: null in the custom values file.	[{ name: kernel.perf_event_paranoid, value: 2 }]
nsight-injector.webhook.hostNetwork	Run the injector webhook Pod in the host network namespace. Required in environments where the Kubernetes API server cannot route directly to the pod network CIDR (for example, some EKS configurations), otherwise admission calls from kube-apiserver to the webhook time out and profiled Pods fail to be mutated. See Sidecar Injection Issues for diagnostics.	false
nsight-injector.webhook.dnsPolicy	Optional Pod DNS policy override for the injector webhook. Leave unset unless your host-networked webhook needs cluster DNS behavior such as ClusterFirstWithHostNet.	"" (unset)
nsight-injector.webhook.podLabels	Extra labels merged into the injector webhook pod template metadata.	{}
nsight-injector.webhook.podAnnotations	Extra annotations merged into the injector webhook pod template metadata.	{}
nsight-injector.imagePullSecrets	Pull secrets for the injector webhook Deployment and workload Pods mutated by the injector for injector images. Chart-managed Python utility images use global.nsightCloud.distrolessPythonImagePullSecrets instead.	[]

Note

Disabling the default injection rules. Earlier releases used an enableDefault flag under the (now deprecated) singular nsight-injector.nsightToolConfig value. That legacy flag is still accepted for backward compatibility, but new configurations should render no default injection rules by setting nsight-injector.injectionRules: [] in the custom values file.

Note

Renamed injector values. Several values were renamed in this release. The previous names are still accepted for backward compatibility but are deprecated and will be removed in a future release. Use only the new name for a given setting. If both the deprecated key and its replacement are set, the replacement key takes precedence.

Deprecated value	Replacement
nsight-injector.nsightToolConfig (singular object)	nsight-injector.nsightToolConfigs (list)
nsightToolConfig.injectionIncludePatterns	injectionRules[].processIncludePatterns
nsightToolConfig.injectionExcludePatterns	injectionRules[].processExcludePatterns
nsightToolConfig.containerIncludePatterns	injectionRules[].containerIncludePatterns
nsightToolConfig.nsightTag	injectionRules[].nsightTags
nsightToolConfig.cloudStorageConfigRef	injectionRules[].cloudStorageConfigRef
nsightToolConfig.logOutput	injectionRules[].logOutput
nsightToolConfig.volumes	injectionRules[].additionalVolumes
nsightToolConfig.volumeMounts	injectionRules[].additionalVolumeMounts
nsightToolConfig.otlpMirroringEnabled	nsightToolConfigs[].opentelemetryTrace
nsightToolConfig.otlpMirroringGracePeriodSec	nsightToolConfigs[].opentelemetryTraceGracePeriodSec
nsightToolConfig.otlpProxyPortRange	nsightToolConfigs[].opentelemetryProxyPortRange
nsightToolConfig.enableDefault: false	injectionRules: []

Readiness Waiter Configuration

The readiness waiter is an init container that the injector adds to profiled Pods. It blocks the main container from starting until the storage configuration, MinIO service, and Coordinator service are reachable. This prevents a profiling session from missing the early stages of a workload when the operator is still starting up (for example, immediately after a rolling restart of the control plane).

Variable	Description	Default
nsight-injector.readinessWaiter.enabled	Enable the readiness waiter init container.	true
nsight-injector.readinessWaiter.timeout	Maximum time to wait for dependencies (seconds). Applies per Pod start; after this elapses the waiter either fails or exits successfully depending on failOnTimeout.	300
nsight-injector.readinessWaiter.interval	Seconds between readiness checks while the waiter is active.	5
nsight-injector.readinessWaiter.failOnTimeout	When true, the Pod fails to start if dependencies are not ready within timeout. When false, the waiter exits successfully on timeout and the container starts – profiling for that container degrades gracefully and is retried on the next collection.	false

Supported Placeholders

Placeholder	Replacement
{NVDT_UID}	The random alphanumeric string (8 symbols)
{NVDT_PROCESS_NAME}	The profiled process name
{NVDT_PROCESS_ID}	The profiled process id
{NVDT_TIMESTAMP}	The UNIX timestamp (in ms)
%{ANY ENVIRONMENT VARIABLE}	The “ANY ENVIRONMENT VARIABLE” environment variable inside a container. NVDT_POD_FULLNAME and NVDT_CONTAINER_NAME environment variables are set by the NVIDIA Nsight Operator

Enabling Profiling on Target Resources

To enable automatic injection for all Pods in a namespace, add the nvidia-nsight-profile=enabled label to the namespace.

kubectl label namespaces <namespace name> nvidia-nsight-profile=enabled

To enable automatic injection for a specific workload in a namespace, add the nvidia-nsight-profile=enabled label to the workload’s Pod template. The injector evaluates newly created Pods, so labeling only the workload metadata does not cause Pods generated by that workload to match the default injection rule.

# Example for a deployment
kubectl patch deployment <deployment-name> -p '{"spec":{"template":{"metadata":{"labels":{"nvidia-nsight-profile":"enabled"}}}}}'

# Example for a statefulset
kubectl patch statefulset <statefulset-name> -p '{"spec":{"template":{"metadata":{"labels":{"nvidia-nsight-profile":"enabled"}}}}}'

At this point, any new Pod will be considered for injection based on labels and the configured injectionRules.

Existing Resources

An already started Pod cannot be injected. After you add or remove namespace labels, Pod template labels, or injection rules, recreate existing Pods so the admission webhook can evaluate the updated configuration.

Sample commands to restart a Pod:

1. Resource with more than one replica

   kubectl rollout restart <resource type>/<resource name>
   

   For example:

   kubectl rollout restart deployment/amazing_service
   

2. Resource with only one replica

   kubectl scale <resource type>/<resource name> --replicas=0
   kubectl scale <resource type>/<resource name> --replicas=1
   

   For example:

   kubectl scale deployment/amazing_service --replicas=0
   kubectl scale deployment/amazing_service --replicas=1
   

Advanced Configuration

In Kubernetes environments, managing sidecar injection and profiling configurations can be challenging, particularly in dynamic scenarios where Pods are created by custom resources or controllers. The process requires more than just filtering Pods – it requires selecting the appropriate configuration for each Pod, application, or namespace. While labels offer a basic level of control, they often lack the granularity required for precise targeting and configuration.

NVIDIA Nsight Operator supports the following mechanisms for filtering and targeting Pods for injection:

• matchExpressions: Specify complex logic using custom expressions to evaluate Pod metadata and dynamically determine injection suitability.

• podNamespaceSelector: Filter Pods based on labels applied to their namespaces.

• podObjectSelector: Filter Pods based on labels applied directly to the Pod objects.

Example Configuration

Below is a sample custom_values_fine_grained.yaml configuration demonstrating the use of these mechanisms for fine-grained injection control.

# Disable the default configuration
nsight-injector:
  injectionRules: []

  injectionConfig:
    nsightToolConfigs:
      - name: "triton-profile"
        nsightToolArgs: "--duration 20 --kill none -o /home/auto_{NVDT_PROCESS_NAME}_%{NVDT_POD_FULLNAME}_%{NVDT_CONTAINER_NAME}_{NVDT_TIMESTAMP}_{NVDT_UID}.nsys-rep"
      - name: "other-profile"
        nsightToolArgs: "--duration 30 --kill none -o /home/auto_{NVDT_PROCESS_NAME}_%{NVDT_POD_FULLNAME}_%{NVDT_CONTAINER_NAME}_{NVDT_TIMESTAMP}_{NVDT_UID}.nsys-rep"
        env:
          - name: NSYS_NVTX_PROFILER_REGISTER_ONLY
            value: "0"
    injectionRules:
      - name: "has-injection-label-or-demo-injection-name"
        nsightToolConfigRef: "triton-profile"
        processIncludePatterns:
          - "^/opt/tritonserver/bin/tritonserver.*$"
        podMatchConditions:
        - name: "has-injection-label-or-demo-injection-name"
          expression: >
            ((has(object.metadata.labels) &&
            'nvidia-nsight-profile' in object.metadata.labels &&
            object.metadata.labels['nvidia-nsight-profile'] == 'enabled') ||
            object.metadata.name.contains('demo-injection'))
      - name: "train-injection"
        nsightToolConfigRef: "other-profile"
        processIncludePatterns:
          - "^python MaxText/train.py.*$"
        podMatchConditions:
          - name: "fine-grained"
            expression: |
                        (
                          object.metadata.generateName.startsWith("example-deployment-name-") &&
                          object.metadata.namespace == "example-ns"
                        ) ||
                        (
                          object.metadata.ownerReferences.exists(ref, ref.kind == "DaemonSet" &&
                          ref.name == "example-daemonset")
                        )
      - name: "namespace-selector-filter"
        nsightToolConfigRef: "other-profile"
        processIncludePatterns:
          - "^python MaxText/train.py.*$"
        podNamespaceSelector:
          matchLabels:
            custom-injection-label: enabled
      - name: "object-selector-filter"
        nsightToolConfigRef: "other-profile"
        processIncludePatterns:
          - "^python MaxText/train.py.*$"
        podObjectSelector:
          matchLabels:
            custom-injection-label: enabled
      - name: "combined-filter"
        nsightToolConfigRef: "other-profile"
        processIncludePatterns:
          - "^python MaxText/train.py.*$"
        podNamespaceSelector:
          matchLabels:
            combined-custom-injection-label: enabled
        podObjectSelector:
          matchLabels:
            combined-custom-injection-label: enabled
        podMatchConditions:
          - name: "combined"
            expression: 'object.metadata.name.startsWith("example-pod-prefix-")'

The above configuration customizes profiling parameters for different applications and Pods based on their metadata.

1. It enables profiling for 20 seconds for all the /opt/tritonserver/bin/tritonserver processes in all the Pods with the nvidia-nsight-profile=enabled label or Pods with the demo-injection in their name.

2. It enables profiling for 30 seconds for all the python MaxText/train.py processes in:

   • all the Pods with generated name starting with example-deployment-name- in the example-ns namespace or Pods owned by the example-daemonset

   • all the Pods in the namespace with the custom-injection-label=enabled label

   • all the Pods with the custom-injection-label=enabled label

   • all the Pods with the namespace label combined-custom-injection-label=enabled and the Pod label combined-custom-injection-label=enabled and the Pod name starting with example-pod-prefix-

Multi-Tenant Configuration

NVIDIA Nsight Operator supports multi-tenant environments where different teams or users require separate configurations. The operator can be configured to apply different profiles and injection rules based on the namespace or Pod name. Below is a sample custom_values_multi_tenant.yaml configuration demonstrating the use of profiles and injection rules for multi-tenant environments. It activates possibility of profiling (profiling is still not enabled after installing) in all namespaces with the nvidia-nsight-profile=enabled label:

# Disable the default configuration
nsight-injector:
  injectionRules: []

  clusterWideInjectionFilter:
    matchConditions:
      - name: "is-pod"
        expression: "object.kind == 'Pod'"
      - name: "not-self-managed"
        expression: "!(has(object.metadata.labels) && 'app' in object.metadata.labels && object.metadata.labels['app'] in ['nvidia-nsight-operator', 'nsight-operator'])"
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: "NotIn"
          values:
            - kube-system
            - kube-node-lease
            - kube-public
        - key: nvidia-nsight-profile
          operator: "In"
          values:
            - enabled

To enable profiling in a specific namespace, the user of this namespace should add the NsightOperatorProfileConfig resource with the profiling configuration content. The spec can include nsightToolConfigs and injectionRules values supported by the installation configuration. Use nsightToolConfigRef on each rule to choose the profile that will be applied when the rule matches.

Sample custom_installation_injection_config.yaml configuration (can be deployed by the kubectl apply -n example-ns -f custom_installation_injection_config.yaml command):

apiVersion: nvidia.com/v1
kind: NsightOperatorProfileConfig
metadata:
  name: custom-profile-config
spec:
  nsightToolConfigs:
    - name: "update-profile"
      nsightToolArgs: "--duration 2 --kill none -o /home/separate_auto_{NVDT_PROCESS_NAME}_%{NVDT_POD_FULLNAME}_%{NVDT_CONTAINER_NAME}_{NVDT_TIMESTAMP}_{NVDT_UID}.nsys-rep"
  injectionRules:
    - name: "has-injection-label-or-demo-injection-name"
      enabled: false
    - name: "starts-with-name"
      nsightToolConfigRef: "update-profile"
      logOutput: /mnt/nv/out.log
      processIncludePatterns:
        - "^/cuda-samples/vectorAdd_forever.*$"
      podMatchConditions:
      - name: "starts-with-name"
        expression: >
          (has(object.metadata.generateName) &&
          object.metadata.generateName.contains('cuda-vector-add-forever'))

The configuration above enables profiling for 2 seconds for all the /cuda-samples/vectorAdd_forever processes in all the Pods with the generated name containing cuda-vector-add-forever. It also disables the “has-injection-label-or-demo-injection-name” injection configuration (if it was specified in the default cluster-wide configuration or any other NsightOperatorProfileConfig in the Pod’s namespace).

There can be multiple NsightOperatorProfileConfig resources in a Pod’s namespace. NVIDIA Nsight Operator will apply all the configuration from the NsightOperatorProfileConfig. Injection configurations cannot have the same name in the same namespace (the only situation when the same name is allowed is when the configuration is disabled).

Advanced Configuration Values

Variable	Type	Description
nsightToolConfigs	list	Defines one or more “profiles”. Each profile describes how the Nsight tool injection should be performed.
nsightToolConfigs[].name	string	A unique name identifying the profile.
nsightToolConfigs[].nsightToolArgs	string	Parameters for Nsight Systems. See the Nsight Systems User Guide. Placeholders will be substituted with actual values during execution.
nsightToolConfigs[].env	list	Environment variables injected only into the profiled process.
nsightToolConfigs[].containerEnv	list	Environment variables to inject into the target container (added to Pod spec). Visible to all processes in the container.
nsightToolConfigs[].coordinator	boolean	When true, enables coordinator mode for this profile.
nsightToolConfigs[].opentelemetryTrace	boolean	Enable or disable OpenTelemetry trace mirroring for this profile.
injectionRules	list	A list of rules that determines which Pods should receive injection.
injectionRules[].name	string	A unique name identifying this set of injection rules.
injectionRules[].nsightToolConfigRef	string	The name of a specific profile to use if this rule matches.
injectionRules[].cloudStorageConfigRef	string	Reference to a NsightCloudStorageConfig resource in the same namespace for sessions matched by this rule.
injectionRules[].logOutput	string	Logging output destination for sessions matched by this rule. Can be stdout, stderr or a file path. By default, logging is disabled.
injectionRules[].processIncludePatterns	list of strings	Regex patterns specifying which processes should be profiled.
injectionRules[].processExcludePatterns	list of strings	Regex patterns specifying which processes should NOT be profiled.
injectionRules[].additionalVolumes	list	Additional volumes that will be injected into profiled Pods.
injectionRules[].additionalVolumeMounts	list	Volume mounts that will be injected into matching containers.
injectionRules[].podMatchConditions	list	A list of conditions to evaluate using CEL expressions.
injectionRules[].podMatchConditions[].name	string	A name for the match condition.
injectionRules[].podMatchConditions[].expression	string (CEL)	A CEL expression that returns true if the Pod should be injected.
injectionRules[].podNamespaceSelector	object	Label selector to match namespace labels.
injectionRules[].podNamespaceSelector.matchLabels	map	Key-value pairs that must be present on the namespace.
injectionRules[].podObjectSelector	object	Label selector to match pod labels.
injectionRules[].podObjectSelector.matchLabels	map	Key-value pairs that must be present on the pod.