Skip to main content
Ctrl+K
NVIDIA NeMo Microservices - Home NVIDIA NeMo Microservices - Home

NVIDIA NeMo Microservices

NVIDIA NeMo Microservices - Home NVIDIA NeMo Microservices - Home

NVIDIA NeMo Microservices

Table of Contents

About NeMo Microservices

  • Overview
  • Key Features
  • Concepts
    • Platform
    • Entities
    • Customization
    • Evaluation
    • Inference
    • Guardrails
    • Synthetic Data Generation
  • Release Notes

Get Started

  • About Getting Started
  • Demo Cluster Setup
    • Requirements
    • Minikube
      • Set Up Using Deployment Scripts
      • Set Up Manually
    • Kubernetes
      • Install
      • Ingress Setup
      • Upgrade
      • Uninstall
  • Install NeMo Microservices Python SDK
  • Beginner Platform Tutorials
    • Deploy NIM
    • Customize and Evaluate LLMs
    • Add Safety Checks to LLMs
    • Use Llama Stack APIs
  • Jupyter Notebooks

Manage Entities

  • About Managing Entities
  • Tutorials
    • Set Up Organizational Entities
    • Create Dataset Files
  • Namespaces
    • Create Namespace
    • Update Namespace
    • Get Namespace
    • List Namespaces
    • Delete Namespace
  • Projects
    • Create Project
    • Update Project
    • Get Project
    • List Projects
    • Delete Project
  • Datasets
    • Create Dataset
    • Get Dataset
    • Update Dataset
    • List Datasets
    • Delete Dataset
  • Models
    • Get Details of a Model
    • Update Model
    • List Models
    • Delete Model
  • Entity Fields Reference

Generate Synthetic Data (Beta)

  • About Generating Synthetic Data
  • Quickstart
  • Tutorials
  • Configure Models
  • Seeding SDG with External Data
  • Generate Realistic Personal Details
  • Define Data Columns
    • Column Types
      • Sampling-Based Columns
      • Expression Columns
      • LLM-Based Columns
    • Multi-Modal Context
    • Add Constraints
    • Using Jinja Templates
    • Structured Outputs
  • Generate Data
    • Data Generation Workflow
    • Manage Jobs
      • Create Data Generation Job
      • Get Job Status
      • List Data Generation Jobs
      • Get Job Logs
      • Get Job Results
      • Download Job Results
  • Data Designer Configuration
    • Save Configurations
    • Load Configurations
  • Data Quality
    • Code Validation
    • Data Evaluation
    • Add Custom Validation

Fine-Tune

  • About Fine-Tuning
  • Tutorials
    • Format Training Dataset
    • LoRA Customization
    • SFT Customization
    • DPO Customization
    • Distillation Customization
    • Check Job Metrics
    • Optimize Throughput
    • Import Private HF Model
  • Model Catalog
    • Llama Models
    • Llama Nemotron Models
    • Phi Models
    • Embedding Models
    • GPT-OSS Models
    • Gemma Models
    • Dataset Format Requirements
  • Manage Targets
    • Create Target
    • Get Target Details
    • List Targets
    • Update Target
    • Delete Target
    • Target Values
  • Manage Configs
    • Create Config
    • Get Config Details
    • List Configs
    • Update Config
  • Manage Jobs
    • Create Job
    • Get Job Status
    • List Active Jobs
    • Cancel Job
    • Hyperparameter Options

Evaluate

  • About Evaluating
  • Quickstart
  • Tutorials
    • Run an LLM Harness Eval
    • Run an LLM Judge Eval
  • Evaluation Flows
    • Academic Benchmarks
      • BigCode
      • BFCL
      • LM Harness
      • Safety Harness
      • Simple Evals
    • Retrieval
    • RAG
    • Agentic
    • LLM-as-a-Judge
    • Template
    • Prompt Optimization
  • Targets
    • Create Target
    • Delete Target
    • Data Source Targets
    • LLM Model Targets
    • Retriever Pipeline Targets
    • RAG Pipeline Targets
    • Target Schema
  • Configurations
    • Create Config
    • Delete Config
    • Config Schema
  • Jobs
    • Create Job
    • Get Job Details
    • Get Job Status
    • List Jobs
    • Get Job Results
    • Download Detailed Results
    • Get Job Logs
    • Delete Job
    • Job Target & Config Matrix
    • Job Durations
    • Job Schema
  • Live Evaluations
  • API Key Authentication
  • Results
  • Filter and Sort Responses
  • Support Matrix

Audit Model Safety (Beta)

  • About Auditing Models
  • Quickstart
  • Tutorials
    • Run a Simple Job
    • Run with Local NIM
    • Notebook
  • Manage Targets
    • Basic Target
    • NVIDIA Hosted NIM
    • NIM Proxy
    • Local NIM
    • OpenAI
    • List Targets
    • Update and Delete a Target
  • Manage Configs
    • Create Config
    • Get a Config
    • List Configs
    • Update and Delete a Config
    • Probes and Plugins
  • Run and Manage Audit Jobs
  • Viewing Audit Job Results
  • Reference

Deploy NIM and Run Inference

  • About Deploying and Running Inference on NIM
  • Tutorials
    • Deploy NIM
    • Run Inference on NIM
  • Manage NIM Deployments
    • Deploy NIM Microservices
    • Get NIM Deployment Details
    • List Deployments
    • Update Deployment
    • Delete NIM Deployment
    • Create Configuration
    • Get Configuration
    • List Configurations
    • Update Configuration
    • Delete Configuration
  • Run Inference on NIM
    • Health Check
    • List Models
    • Chat Completions
    • Completions
    • Embeddings

Manage Guardrails

  • About Guardrails
  • Terminology
  • Quickstart
  • Tutorials
    • Integrate with NemoGuard NIM
    • Parallel Rails
    • Multimodal Data
    • Injection Detection
    • Custom HTTP Headers
    • Custom LLM Providers
  • Manage Configurations
    • Creating a Configuration
    • Listing Configurations
    • Getting a Configuration
    • Updating a Configuration
    • Deleting a Configuration
    • Configuration Store
  • Manage Access to Models
  • Check a Guardrail
  • Inference with Guardrails
  • Streaming Output
  • Reference

Admin Setup

  • About Admin Setup
  • Helm Installation Overview
  • Helm Installation Options
  • Install as Platform
    • Install
    • Ingress Setup
    • Upgrade
    • Uninstall
  • Install Individually
    • NeMo Auditor
      • Helm Chart
    • NeMo Customizer
    • NeMo Data Store
    • NeMo Deployment Management
    • NeMo Entity Store
    • NeMo Evaluator
      • Helm Chart
      • Chart Config Options
    • NeMo Guardrails
      • Basic Installation
      • Integrate with NIM Deployed in Cluster
      • Integrate with NIM from build.nvidia.com
      • High Availability
      • Configuration Store
      • Custom Dependencies
    • NeMo NIM Proxy
    • NeMo Operator
  • Configure Models
  • Manage GPUs
    • Configure Cluster GPUs
    • Model Configurations Matrix
    • Troubleshooting GPU Jobs
  • Custom Resource Definitions
  • Manage Storage
    • Databases
      • PostgreSQL
      • Milvus
    • PVCs
      • AWS Peristent Volumes
      • Oracle Persistent Volumes
    • Object Storage
      • Amazon S3
    • Backup and Restore
  • Manage Secrets
    • Secrets for Accessing NGC Catalog
    • External Database Secrets
    • JSON Web Token Secrets
    • Object Store Secrets
    • MLFlow Customizer Secrets
    • Weights & Biases Keys
    • Hugging Face API Key Secret
  • Open Telemetry Setup
  • Tenant Configuration Options
  • Security for NeMo Microservices

Reference

  • System Requirements
  • NeMo Microservice API Reference
    • Platform
    • Auditor
    • Customizer
    • Data Designer
    • Deployment Management
    • Entity Store
    • Evaluator
    • Guardrails
    • NIM Proxy
  • NeMo Microservices Python SDK Reference
    • Client APIs
    • Resource APIs
      • Audit Safety
        • Audit Configuration Resource
        • Audit Targets Resource
        • Auditor Jobs Resource
        • Auditor Results Resource
      • Customization
        • Customization Configs Resource
        • Customization Resource
        • Customization Jobs Resource
        • Customization Targets Resource
      • Data Designer
        • Data Designer Resource
        • Data Designer Jobs Resource
        • Data Designer Results Resource
      • Deployment
        • Deployment Resource
        • Deployment Configs Resource
        • Deployment Model Deployments Resource
      • Entity
        • Entity Datasets Resource
        • Entity Models Resource
        • Entity Namespaces Resource
        • Entity Projects Resource
      • Evaluation
        • Evaluation Resource
        • Evaluation Configs Resource
        • Evaluation Jobs Resource
        • Evaluation Results Resource
        • Evaluation Targets Resource
      • Guardrails
        • Guardrail Resource
        • Guardrail Configs Resource
        • Guardrail Models Resource
        • Guardrail Completions Resource
      • Inference
        • Inference Resource
        • Inference Models Resource
        • Inference Chat Resource
        • Inference Chat Completions Resource
        • Inference Completions Resource
        • Inference Embeddings Resource
    • Type APIs
      • Beta Types
        • Audit Types
        • Data Designer Types
        • AuditPlugin
        • AuditTarget
        • AuditRunData
        • Audit Report Data
        • Audit System Data
        • Audit Config Param
        • Audit Plugins Page
        • Audit Target Param
        • Audit Plugin Filter
      • Chat Types
        • ChatCompletionResponse
        • ChatCompletionStreamResponse
        • CompletionCreateParams
      • Customization Types
        • ConfigCreateParams
        • ConfigUpdateParams
        • CustomizationJob
        • DatasetCu
        • DatasetParameters
        • Hyperparameters
        • JobEntry
        • JobCreateParams
        • LoraParameters
        • SftParameters
        • WandBIntegration
      • Deployment Types
        • ConfigCreateParams for Deployment
        • ConfigListParams
        • ConfigUpdateParams for Deployment
        • DeploymentConfigFilterParam
        • DeploymentConfigsPage
        • ModelDeployment
        • ModelDeploymentCreateParams
        • ModelDeploymentFilterParam
        • ModelDeploymentListParams
        • ModelDeploymentStatusDetails
        • ModelDeploymentUpdateParams
        • ModelDeploymentsPage
      • Evaluation Types
        • ConfigCreateParams for Evaluation
        • ConfigListParams for Evaluation
        • ConfigUpdateParams for Evaluation
        • EvaluationConfigsPage
        • EvaluationJob
        • EvaluationJobFilter
        • EvaluationJobFilterParam
        • EvaluationJobsPage
        • EvaluationResultFilter
        • EvaluationResultFilterParam
        • EvaluationResultsPage
        • EvaluationTargetsPage
        • GroupResultParam
        • JobCreateParams for Evaluation
        • JobListParams
        • MetricResultParam
        • ResultCreateParams
        • ResultListParams
        • ResultUpdateParams
        • TargetCreateParams
        • TargetListParams
        • TargetUpdateParams
        • TaskResultParam
      • Guardrail Types
        • CompletionCreateParams
        • ConfigCreateParams
        • ConfigListParams
        • ConfigUpdateParams
        • GuardrailCompletionResponse
        • GuardrailCompletionStreamResponse
        • GuardrailConfigFilterParam
        • GuardrailConfigsPage
        • ModelListParams
        • ModelsResponse
        • ModelsResponseEntry
      • Shared Types
        • ActionRails
        • APIEndpointData
        • APIEndpointFormat
        • ArtifactStatus
        • AutoAlignOptions
        • AutoAlignRailConfig
        • BackendEngineType
        • ChatCompletionAssistantMessageParam
        • ChatCompletionContentPartImageParam
        • ChatCompletionContentPartTextParam
        • ChatCompletionFunctionMessageParam
        • ChatCompletionMessageToolCallParam
        • ChatCompletionMessageToolCall
        • ChatCompletionMessage
        • ChatCompletionResponseChoice
        • ChatCompletionResponseStreamChoice
        • ChatCompletionSystemMessageParam
        • ChatCompletionTokenLogprob
        • ChatCompletionToolMessageParam
        • ChatCompletionUserMessageParam
        • ChoiceDeltaFunctionCall
        • ChoiceDeltaToolCallFunction
        • ChoiceDeltaToolCall
        • ChoiceLogprobs
        • ClavataRailConfig
        • ClavataRailOptions
        • CompletionResponseChoice
        • CompletionResponseStreamChoice
        • ConfigDataParam
        • ConfigData
        • DeleteResponse
        • DeltaMessage
        • DialogRails
        • ErrorResponse
        • FactCheckingRailConfig
        • FiddlerGuardrails
        • FinetuningType
        • FunctionCall
        • Function
        • GenericSortField
        • GuardrailConfigParam
        • GuardrailConfig
        • GuardrailModel
        • HTTPValidationError
        • ImageURL
        • InferenceParams
        • InjectionDetection
        • InputRails
        • Instruction
        • JailbreakDetectionConfig
        • JobStatus
        • LogProbs
        • LoraFinetuningData
        • MessageTemplate
        • ModelArtifact
        • ModelPrecision
        • ModelSpec
        • OutputRailsStreamingConfig
        • OutputRails
        • Ownership
        • PTuningFinetuningData
        • PaginationData
        • ParameterEfficientFinetuningData
        • PatronusEvaluateAPIParams
        • PatronusEvaluateConfigParam
        • PatronusEvaluateConfig
        • PatronusEvaluationSuccessStrategy
        • PatronusRailConfigParam
        • PatronusRailConfig
        • PrivateAIDetectionOptions
        • PrivateAIDetection
        • PromptData
        • RailsConfigDataParam
        • RailsConfigData
        • RailsParam
        • Rails
        • ReasoningModelConfig
        • ReasoningParams
        • RetrievalRails
        • SensitiveDataDetectionOptions
        • SensitiveDataDetection
        • SingleCallConfig
        • TaskPrompt
        • TopLogprob
        • UsageInfo
        • UserMessagesConfig
        • ValidationError
        • VersionTag
      • Shared Params Types
        • ActionRails
        • APIEndpointData
        • APIEndpointFormat
        • ArtifactStatus
        • AutoAlignOptions
        • AutoAlignRailConfig
        • BackendEngineType
        • ChatCompletionAssistantMessageParam
        • ChatCompletionContentPartImageParam
        • ChatCompletionContentPartTextParam
        • ChatCompletionFunctionMessageParam
        • ChatCompletionMessageToolCallParam
        • ChatCompletionSystemMessageParam
        • ChatCompletionToolMessageParam
        • ChatCompletionUserMessageParam
        • ClavataRailConfig
        • ClavataRailOptions
        • ConfigDataParam
        • DialogRails
        • FactCheckingRailConfig
        • FiddlerGuardrails
        • FinetuningType
        • Function
        • FunctionCall
        • GenericSortField
        • GuardrailConfigParam
        • GuardrailModel
        • ImageURL
        • InferenceParams
        • InjectionDetection
        • InputRails
        • Instruction
        • JailbreakDetectionConfig
        • JobStatus
        • LoraFinetuningData
        • MessageTemplate
        • ModelArtifact
        • ModelPrecision
        • ModelSpec
        • OutputRails
        • OutputRailsStreamingConfig
        • Ownership
        • ParameterEfficientFinetuningData
        • PatronusEvaluateAPIParams
        • PatronusEvaluateConfigParam
        • PatronusEvaluationSuccessStrategy
        • PatronusRailConfigParam
        • PTuningFinetuningData
        • PrivateAIDetection
        • PrivateAIDetectionOptions
        • PromptData
        • RailsConfigDataParam
        • RailsParam
        • ReasoningModelConfig
        • ReasoningParams
        • RetrievalRails
        • SensitiveDataDetection
        • SensitiveDataDetectionOptions
        • SingleCallConfig
        • TaskPrompt
        • UserMessagesConfig
        • VersionTag
      • Common Types
        • ActivatedRail
        • ArtifactStatusDe
        • BackendEngineTypeDe
        • BaseModelFilterParam
        • BaseModelFilter
        • CachedOutputsDataParam
        • CachedOutputsData
        • CompletionCreateParams
        • CompletionResponse
        • CompletionStreamResponse
        • CreateEmbeddingResponse
        • CreatedAtFilterParam
        • CreatedAtFilter
        • CustomizationConfigParam
        • CustomizationConfig
        • CustomizationTargetParam
        • CustomizationTarget
        • CustomizationTrainingOptionParam
        • CustomizationTrainingOption
        • DatasetCreateParams
        • DatasetEvParam
        • DatasetEv
        • DatasetFilterParam
        • DatasetFilter
        • DatasetListParams
        • DatasetSortField
        • DatasetUpdateParams
        • Dataset
        • DatasetsPage
        • DateTimeFilterParam
        • DateTimeFilter
        • DeploymentConfigParam
        • DeploymentConfig
        • EmbeddingCreateParams
        • Embedding
        • EvaluationConfigFilterParam
        • EvaluationConfigFilter
        • EvaluationConfigParam
        • EvaluationConfig
        • EvaluationLiveParams
        • EvaluationParamsParam
        • EvaluationParams
        • EvaluationResult
        • EvaluationStatusDetailsParam
        • EvaluationStatusDetails
        • EvaluationTargetFilterParam
        • EvaluationTargetFilter
        • EvaluationTargetParam
        • EvaluationTarget
        • ExecutedAction
        • ExternalEndpointConfigParam
        • ExternalEndpointConfig
        • FinetuningTypeDe
        • GenerationLogOptionsParam
        • GenerationLog
        • GenerationOptionsParam
        • GenerationRailsOptionsParam
        • GenerationStats
        • GroupConfigParam
        • GroupConfig
        • GroupResult
        • GuardrailCheckParams
        • GuardrailCheckResponse
        • GuardrailConfigDeParam
        • GuardrailConfigDe
        • GuardrailsDataParam
        • GuardrailsData
        • LabelSelectorRequirementParam
        • LabelSelectorRequirement
        • LabelSelectorTermParam
        • LabelSelectorTerm
        • LiveEvaluation
        • LlmCallInfo
        • MetricConfigParam
        • MetricConfig
        • MetricResult
        • ModelArtifactDeParam
        • ModelArtifactDe
        • ModelCreateParams
        • ModelDeParam
        • ModelDe
        • ModelEvParam
        • ModelEv
        • ModelFilterParam
        • ModelFilter
        • ModelListParams
        • ModelParam
        • ModelPeftFilterParam
        • ModelPeftFilter
        • ModelPrecisionDe
        • ModelSortField
        • ModelSpecDeParam
        • ModelSpecDe
        • ModelUpdateParams
        • Model
        • ModelsPage
        • NamespaceCreateParams
        • NamespaceListParams
        • NamespaceUpdateParams
        • Namespace
        • NamespacesPage
        • NIMDeploymentConfigParam
        • NIMDeploymentConfig
        • NodeAffinityParam
        • NodeAffinity
        • NodeSelectorParam
        • NodeSelectorTermParam
        • NodeSelectorTerm
        • NodeSelector
        • ParameterEfficientFinetuningDataDeParam
        • ParameterEfficientFinetuningDataDe
        • PreferredSchedulingTermParam
        • PreferredSchedulingTerm
        • ProjectCreateParams
        • ProjectFilterParam
        • ProjectFilter
        • ProjectListParams
        • ProjectSortField
        • ProjectUpdateParams
        • Project
        • ProjectsPage
        • PromptDataDeParam
        • PromptDataDe
        • RagPipelineDataParam
        • RagPipelineData
        • RagTargetParam
        • RagTarget
        • RailStatus
        • RetrieverPipelineDataParam
        • RetrieverPipelineData
        • RetrieverTargetParam
        • RetrieverTarget
        • ScoreParam
        • ScoreStatsParam
        • ScoreStats
        • Score
        • StatusEnum
        • TargetStatus
        • TargetType
        • TaskConfigParam
        • TaskConfig
        • TaskResult
        • TaskStatus
        • TolerationParam
        • Toleration
        • TrainingPodSpecParam
        • TrainingPodSpec
        • TrainingType
  • NeMo Microservices Helm Chart
  • Troubleshooting
    • Troubleshoot Auditor
    • Troubleshoot Customizer
    • Troubleshoot Evaluator
    • Troubleshoot Guardrails
    • Troubleshoot Data Designer
    • Troubleshoot Setup
  • NVIDIA Distribution in Llama Stack
  • EULA

Resources

  • OSS License Acknowledgements
  • NeMo Microservices Python SDK Reference
  • Resource APIs
  • NeMo Auditor Resources

NeMo Auditor Resources#

The resources are for managing audit jobs, configurations, targets, and results through the NeMo Auditor microservice.

Note

Auditor resources are available in the main NeMo Microservices SDK under the beta namespace. To use these resources, install the main SDK with the audit extra:

pip install 'nemo-microservices[audit]'
  • Audit Configuration Resource
  • Audit Targets Resource
  • Auditor Jobs Resource
  • Auditor Results Resource

previous

Resource APIs

next

Audit Configuration Resource

NVIDIA NVIDIA
Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2024-2025, NVIDIA Corporation.

Last updated on Sep 10, 2025.