For AI agents: a documentation index is available at the root level at /llms.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.

About NVIDIA NeMo Relay
Getting Started
NVIDIA NeMo Relay CLI
Supported Integrations
Instrument Applications
Observability Plugin
Adaptive Plugin
NeMo Guardrails Plugin
- About
- Configuration
Integrate into Frameworks
Build Plugins
Contribute
Reference
- APIs
- Performance
Resources

Overview
Architecture
Ecosystem
Concepts
Scopes
Middleware
Plugins
Events
Subscribers
Framework Integrations
Release Notes
Highlights
Known Issues
Agent Runtime Primer
Prerequisites
Installation
Configuration / Setup
Quick Start
Python Quick Start
Node.js Quick Start
Rust Quick Start
About
Basic Usage
Claude Code
Codex
Cursor
Hermes Agent
About
OpenClaw Plugin Guide
LangChain Integration Guide
LangGraph Integration Guide
Deep Agents Integration Guide
About
Adding Scopes and Marks
Instrument a Tool Call
Instrument an LLM Call
Add Middleware
Code Examples
About
Configuration
Agent Trajectory Interchange Format (ATIF)
Agent Trajectory Observability Format (ATOF)
OpenTelemetry
OpenInference
About
Configuration
Adaptive Cache Governor (ACG)
Adaptive Hints
About
Configuration
About
Adding Scopes
Wrap Tool Calls
Wrap LLM Calls
Handle Non-Serializable Data
Using Codecs
Provider Codecs
Provider Response Codecs
Code Examples
About
Define a Plugin
Validate Plugin Configuration
Plugin Configuration Files
Register Plugin Behavior
Design Plugin Configuration
NeMo Guardrails Example Plugin
Code Examples
About
Development Setup
Workflow and Reviews
Testing and Documentation
APIs
Python Library Reference
nemo_relay
adaptive
codecs
guardrails
integrations
deepagents
callbacks
middleware
langchain
callbacks
middleware
langgraph
callbacks
intercepts
llm
observability
plugin
scope
scope_local
subscribers
tools
typed
utils
Node.js Library Reference
Runtime
Typed Helpers
Plugin Helpers
Adaptive Helpers
Observability Helpers
Rust Library Reference
nemo-relay
api
event
BaseEvent
CategoryProfile
DataSchema
EventCategory
MarkEvent
ScopeEvent
Event
ScopeCategory
ATOF_VERSION
attributes_from_handle
llm_attributes_to_strings
scope_attributes_to_strings
tool_attributes_to_strings
llm
CreateLlmHandleParams
EndLlmHandleParams
LlmAttributes
LlmCallEndParams
LlmCallExecuteParams
LlmCallParams
LlmHandle
LlmRequest
LlmStreamCallExecuteParams
llm_call
llm_call_end
llm_call_execute
llm_conditional_execution
llm_request_intercepts
llm_stream_call_execute
registry
deregister_llm_conditional_execution_guardrail
deregister_llm_execution_intercept
deregister_llm_request_intercept
deregister_llm_sanitize_request_guardrail
deregister_llm_sanitize_response_guardrail
deregister_llm_stream_execution_intercept
deregister_tool_conditional_execution_guardrail
deregister_tool_execution_intercept
deregister_tool_request_intercept
deregister_tool_sanitize_request_guardrail
deregister_tool_sanitize_response_guardrail
register_llm_conditional_execution_guardrail
register_llm_execution_intercept
register_llm_request_intercept
register_llm_sanitize_request_guardrail
register_llm_sanitize_response_guardrail
register_llm_stream_execution_intercept
register_tool_conditional_execution_guardrail
register_tool_execution_intercept
register_tool_request_intercept
register_tool_sanitize_request_guardrail
register_tool_sanitize_response_guardrail
scope_deregister_llm_conditional_execution_guardrail
scope_deregister_llm_execution_intercept
scope_deregister_llm_request_intercept
scope_deregister_llm_sanitize_request_guardrail
scope_deregister_llm_sanitize_response_guardrail
scope_deregister_llm_stream_execution_intercept
scope_deregister_tool_conditional_execution_guardrail
scope_deregister_tool_execution_intercept
scope_deregister_tool_request_intercept
scope_deregister_tool_sanitize_request_guardrail
scope_deregister_tool_sanitize_response_guardrail
scope_register_llm_conditional_execution_guardrail
scope_register_llm_execution_intercept
scope_register_llm_request_intercept
scope_register_llm_sanitize_request_guardrail
scope_register_llm_sanitize_response_guardrail
scope_register_llm_stream_execution_intercept
scope_register_tool_conditional_execution_guardrail
scope_register_tool_execution_intercept
scope_register_tool_request_intercept
scope_register_tool_sanitize_request_guardrail
scope_register_tool_sanitize_response_guardrail
runtime
callbacks
EventSubscriberFn
LlmCollectorFn
LlmConditionalFn
LlmExecutionFn
LlmExecutionNextFn
LlmFinalizerFn
LlmJsonStream
LlmRequestInterceptFn
LlmSanitizeRequestFn
LlmSanitizeResponseFn
LlmStreamExecutionFn
LlmStreamExecutionNextFn
ToolConditionalFn
ToolExecutionFn
ToolExecutionNextFn
ToolInterceptFn
ToolSanitizeFn
global
global_context
scope_stack
ScopeStack
ThreadScopeStackBinding
TASK_SCOPE_STACK
capture_thread_scope_stack
create_scope_stack
current_scope_stack
propagate_scope_to_thread
restore_thread_scope_stack
scope_stack_active
set_thread_scope_stack
sync_thread_scope_stack
task_scope_push
task_scope_remove
task_scope_top
ScopeStackHandle
state
NemoRelayContextState
subscriber_dispatcher
flush_subscribers
scope
CreateScopeHandleParams
EmitMarkEventParams
EndScopeHandleParams
PopScopeParams
PushScopeParams
ScopeAttributes
ScopeHandle
HandleAttributes
ScopeType
event
get_handle
pop_scope
push_scope
subscriber
deregister_subscriber
flush_subscribers
register_subscriber
scope_deregister_subscriber
scope_register_subscriber
tool
CreateToolHandleParams
EndToolHandleParams
ToolAttributes
ToolCallEndParams
ToolCallExecuteParams
ToolCallParams
ToolHandle
tool_call
tool_call_end
tool_call_execute
tool_conditional_execution
tool_request_intercepts
codec
anthropic
AnthropicMessagesCodec
AnthropicMessagesStreamingCodec
openai_chat
OpenAIChatCodec
OpenAIChatStreamingCodec
openai_responses
OpenAIResponsesCodec
OpenAIResponsesStreamingCodec
request
AnnotatedLlmRequest
FunctionCall
FunctionDefinition
GenerationParams
OpenAiImageUrl
ToolCall
ToolChoiceFunction
ToolChoiceFunctionName
ToolDefinition
ContentPart
Message
MessageContent
ToolChoice
response
AnnotatedLlmResponse
ResponseToolCall
Usage
ApiSpecificResponse
FinishReason
streaming
SseEvent
SseEventDecoder
StreamingCodec
traits
LlmCodec
LlmResponseCodec
config_editor
EditorFieldSpec
EditorSchema
EditorFieldKind
EditorConfig
error
FlowError
Result
json
merge_json
Json
observability
atif
AtifAgentInfo
AtifAncestry
AtifExporter
AtifFinalMetrics
AtifInvocationInfo
AtifMetrics
AtifObservation
AtifObservationResult
AtifStep
AtifStepExtra
AtifSubagentTrajectoryRef
AtifToolCall
AtifTrajectory
ATIF_SCHEMA_VERSION
atof
AtofExporter
AtofExporterConfig
AtofExporterError
AtofExporterMode
Result
openinference
OpenInferenceConfig
OpenInferenceSubscriber
OpenInferenceError
OtlpTransport
Result
otel
OpenTelemetryConfig
OpenTelemetrySubscriber
OpenTelemetryError
OtlpTransport
Result
plugin_component
AtifSectionConfig
AtofSectionConfig
ComponentSpec
ObservabilityConfig
OtlpSectionConfig
S3StorageConfig
AtifStorageConfig
OBSERVABILITY_PLUGIN_KIND
deregister_observability_component
register_observability_component
plugin
ConfigDiagnostic
ConfigPolicy
ConfigReport
PluginComponentSpec
PluginConfig
PluginRegistration
PluginRegistrationContext
DiagnosticLevel
PluginError
UnsupportedBehavior
Plugin
active_plugin_report
clear_plugin_configuration
deregister_plugin
ensure_builtin_plugins_registered
initialize_plugins
list_plugin_kinds
lookup_plugin
register_plugin
rollback_registrations
validate_plugin_config
Result
plugins
nemo_guardrails
component
ComponentSpec
LocalBackendConfig
NeMoGuardrailsConfig
RemoteBackendConfig
RequestDefaultsConfig
RequestRailsConfig
RailSelector
NEMO_GUARDRAILS_PLUGIN_KIND
deregister_nemo_guardrails_component
register_nemo_guardrails_component
stream
LlmStreamWrapper
editor_config
nemo-relay-adaptive
acg
anthropic_plugin
AnthropicCachePlugin
canonicalize
canonicalize_json
canonicalize_value
normalize_whitespace
sha256_hex
capability
BackendCapabilities
CacheEconomics
CapabilityRegistry
ModelFamilyCapabilities
ProviderFeature
ir_builder
build_prompt_ir
openai_plugin
OpenAICachePlugin
passthrough
PassthroughPlugin
plugin
PluginInput
PluginOutput
ProviderPlugin
plugin_registry
PluginRegistry
policy
ArchetypeRoutingOverride
CachePolicy
PolicyEnvelope
RetentionPolicy
RewritePolicy
RoutingPolicy
ScopeRetentionOverride
TransformationClass
profile
BehavioralProfile
BlockStabilityScore
DistributionSummary
ParallelismPattern
ToolUsagePhase
SessionArchetype
StabilityClass
prompt_ir
PromptBlock
PromptIR
SpanId
TokenizationMetadata
ToolSchemaHash
BlockContentType
PromptRole
ProvenanceLabel
SensitivityLabel
retention
RetentionThresholds
generate_retention_intent
generate_retention_intent_default
stability
StabilityAnalysisResult
StabilityThresholds
analyze_stability
find_stable_prefix_length
telemetry
CacheHitRate
CacheMissDiagnosis
CacheRequestFacts
CacheTelemetryEvent
CacheMissEvidence
CacheMissReason
CacheTelemetryProvider
types
AgentIdentity
CacheStabilityIntent
CompressionIntent
ContentExtractionIntent
IntentOutcome
ModelRoutingIntent
OptimizationIntentBundle
PlacementIntent
PriorityIntent
RetentionIntent
SerializationIntent
ToolScopeIntent
TranslationReport
IntentType
ModelClass
OptimizationIntent
PlacementTarget
ReasonCode
RetentionTier
SharingScope
TranslationStatus
variable_extractor
ExtractedVariable
ExtractionResult
VariablePattern
VariableCategory
default_variable_patterns
extract_variables
extract_variables_from_blocks
error
AcgError
Result
MIN_ACG_OBSERVATIONS
acg_component
acg_learner
AcgLearner
acg_profile
adaptive_hints_intercept
AdaptiveHintsIntercept
cache_diagnostics
CacheDiagnosticsTracker
build_cache_request_facts
config
AcgComponentConfig
AdaptiveConfig
AdaptiveHintsComponentConfig
BackendSpec
StateConfig
TelemetryComponentConfig
ToolParallelismComponentConfig
context_helpers
SharedParentScopeIdentity
LATENCY_SENSITIVITY_POINTER
extract_scope_path
read_manual_latency_sensitivity
resolve_agent_id
resolve_shared_parent_scope_identity
set_latency_sensitivity
drain
error
AdaptiveError
Result
intercepts
AGENT_HINTS_HEADER_KEY
learner
latency
LatencySensitivityLearner
compute_default_hints
traits
Learner
plugin_component
ComponentSpec
ADAPTIVE_PLUGIN_KIND
deregister_adaptive_component
register_adaptive_component
redis
RedisBackend
storage
erased
AnyBackend
memory
InMemoryBackend
traits
StorageBackend
StorageBackendDyn
subscriber
tool_parallelism_learner
ToolParallelismLearner
trie
accumulator
AccumulatorState
NodeAccumulators
RunningStats
builder
PredictionTrieBuilder
SensitivityConfig
data_models
LlmCallPrediction
PredictionMetrics
PredictionTrieNode
lookup
PredictionTrieLookup
serialization
TrieEnvelope
CURRENT_VERSION
types
cache
HotCache
metadata
AgentHints
MetadataEnvelope
ParallelHint
Json
plan
ExecutionPlan
ParallelGroup
records
CallRecord
RunRecord
CallKind
AdaptiveRuntime
nemo-relay-ffi
api
FfiStream
nemo_relay_active_plugin_report_json
nemo_relay_anthropic_messages_codec_new
nemo_relay_atif_exporter_clear
nemo_relay_atif_exporter_create
nemo_relay_atif_exporter_deregister
nemo_relay_atif_exporter_export
nemo_relay_atif_exporter_register
nemo_relay_atof_exporter_create
nemo_relay_atof_exporter_deregister
nemo_relay_atof_exporter_force_flush
nemo_relay_atof_exporter_path
nemo_relay_atof_exporter_register
nemo_relay_atof_exporter_shutdown
nemo_relay_clear_plugin_configuration
nemo_relay_deregister_llm_conditional_execution_guardrail
nemo_relay_deregister_llm_execution_intercept
nemo_relay_deregister_llm_request_intercept
nemo_relay_deregister_llm_sanitize_request_guardrail
nemo_relay_deregister_llm_sanitize_response_guardrail
nemo_relay_deregister_llm_stream_execution_intercept
nemo_relay_deregister_plugin
nemo_relay_deregister_subscriber
nemo_relay_deregister_tool_conditional_execution_guardrail
nemo_relay_deregister_tool_execution_intercept
nemo_relay_deregister_tool_request_intercept
nemo_relay_deregister_tool_sanitize_request_guardrail
nemo_relay_deregister_tool_sanitize_response_guardrail
nemo_relay_event
nemo_relay_flush_subscribers
nemo_relay_get_handle
nemo_relay_initialize_plugins
nemo_relay_list_plugin_kinds_json
nemo_relay_llm_call
nemo_relay_llm_call_end
nemo_relay_llm_call_execute
nemo_relay_llm_conditional_execution
nemo_relay_llm_request_intercepts
nemo_relay_llm_stream_call_execute
nemo_relay_observability_component_spec_json
nemo_relay_observability_default_config_json
nemo_relay_observability_plugin_kind
nemo_relay_openai_chat_codec_new
nemo_relay_openai_responses_codec_new
nemo_relay_openinference_subscriber_create
nemo_relay_openinference_subscriber_deregister
nemo_relay_openinference_subscriber_force_flush
nemo_relay_openinference_subscriber_register
nemo_relay_openinference_subscriber_shutdown
nemo_relay_otel_subscriber_create
nemo_relay_otel_subscriber_deregister
nemo_relay_otel_subscriber_force_flush
nemo_relay_otel_subscriber_register
nemo_relay_otel_subscriber_shutdown
nemo_relay_plugin_context_register_llm_conditional_execution_guardrail
nemo_relay_plugin_context_register_llm_execution_intercept
nemo_relay_plugin_context_register_llm_request_intercept
nemo_relay_plugin_context_register_llm_sanitize_request_guardrail
nemo_relay_plugin_context_register_llm_sanitize_response_guardrail
nemo_relay_plugin_context_register_llm_stream_execution_intercept
nemo_relay_plugin_context_register_subscriber
nemo_relay_plugin_context_register_tool_conditional_execution_guardrail
nemo_relay_plugin_context_register_tool_execution_intercept
nemo_relay_plugin_context_register_tool_request_intercept
nemo_relay_plugin_context_register_tool_sanitize_request_guardrail
nemo_relay_plugin_context_register_tool_sanitize_response_guardrail
nemo_relay_pop_scope
nemo_relay_push_scope
nemo_relay_register_llm_conditional_execution_guardrail
nemo_relay_register_llm_execution_intercept
nemo_relay_register_llm_request_intercept
nemo_relay_register_llm_sanitize_request_guardrail
nemo_relay_register_llm_sanitize_response_guardrail
nemo_relay_register_llm_stream_execution_intercept
nemo_relay_register_plugin
nemo_relay_register_subscriber
nemo_relay_register_tool_conditional_execution_guardrail
nemo_relay_register_tool_execution_intercept
nemo_relay_register_tool_request_intercept
nemo_relay_register_tool_sanitize_request_guardrail
nemo_relay_register_tool_sanitize_response_guardrail
nemo_relay_scope_deregister_llm_conditional_execution_guardrail
nemo_relay_scope_deregister_llm_execution_intercept
nemo_relay_scope_deregister_llm_request_intercept
nemo_relay_scope_deregister_llm_sanitize_request_guardrail
nemo_relay_scope_deregister_llm_sanitize_response_guardrail
nemo_relay_scope_deregister_llm_stream_execution_intercept
nemo_relay_scope_deregister_subscriber
nemo_relay_scope_deregister_tool_conditional_execution_guardrail
nemo_relay_scope_deregister_tool_execution_intercept
nemo_relay_scope_deregister_tool_request_intercept
nemo_relay_scope_deregister_tool_sanitize_request_guardrail
nemo_relay_scope_deregister_tool_sanitize_response_guardrail
nemo_relay_scope_register_llm_conditional_execution_guardrail
nemo_relay_scope_register_llm_execution_intercept
nemo_relay_scope_register_llm_request_intercept
nemo_relay_scope_register_llm_sanitize_request_guardrail
nemo_relay_scope_register_llm_sanitize_response_guardrail
nemo_relay_scope_register_llm_stream_execution_intercept
nemo_relay_scope_register_subscriber
nemo_relay_scope_register_tool_conditional_execution_guardrail
nemo_relay_scope_register_tool_execution_intercept
nemo_relay_scope_register_tool_request_intercept
nemo_relay_scope_register_tool_sanitize_request_guardrail
nemo_relay_scope_register_tool_sanitize_response_guardrail
nemo_relay_scope_stack_active
nemo_relay_scope_stack_capture_thread
nemo_relay_scope_stack_create
nemo_relay_scope_stack_restore_thread
nemo_relay_scope_stack_set_thread
nemo_relay_stream_free
nemo_relay_stream_next
nemo_relay_tool_call
nemo_relay_tool_call_end
nemo_relay_tool_call_execute
nemo_relay_tool_conditional_execution
nemo_relay_tool_request_intercepts
nemo_relay_validate_plugin_config
callable
wrap_codec_fn
wrap_collector_fn
wrap_event_subscriber
wrap_finalizer_fn
wrap_json_fn
wrap_llm_conditional_fn
wrap_llm_exec_fn
wrap_llm_exec_intercept_fn
wrap_llm_request_intercept_fn
wrap_llm_response_fn
wrap_llm_sanitize_request_fn
wrap_llm_stream_exec_fn
wrap_llm_stream_exec_intercept_fn
wrap_tool_conditional_fn
wrap_tool_exec_fn
wrap_tool_exec_intercept_fn
wrap_tool_request_intercept_fn
wrap_tool_sanitize_fn
NemoRelayCodecDecodeCb
NemoRelayCodecDecodeFn
NemoRelayCodecEncodeCb
NemoRelayCodecEncodeFn
NemoRelayCollectorCb
NemoRelayEventSubscriberCb
NemoRelayFinalizerCb
NemoRelayFreeFn
NemoRelayJsonCb
NemoRelayLlmConditionalCb
NemoRelayLlmExecCb
NemoRelayLlmExecInterceptCb
NemoRelayLlmExecNextFn
NemoRelayLlmRequestCb
NemoRelayLlmRequestInterceptCb
NemoRelayPluginRegisterCb
NemoRelayPluginValidateCb
NemoRelayToolConditionalCb
NemoRelayToolExecCb
NemoRelayToolExecInterceptCb
NemoRelayToolExecNextFn
NemoRelayToolSanitizeCb
convert
c_str_to_json
c_str_to_opt_json
c_str_to_string
json_to_c_string
nemo_relay_string_free
str_to_c_string
unix_micros_to_opt_timestamp
error
NemoRelayStatus
clear_last_error
last_error_message
nemo_relay_last_error
nemo_relay_set_last_error_message
set_last_error
status_from_error
status_from_plugin_error
types
FfiAtifExporter
FfiAtofExporter
FfiCodecHandle
FfiEvent
FfiLLMHandle
FfiLLMRequest
FfiOpenInferenceSubscriber
FfiOpenTelemetrySubscriber
FfiPluginContext
FfiScopeHandle
FfiScopeStack
FfiThreadScopeStackBinding
FfiToolHandle
NemoRelayScopeType
nemo_relay_atif_exporter_free
nemo_relay_atof_exporter_free
nemo_relay_codec_free
nemo_relay_event_annotated_request
nemo_relay_event_annotated_response
nemo_relay_event_atof_version
nemo_relay_event_attributes
nemo_relay_event_attributes_json
nemo_relay_event_category
nemo_relay_event_category_profile
nemo_relay_event_data
nemo_relay_event_data_schema
nemo_relay_event_free
nemo_relay_event_input
nemo_relay_event_json
nemo_relay_event_kind
nemo_relay_event_metadata
nemo_relay_event_model_name
nemo_relay_event_name
nemo_relay_event_output
nemo_relay_event_parent_uuid
nemo_relay_event_scope_category
nemo_relay_event_scope_type
nemo_relay_event_timestamp
nemo_relay_event_tool_call_id
nemo_relay_event_uuid
nemo_relay_llm_handle_attributes
nemo_relay_llm_handle_free
nemo_relay_llm_handle_name
nemo_relay_llm_handle_parent_uuid
nemo_relay_llm_handle_uuid
nemo_relay_llm_request_content
nemo_relay_llm_request_free
nemo_relay_llm_request_headers
nemo_relay_llm_request_new
nemo_relay_openinference_subscriber_free
nemo_relay_otel_subscriber_free
nemo_relay_scope_handle_attributes
nemo_relay_scope_handle_data
nemo_relay_scope_handle_free
nemo_relay_scope_handle_metadata
nemo_relay_scope_handle_name
nemo_relay_scope_handle_parent_uuid
nemo_relay_scope_handle_scope_type
nemo_relay_scope_handle_uuid
nemo_relay_scope_stack_free
nemo_relay_tool_handle_attributes
nemo_relay_tool_handle_free
nemo_relay_tool_handle_name
nemo_relay_tool_handle_parent_uuid
nemo_relay_tool_handle_uuid
Performance
Support and FAQs
Glossary
Troubleshooting Guide
Trace Incident Runbook
Community
Legal
Open Source Software (OSS)
License Agreement

Resources

Legal

||View as Markdown|

Use these pages for open source software attribution and license information.

Developer-friendly docs for your API