C++ API¶

Class Hierarchy¶

- Namespace nvidia
  - Namespace nvidia::inferenceserver
    - Namespace nvidia::inferenceserver::client
      - Class Error
      - Class InferContext
        Struct InferContext::Stat
        Class InferContext::Input
        Class InferContext::Options
        Class InferContext::Output
        Class InferContext::Request
        Class InferContext::Result
        Struct Result::ClassResult
      - Class InferGrpcContext
      - Class InferGrpcStreamContext
      - Class InferHttpContext
      - Class ModelControlContext
      - Class ModelControlGrpcContext
      - Class ModelControlHttpContext
      - Class ModelRepositoryContext
      - Class ModelRepositoryGrpcContext
      - Class ModelRepositoryHttpContext
      - Class ServerHealthContext
      - Class ServerHealthGrpcContext
      - Class ServerHealthHttpContext
      - Class ServerStatusContext
      - Class ServerStatusGrpcContext
      - Class ServerStatusHttpContext
      - Class SharedMemoryControlContext
      - Class SharedMemoryControlGrpcContext
      - Class SharedMemoryControlHttpContext
    - Namespace nvidia::inferenceserver::custom
      - Class CustomInstance
- Struct cudaIpcMemHandle_t
- Struct custom_initdata_struct
- Struct custom_payload_struct
- Enum custom_memorytype_enum
- Enum custom_serverparamkind_enum
- Enum trtserver_errorcode_enum
- Enum trtserver_memorytype_enum
- Enum trtserver_metricformat_enum
- Enum trtserver_modelcontrolmode_enum
- Enum trtserver_requestoptionsflag_enum
- Enum trtserver_traceactivity_enum
- Enum trtserver_tracelevel_enum

File Hierarchy¶

- Directory src
  - Directory backends
    - Directory custom
      - File custom.h
  - Directory clients
    - Directory c++
      - Directory library
        File request.h
        File request_grpc.h
        File request_http.h
  - Directory core
    - File trtserver.h
  - Directory custom
    - Directory sdk
      - File custom_instance.h

Full API¶

Namespaces¶

Namespace nvidia
- Namespaces

Namespace nvidia::inferenceserver
- Namespaces
- Typedefs

Namespace nvidia::inferenceserver::client
- Classes
- Functions

Namespace nvidia::inferenceserver::custom
- Classes

Classes and Structs¶

Struct cudaIpcMemHandle_t
- Struct Documentation

Struct custom_initdata_struct
- Struct Documentation

Struct custom_payload_struct
- Struct Documentation

Struct Result::ClassResult
- Nested Relationships
- Struct Documentation

Struct InferContext::Stat
- Nested Relationships
- Struct Documentation

Class Error
- Class Documentation

Class InferContext
- Nested Relationships
  - Nested Types
- Class Documentation

Class InferContext::Input
- Nested Relationships
- Class Documentation

Class InferContext::Options
- Nested Relationships
- Class Documentation

Class InferContext::Output
- Nested Relationships
- Class Documentation

Class InferContext::Request
- Nested Relationships
- Class Documentation

Class InferContext::Result
- Nested Relationships
  - Nested Types
- Class Documentation

Class InferGrpcContext
- Class Documentation

Class InferGrpcStreamContext
- Class Documentation

Class InferHttpContext
- Class Documentation

Class ModelControlContext
- Class Documentation

Class ModelControlGrpcContext
- Class Documentation

Class ModelControlHttpContext
- Class Documentation

Class ModelRepositoryContext
- Class Documentation

Class ModelRepositoryGrpcContext
- Class Documentation

Class ModelRepositoryHttpContext
- Class Documentation

Class ServerHealthContext
- Class Documentation

Class ServerHealthGrpcContext
- Class Documentation

Class ServerHealthHttpContext
- Class Documentation

Class ServerStatusContext
- Class Documentation

Class ServerStatusGrpcContext
- Class Documentation

Class ServerStatusHttpContext
- Class Documentation

Class SharedMemoryControlContext
- Class Documentation

Class SharedMemoryControlGrpcContext
- Class Documentation

Class SharedMemoryControlHttpContext
- Class Documentation

Class CustomInstance
- Class Documentation

Enums¶

Enum custom_memorytype_enum
- Enum Documentation

Enum custom_serverparamkind_enum
- Enum Documentation

Enum trtserver_errorcode_enum
- Enum Documentation

Enum trtserver_memorytype_enum
- Enum Documentation

Enum trtserver_metricformat_enum
- Enum Documentation

Enum trtserver_modelcontrolmode_enum
- Enum Documentation

Enum trtserver_requestoptionsflag_enum
- Enum Documentation

Enum trtserver_traceactivity_enum
- Enum Documentation

Enum trtserver_tracelevel_enum
- Enum Documentation

Functions¶

Function CustomErrorString
- Function Documentation

Function CustomExecute
- Function Documentation

Function CustomExecuteV2
- Function Documentation

Function CustomFinalize
- Function Documentation

Function CustomInitialize
- Function Documentation

Function CustomVersion
- Function Documentation

Function nvidia::inferenceserver::client::operator<<
- Function Documentation

Function TRTSERVER_ErrorCode
- Function Documentation

Function TRTSERVER_ErrorCodeString
- Function Documentation

Function TRTSERVER_ErrorDelete
- Function Documentation

Function TRTSERVER_ErrorMessage
- Function Documentation

Function TRTSERVER_ErrorNew
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsAddClassificationOutput
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsAddInput
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsAddOutput
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsDelete
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsNew
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsSetBatchSize
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsSetCorrelationId
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsSetFlags
- Function Documentation

Function TRTSERVER_InferenceRequestOptionsSetId
- Function Documentation

Function TRTSERVER_InferenceRequestProviderDelete
- Function Documentation

Function TRTSERVER_InferenceRequestProviderInputBatchByteSize
- Function Documentation

Function TRTSERVER_InferenceRequestProviderNew
- Function Documentation

Function TRTSERVER_InferenceRequestProviderNewV2
- Function Documentation

Function TRTSERVER_InferenceRequestProviderSetInputData
- Function Documentation

Function TRTSERVER_InferenceResponseDelete
- Function Documentation

Function TRTSERVER_InferenceResponseHeader
- Function Documentation

Function TRTSERVER_InferenceResponseOutputData
- Function Documentation

Function TRTSERVER_InferenceResponseStatus
- Function Documentation

Function TRTSERVER_MetricsDelete
- Function Documentation

Function TRTSERVER_MetricsFormatted
- Function Documentation

Function TRTSERVER_ProtobufDelete
- Function Documentation

Function TRTSERVER_ProtobufSerialize
- Function Documentation

Function TRTSERVER_ResponseAllocatorDelete
- Function Documentation

Function TRTSERVER_ResponseAllocatorNew
- Function Documentation

Function TRTSERVER_ServerDelete
- Function Documentation

Function TRTSERVER_ServerId
- Function Documentation

Function TRTSERVER_ServerInferAsync
- Function Documentation

Function TRTSERVER_ServerIsLive
- Function Documentation

Function TRTSERVER_ServerIsReady
- Function Documentation

Function TRTSERVER_ServerLoadModel
- Function Documentation

Function TRTSERVER_ServerMetrics
- Function Documentation

Function TRTSERVER_ServerModelRepositoryIndex
- Function Documentation

Function TRTSERVER_ServerModelStatus
- Function Documentation

Function TRTSERVER_ServerNew
- Function Documentation

Function TRTSERVER_ServerOptionsAddTensorFlowVgpuMemoryLimits
- Function Documentation

Function TRTSERVER_ServerOptionsDelete
- Function Documentation

Function TRTSERVER_ServerOptionsNew
- Function Documentation

Function TRTSERVER_ServerOptionsSetExitOnError
- Function Documentation

Function TRTSERVER_ServerOptionsSetExitTimeout
- Function Documentation

Function TRTSERVER_ServerOptionsSetGpuMetrics
- Function Documentation

Function TRTSERVER_ServerOptionsSetLogError
- Function Documentation

Function TRTSERVER_ServerOptionsSetLogInfo
- Function Documentation

Function TRTSERVER_ServerOptionsSetLogVerbose
- Function Documentation

Function TRTSERVER_ServerOptionsSetLogWarn
- Function Documentation

Function TRTSERVER_ServerOptionsSetMetrics
- Function Documentation

Function TRTSERVER_ServerOptionsSetModelControlMode
- Function Documentation

Function TRTSERVER_ServerOptionsSetModelRepositoryPath
- Function Documentation

Function TRTSERVER_ServerOptionsSetPinnedMemoryPoolByteSize
- Function Documentation

Function TRTSERVER_ServerOptionsSetServerId
- Function Documentation

Function TRTSERVER_ServerOptionsSetStartupModel
- Function Documentation

Function TRTSERVER_ServerOptionsSetStrictModelConfig
- Function Documentation

Function TRTSERVER_ServerOptionsSetStrictReadiness
- Function Documentation

Function TRTSERVER_ServerOptionsSetTensorFlowGpuMemoryFraction
- Function Documentation

Function TRTSERVER_ServerOptionsSetTensorFlowSoftPlacement
- Function Documentation

Function TRTSERVER_ServerPollModelRepository
- Function Documentation

Function TRTSERVER_ServerRegisterSharedMemory
- Function Documentation

Function TRTSERVER_ServerSharedMemoryAddress
- Function Documentation

Function TRTSERVER_ServerSharedMemoryStatus
- Function Documentation

Function TRTSERVER_ServerStatus
- Function Documentation

Function TRTSERVER_ServerStop
- Function Documentation

Function TRTSERVER_ServerUnloadModel
- Function Documentation

Function TRTSERVER_ServerUnregisterAllSharedMemory
- Function Documentation

Function TRTSERVER_ServerUnregisterSharedMemory
- Function Documentation

Function TRTSERVER_SharedMemoryBlockCpuNew
- Function Documentation

Function TRTSERVER_SharedMemoryBlockDelete
- Function Documentation

Function TRTSERVER_SharedMemoryBlockGpuNew
- Function Documentation

Function TRTSERVER_SharedMemoryBlockMemoryType
- Function Documentation

Function TRTSERVER_SharedMemoryBlockMemoryTypeId
- Function Documentation

Function TRTSERVER_TraceDelete
- Function Documentation

Function TRTSERVER_TraceId
- Function Documentation

Function TRTSERVER_TraceManagerDelete
- Function Documentation

Function TRTSERVER_TraceManagerNew
- Function Documentation

Function TRTSERVER_TraceModelName
- Function Documentation

Function TRTSERVER_TraceModelVersion
- Function Documentation

Function TRTSERVER_TraceNew
- Function Documentation

Function TRTSERVER_TraceParentId
- Function Documentation

Defines¶

Define CUSTOM_NO_GPU_DEVICE
- Define Documentation

Define CUSTOM_SERVER_PARAMETER_CNT
- Define Documentation

Define DECLSPEC
- Define Documentation

Define TRTIS_CLIENT_HEADER_FLAT
- Define Documentation

Define TRTIS_CLIENT_HEADER_FLAT
- Define Documentation

Define TRTIS_CLIENT_HEADER_FLAT
- Define Documentation

Define TRTIS_CUSTOM_EXPORT
- Define Documentation

Define TRTSERVER_EXPORT
- Define Documentation

Typedefs¶

Typedef cudaIpcMemHandle_t
- Typedef Documentation

Typedef CustomErrorStringFn_t
- Typedef Documentation

Typedef CustomExecuteFn_t
- Typedef Documentation

Typedef CustomExecuteV2Fn_t
- Typedef Documentation

Typedef CustomFinalizeFn_t
- Typedef Documentation

Typedef CustomGetNextInputFn_t
- Typedef Documentation

Typedef CustomGetNextInputV2Fn_t
- Typedef Documentation

Typedef CustomGetOutputFn_t
- Typedef Documentation

Typedef CustomGetOutputV2Fn_t
- Typedef Documentation

Typedef CustomInitializeData
- Typedef Documentation

Typedef CustomInitializeFn_t
- Typedef Documentation

Typedef CustomMemoryType
- Typedef Documentation

Typedef CustomPayload
- Typedef Documentation

Typedef CustomServerParameter
- Typedef Documentation

Typedef CustomVersionFn_t
- Typedef Documentation

Typedef nvidia::inferenceserver::CorrelationID
- Typedef Documentation

Typedef nvidia::inferenceserver::DimsList
- Typedef Documentation

Typedef TRTSERVER_Error_Code
- Typedef Documentation

Typedef TRTSERVER_InferenceCompleteFn_t
- Typedef Documentation

Typedef TRTSERVER_Memory_Type
- Typedef Documentation

Typedef TRTSERVER_Metric_Format
- Typedef Documentation

Typedef TRTSERVER_Model_Control_Mode
- Typedef Documentation

Typedef TRTSERVER_Request_Options_Flag
- Typedef Documentation

Typedef TRTSERVER_ResponseAllocatorAllocFn_t
- Typedef Documentation

Typedef TRTSERVER_ResponseAllocatorReleaseFn_t
- Typedef Documentation

Typedef TRTSERVER_Trace_Activity
- Typedef Documentation

Typedef TRTSERVER_Trace_Level
- Typedef Documentation

Typedef TRTSERVER_TraceActivityFn_t
- Typedef Documentation

Typedef TRTSERVER_TraceManagerCreateTraceFn_t
- Typedef Documentation

Typedef TRTSERVER_TraceManagerReleaseTraceFn_t
- Typedef Documentation