> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aistore/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aistore/_mcp/server.
Oct 31, 2026 · Justin Mehes
aistore tls-offload zero-copy
AIStore [v5.1](https://github.com/NVIDIA/aistore/releases/tag/v1.5.1) introduced kernel TLS (kTLS) as an opt-in feature. [v5.2](https://github.com/NVIDIA/aistore/releases/tag/v1.5.2#ktls) enables it by default for HTTPS targets on Linux. kTLS moves response encryption from AIStore to the Linux kernel or a NIC with TLS offload support. This lets eligible HTTPS GETs use the [`sendfile(2)` fast path](https://aistore.nvidia.com/blog/2026/07/01/sendfile), reducing copies through userspace and CPU work per GET. In our initial benchmarks, hardware kTLS doubled HTTPS GET throughput and nearly halved target CPU time per GiB transferred. These TLS 1.3 hardware tests used a patched kernel; see [Setup](#setup) and [Limitations](#limitations). --- ## Table of Contents - [Background](#background) - [Configuration](#configuration) - [Benchmarks](#benchmarks) - [Setup](#setup) - [Results](#results) - [Limitations](#limitations) - [Conclusion](#conclusion) - [References](#references) --- ## Background With userspace TLS, the target reads object data from a local file into a userspace buffer, encrypts it with Go's `crypto/tls`, and writes the encrypted data back to the kernel for transmission. Both copying and encryption consume CPU time. With kTLS, AIStore still establishes the TLS session in userspace, then passes the transmit keys to the kernel. Outgoing data is encrypted in one of two modes: 1. **Software kTLS**: the kernel encrypts the data on the host CPU. 2. **Hardware kTLS**: a NIC with TLS offload support, such as the ConnectX-7, encrypts the data as it transmits the packets. Both modes let eligible warm HTTPS GETs use `sendfile(2)` without copying object data through userspace. Hardware kTLS also moves encryption from the host CPU to the NIC. These CPU savings can increase throughput when the target is CPU-bound. ## Configuration kTLS performs best with kernel auto-tuned socket buffers (`net.l4.sndrcv_buf_size=-1`). Starting with v5.2, new deployments set this value automatically. Existing clusters retain their setting after an upgrade; if it is not `-1`, change it and restart the targets. Small fixed buffers, including the 128 KiB AIS default, require frequent refills that can offset much of the CPU savings. To disable kTLS, set the `Disable-KTLS` feature flag and restart the targets. The flag is reserved for internal use; its name and meaning may change in a later release. It uses the same bit as v5.1's `System-Reserved-KTLS`, with the opposite meaning. A cluster that set the old flag therefore runs with kTLS disabled after upgrading. To re-enable it, remove `Disable-KTLS` from the feature list and restart the targets. Use [`scripts/check-ktls.sh`](https://github.com/NVIDIA/aistore/blob/main/scripts/check-ktls.sh) for a basic host check. With `--armed`, it reports kTLS transmit connection counters in the current network namespace; run it in the same host or container as the target. These counters cover all kTLS users in that namespace. For compatibility and configuration details, see [kTLS transmit offload](https://github.com/NVIDIA/aistore/blob/main/docs/https.md#ktls-transmit-offload) and the [v5.2 release notes](https://github.com/NVIDIA/aistore/releases/tag/v1.5.2#ktls). ## Benchmarks ### Setup The AIS node ran one proxy and one target with 6 vCPUs and 12 GiB of memory. A crypto-enabled ConnectX-7 NIC rated at 200 Gb/s (about 23.28 GiB/s) connected it directly to the client node, which ran [`aisloader`](https://github.com/NVIDIA/aistore/blob/main/docs/aisloader.md) with 24 vCPUs and 48 GiB of memory. Go negotiates TLS 1.3 by default, but the upstream Linux kernel supports hardware kTLS only for TLS 1.2. The AIS node therefore ran Linux `7.3.0-rc3` with the [v17 patch series](https://lists.openwall.net/netdev/2026/09/17/681) that adds TLS 1.3 hardware offload. Objects were read once before the runs to warm the page cache. Separate [`strace`](https://man7.org/linux/man-pages/man1/strace.1.html) checks showed `sendfile` calls in both kTLS modes and none in userspace TLS. In the hardware kTLS runs, the NIC reported encrypted bytes equal to the GET payload. ### Results These are our initial measurements. The performance work is ongoing: broader testing, tuning, and analysis remain. The tables compare the three TLS 1.3 modes at 64 concurrent workers, with unencrypted HTTP using `sendfile(2)` as a reference. At this load, the target CPU was close to saturation in the TLS runs. Average GET throughput, in GiB/s (higher is better): | Object size | HTTP | Userspace TLS | Software kTLS | Hardware kTLS | |---|---:|---:|---:|---:| | 1 MiB | 14.33 | 4.14 | 5.04 | 7.54 | | 8 MiB | 20.71 | 4.71 | 8.92 | 10.75 | | 64 MiB | 21.39 | 5.54 | 7.92 | 11.60 | Target CPU-seconds per GiB transferred, including user and system time (lower is better): | Object size | HTTP | Userspace TLS | Software kTLS | Hardware kTLS | |---|---:|---:|---:|---:| | 1 MiB | 0.25 | 1.15 | 0.93 | 0.59 | | 8 MiB | 0.12 | 1.09 | 0.63 | 0.51 | | 64 MiB | 0.09 | 0.95 | 0.74 | 0.50 | Hardware kTLS delivered the largest improvement. With 64 MiB objects, throughput rose from 5.54 to 11.60 GiB/s, about **2x** userspace TLS, while CPU-seconds per GiB dropped by nearly **half**. Software kTLS still uses the host CPU for encryption, but delivered 1.2x to 1.9x the throughput of userspace TLS and lowered CPU cost per GiB at every size. Each GET also carries fixed request-handling costs that kTLS does not remove. Hardware kTLS's throughput gain over userspace TLS was 1.8x at 1 MiB and 2.1x to 2.3x at 8 and 64 MiB. The results do not show a steady increase with object size: software kTLS had higher throughput and lower CPU cost per GiB at 8 MiB than at 64 MiB. More testing is needed to explain that difference. Only plain HTTP approached the NIC's rated bandwidth of 23.28 GiB/s. Even with hardware kTLS, HTTPS used 2.4x to 5.6x the CPU time per GiB of plain HTTP in these runs. ## Limitations The largest gains in these tests came from hardware kTLS. At the time of writing, TLS 1.3 hardware offload requires a patched kernel. On upstream kernels, TLS 1.3 uses software kTLS. AIS decides whether to arm kTLS once per connection, before writing the first response. That response must serve an eligible monolithic GET of at least 1 MiB. A HEAD, smaller GET, or other ineligible first response leaves the connection in userspace TLS for its lifetime. Use a fresh connection when checking offload. The `sendfile(2)` eligibility rules from the [`sendfile` post](https://aistore.nvidia.com/blog/2026/07/01/sendfile#limitations) still apply. Chunked objects, range reads that require checksum validation, and transformed responses such as [archive extraction](https://github.com/NVIDIA/aistore/blob/main/docs/archive.md#operations) and [ETL](https://github.com/NVIDIA/aistore/blob/main/docs/etl.md) use the regular transmit path. If the connection is already armed for kTLS, encryption remains offloaded, but these responses do not use `sendfile(2)`. ## Conclusion These initial results show that both software and hardware kTLS can improve HTTPS throughput and reduce CPU time per GiB. The largest gains came from hardware offload. Further testing and tuning will help establish how those gains vary across workloads and systems. If you run AIStore with kTLS, we would like to hear about your results. --- ## References __AIStore__ - [Improving GET Performance with Zero-Copy File Transfers](https://aistore.nvidia.com/blog/2026/07/01/sendfile) - [kTLS transmit offload](https://github.com/NVIDIA/aistore/blob/main/docs/https.md#ktls-transmit-offload) - [AIStore v5.2 release notes](https://github.com/NVIDIA/aistore/releases/tag/v1.5.2#ktls) - [Feature flags](https://github.com/NVIDIA/aistore/blob/main/docs/feature_flags.md) - [Archives](https://github.com/NVIDIA/aistore/blob/main/docs/archive.md) - [ETL](https://github.com/NVIDIA/aistore/blob/main/docs/etl.md) - [aisloader](https://github.com/NVIDIA/aistore/blob/main/docs/aisloader.md) - [`scripts/check-ktls.sh`](https://github.com/NVIDIA/aistore/blob/main/scripts/check-ktls.sh) __Linux__ - [Kernel TLS](https://docs.kernel.org/networking/tls.html) - [Kernel TLS offload](https://docs.kernel.org/networking/tls-offload.html) - [TLS 1.3 hardware offload, v17 patch series](https://lists.openwall.net/netdev/2026/09/17/681) - [`sendfile(2)`](https://man7.org/linux/man-pages/man2/sendfile.2.html) - [`strace(1)`](https://man7.org/linux/man-pages/man1/strace.1.html)