> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aistore/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aistore/_mcp/server.
Jul 01, 2026
·
Justin Mehes
aistore
get
performance
benchmarks
AIStore 4.7 adds a zero-copy fast path for eligible GET requests using the Linux [`sendfile`](https://man7.org/linux/man-pages/man2/sendfile.2.html) system call. Instead of copying object data through userspace on the regular transmit path, `sendfile` lets the kernel move file data directly from the local filesystem to the client socket, reducing target-side CPU overhead while preserving the same client-facing response.
In the benchmarks, `sendfile` reduced CPU time per GiB transferred by up to **70%** on AIS target nodes serving GET responses. When network bandwidth is not the bottleneck, those CPU savings translate directly into higher throughput, with benchmarks showing up to a **3x** increase. Even during a network bottleneck, the reclaimed CPU can be used for colocated work such as [ETL](https://docs.nvidia.com/aistore/etl), [jobs](https://docs.nvidia.com/aistore/cli/job), and checksum validation.
---
## Table of Contents
- [Background](#background)
- [Benchmarks](#benchmarks)
- [Object Size](#object-size)
- [Throughput Gains](#throughput-gains)
- [Limitations](#limitations)
- [TLS](#tls)
- [Chunked Objects](#chunked-objects)
- [Range Reads](#range-reads)
- [Transformed Responses](#transformed-responses)
- [Conclusion](#conclusion)
- [References](#references)
---
## Background
For every GET request to AIStore, it first reaches an AIS proxy, which redirects the request to the target that owns the object. On a warm GET, where the object is already local, the target serves the response through the regular transmit path:
1. Load the object's local metadata.
2. Open a reader for the local object.
3. Construct the HTTP response headers from the object metadata.
4. Read object data from the local file into a reusable userspace buffer.
5. Perform any required processing on the buffered data.
6. Write the buffer to the response writer, which sends the bytes to the client socket as the HTTP response body.
7. Repeat until the entire object has been transmitted.
This process is illustrated in the diagram below, but a more in-depth overview is available in [AIStore documentation](https://docs.nvidia.com/aistore/overview).
By passing object data through userspace, the transmit path supports additional request handling such as HTTPS and chunked object reads, though not every GET request needs that flexibility. When a request can be served as a plain HTTP transfer of a large, local, monolithic object, copying data through a userspace buffer becomes unnecessary overhead.
The `sendfile` path removes that extra copy. AIS still performs the same request handling, but the CPU no longer spends additional cycles copying every byte through a userspace buffer. This reduction in CPU usage within the target's data transfer component provides significant savings for the overall GET request.
## Benchmarks
### Setup
The benchmarks used a 3-node Kubernetes cluster with three AIS targets and three proxies. Each node ran Ubuntu 20.04 on a 24-core machine with about 1 TB of RAM and a 100 Gb/s NIC (corresponding to a maximum bandwidth of around 11.6 GiB/s).
The two paths were compared using separate AIS node container images: one built with `sendfile` enabled and one with it disabled. Before each run, the target `aisnode` process was traced with [`strace`](https://man7.org/linux/man-pages/man1/strace.1.html) to confirm it was taking the expected path (`sendfile` vs userspace read/write on the response body).
All benchmark objects were prepopulated into AIS buckets before the GET tests so that the results reflect what `sendfile` directly impacts: warm GETs on local objects. Every GET targeted objects owned by a single chosen target node, and that node was monitored for the results reported below.
The reported CPU and throughput numbers come from target-node [`sar`](https://man7.org/linux/man-pages/man1/sar.1.html) data collected and processed offline to filter out unrelated cluster activity during each run. Alongside those numbers, visualizations from the [AIS monitoring stack](https://github.com/NVIDIA/ais-k8s/tree/main/monitoring) are also included to provide a high-level overview of cluster performance during each run.
### Object Size
Object size is one of the largest factors in the effectiveness of `sendfile`. Each GET carries the same fixed per-request overhead (object lookup and locking, metadata load, HTTP header construction) while `sendfile` only reduces the cost of copying the object bytes into the response body, so the benefit is easier to see when the body is large enough for that copy cost to dominate the fixed per-request work.
To measure this effect, the regular transmit path was compared with the `sendfile` path for object sizes ranging from 64 KiB to 4 GiB, increasing by a factor of 4 at each step. Each size was tested for 3 minutes with 120 concurrent workers.
Both the transmit and `sendfile` object-size runs quickly reached a network ceiling of roughly 11.3 GiB/s. Because throughput was capped by the network, this test did not reveal any significant throughput differences between the two paths. Instead, the useful comparison is CPU cost at equal throughput: at comparable network throughput, the target node consumed meaningfully less CPU with `sendfile`.
The plots below normalize target CPU usage as CPU-seconds per GiB transferred and show the relative CPU reduction of `sendfile` compared with transmit.
For 64 KiB objects, `sendfile` showed no benefit in this run as the measured CPU reduction was slightly negative. That is expected for very small responses because the fixed HTTP and AIS request costs dominate the body-copy savings. The CPU reduction rates peaked at around **20%** for object sizes over 256 MiB, which suggests that per-request overhead is small relative to body transfer beyond that point.
There were minor run-to-run fluctuations from variations in network latency, but the data did show a consistent trend: `sendfile` is more CPU-efficient than the transmit path, especially for larger object sizes.
### Throughput Gains
The object-size benchmark did show CPU savings, but the network ceiling was masking potential throughput gains. To eliminate the network bottleneck, a second benchmark was conducted with the AIS client colocated on the same node as the AIS proxy and target. This setup isolates the test from network limits and shows the pure performance improvement that `sendfile` brings.
While running the AIS client on the same node as the AIS proxy and target is not a typical AIStore deployment, such a setup simulates environments where target nodes have enough network capacity that CPU becomes the limiting resource.
In the second benchmark, each test ran for 5 minutes against a bucket of 1 GiB objects, with worker count swept from 50 to 800 (doubling at each step).
In this CPU-bound setup, the results show a significant difference in throughput. The transmit path achieved 15–17 GiB/s while the `sendfile` path reached 44–50 GiB/s at roughly the same high CPU utilization. These results indicate that without network limitations, the reduced CPU cost translates directly into higher GET throughput.
The normalized CPU plots show a **62-72%** reduction in target CPU-seconds per GiB for `sendfile` compared with transmit. The reduction percentage is higher here than in the previous benchmark because the CPU-bound setup produces a compounding effect: transmit spends more CPU per delivered GiB, while `sendfile` keeps CPU cost lower and converts the saved CPU into additional throughput.
## Limitations
The `sendfile` path is intentionally narrow. AIS only uses it when the request can be served as a correctly framed HTTP response whose body is a direct view of a local file. When AIS needs to inspect, transform, encrypt, or reassemble bytes before sending them, it falls back to the regular transmit path.
### TLS
`sendfile` cannot be used for HTTPS traffic. With TLS, object bytes must be encrypted before reaching the client, which means AIS must continue reading data into userspace so the TLS stack can encrypt it before writing it to the socket.
### Chunked Objects
AIS [chunked objects](https://aistore.nvidia.com/blog/2026/03/25/parallel-download) are represented by a manifest and multiple backing chunks, not by one continuous local file that can be passed directly to `sendfile`. For chunked object reads, AIS has to resolve the object layout, open the relevant chunks, and stream them in the correct order. That reassembly work requires AIS-controlled readers, so the request remains on the transmit path.
### Range Reads
Range reads can use `sendfile` when AIS only needs to validate the HTTP `Range` header, set the `206 Partial Content` response headers, seek to the requested offset, and transfer the requested byte range directly. When AIS must compute or validate a checksum for the requested range, it may need to read the range first, materialize it, and then send it. In those cases, the transfer is no longer a pure file-backed response body, so AIS has to fall back to the transmit path.
### Transformed Responses
`sendfile` does not apply when the response body is not the original local object file. Examples include archive extraction, ETL-transformed reads, AIS Select-style processing, and other cases where AIS must generate a response from object data. For these requests, AIS needs to inspect or transform the bytes before sending them to the client. Since the response is produced by AIS rather than being a direct view of the local file, the userspace transmit path remains necessary.
## Conclusion
On eligible warm GETs, `sendfile` lowered target CPU-seconds per GiB by as much as **72%** and raised throughput by roughly **3x**. Those peaks reflect the best-case scenario from the benchmarks: CPU-bound deployments serving large, local, monolithic objects. In network-bound runs, `sendfile` delivered comparable throughput while reducing target CPU cost by about **20%** for large objects.
Despite the performance gains, the `sendfile` fast path is reserved only when the response body is a direct view of the requested file and can be transferred without userspace processing. All other GET requests automatically fall back to the transmit path. That narrow eligibility is the main limitation on `sendfile` in AIStore today. Additionally, the benchmarks showed that even eligible GETs do not benefit equally, as object size and network saturation significantly influence the effectiveness of `sendfile`.
---
## References
__AIStore__
- [AIStore Documentation](https://docs.nvidia.com/aistore/overview)
- [Parallel Download: 9x Lower Latency for Large-Object Reads](https://aistore.nvidia.com/blog/2026/03/25/parallel-download)
- [ETL](https://docs.nvidia.com/aistore/etl)
- [Jobs CLI](https://docs.nvidia.com/aistore/cli/job)
- [AIS Monitoring Stack](https://github.com/NVIDIA/ais-k8s/tree/main/monitoring)
__Linux__
- [`sendfile(2)`](https://man7.org/linux/man-pages/man2/sendfile.2.html)
- [`strace(1)`](https://man7.org/linux/man-pages/man1/strace.1.html)
- [`sar(1)`](https://man7.org/linux/man-pages/man1/sar.1.html)