> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aistore/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aistore/_mcp/server.
Sep 02, 2026
·
Alex Aizman
aistore
multi-petabyte
storage-media
cost
When we consider future multi-petabyte deployments for AIStore, there's certainly the hyper-converged option.
But there's also one other option that never gets old: today's enterprise-grade HDDs.
Here is a brief back-of-the-envelope comparison. All prices below are USD, and the entire analysis rests on information in the public domain (references below).
An OCI [`BM.DenseIO.E5.128`](https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm#bm-dense) provides 12 x 6.8 TB NVMe (81.6 TB raw), a single 100 Gb/s interface, and costs approximately \$11.91/hour at list price -- \$104,296/year.
We have measured this shape. In [MLPerf Storage v3](https://docs.nvidia.com/aistore/blog/mlperf-storage-v3), twelve E5 nodes delivered 136.54 GiB/s of checkpoint-recovery read throughput against 139.7 GiB/s of nominal aggregate NIC bandwidth -- 97.7%. On this platform the network interface, not the NVMe behind it, is the ceiling.
| Raw capacity | E5 nodes | Annual list price |
| ---: | ---: | ---: |
| 6 PB | 74 | \$7.7M |
| 8 PB | 99 | \$10.3M |
A conventional HDD server with 24 x 24 TB drives provides 576 TB raw.
Seagate rates the Exos X24 at up to 285 MB/s, which is an outer-track, single-stream figure - so let's assume 200-230 MB/s sustained.
Twenty-four drives at that derated rate give roughly 4.8-5.5 GB/s, or 38-44 Gb/s, of aggregate sequential media bandwidth.
A 25 GbE link per storage node therefore stays comfortably inside what the media can deliver even at the pessimistic end --
deliberately so, since the E5 side of this comparison is measured and this side is arithmetic.
> For simplicity, we further assume a 24-HDD node with a single 25 GbE interface - 4 nodes providing 100 Gbps in aggregate.
This also assumes enough concurrent object activity for AIStore to spread reads across disks and nodes.
A single or low-concurrency stream remains bounded by one HDD, and small random reads are bounded by seek time rather than by any of the numbers above.
The comparison that follows is a large-object, high-concurrency comparison and nothing else.
| Raw capacity | HDD servers | Drives | Aggregate 25 GbE |
| ---: | ---: | ---: | ---: |
| 6 PB | 11 | 264 | 275 Gb/s |
| 8 PB | 14 | 336 | 350 Gb/s |
At a current bulk-market asking price of approx. \$16-\$17.5/TB for 24TB enterprise drives, the installed HDD media comes to:
| Raw capacity | Drives | Media | Media cost |
| ---: | ---: | ---: | ---: |
| 6 PB | 264 | 6,336 TB | \$101K-\$111K |
| 8 PB | 336 | 8,064 TB | \$129K-\$141K |
Servers, networking, spares, support, and power must of course be added. Erasure coding and free-space watermarks are not on that list: both apply to either architecture at the same overhead, so they cancel out of the ratio.
This is also CAPEX versus OCI list-price OPEX, so it is not yet a proper TCO comparison. But the order-of-magnitude difference is difficult to miss.
## The Bandwidth You May Never Need
At matched capacity, the two configurations are nowhere near matched on bandwidth:
| Raw capacity | E5 aggregate | HDD aggregate | Ratio |
| ---: | ---: | ---: | ---: |
| 6 PB | 7,400 Gb/s | 275 Gb/s | 26.9x |
| 8 PB | 9,900 Gb/s | 350 Gb/s | 28.3x |
That is the honest headline - and the question it raises is not whether 275 Gb/s equals 7,400 Gb/s.
It is why the 7,400 Gb/s was purchased at all. At 6 PB, DenseIO is the shape that holds the capacity; the bandwidth is what came attached.
And if a deployment does need a tenth of that nominal figure, size for it directly: thirty HDD servers provide a 750 Gb/s aggregate network ceiling -- enough for 740 Gb/s -- along with 17 PB of raw capacity and \$276K-\$302K of media.
The E5 side does not move -- the same 74 nodes are required by the 6 PB capacity floor. On the cost basis used here, that is \$276K-\$302K of one-time HDD media versus \$7.7M/year for complete E5 nodes.
Where throughput genuinely is the requirement, size for it directly:
one OCI E5 node provides 100 Gb/s and 81.6 TB; four 25 GbE HDD nodes provide 100 Gb/s and 2.3 PB; five provide 125 Gb/s and 2.9 PB.
For concurrent large-object workloads, a one-to-four or one-to-five ratio can plausibly match application-level aggregate throughput while providing 28-35 times more raw capacity.
## Isolating the storage
Oracle [itemizes the E5 NVMe component at $0.0612/TB-hour](https://www.oracle.com/asean/a/ocom/docs/corporate/pricing/oracle-paas-and-iaas-global-price-list.pdf).
For 81.6 TB, that is \$43.7K/year -- about **42%** of the full-node price.
The remaining 58% includes CPU and memory, which an HDD storage node also needs and which is priced into neither column below.
Amortizing HDD media over a five-year service life puts both sides in \$/year:
| Configuration | Network ceiling | Raw capacity | Storage cost basis | Price/throughput \$/(GB/s)/year |
| --- | ---: | ---: | ---: | ---: |
| 1 E5 node | 12.5 GB/s | 81.6 TB | \$43.7K/year NVMe service | \$3.5K |
| 4 HDD nodes | 12.5 GB/s | 2.3 PB | \$7.4K-\$8.1K/year (media, 5 yr) | \$0.6K |
| 5 HDD nodes | 15.6 GB/s | 2.9 PB | \$9.2K-\$10.1K/year (media, 5 yr) | \$0.6K |
> The last column divides each cost basis by its network ceiling: for four HDD nodes, \$7.4K-\$8.1K/year over 12.5 GB/s gives approximately \$0.6K/(GB/s)/year.
> The two bases are not the same kind of cost, though -- the E5 figure is rent that continues as long as the node runs, while the HDD figure is a purchase spread across an assumed five-year life.
Roughly six to one on price per unit of delivered throughput, at 28 to 35 times the capacity.
The five-year life is an assumption; at three years the media column rises to \$12.3K-\$13.4K and the ratio narrows to about 3.5x, which does not change the conclusion.
Six to eight petabytes is expensive when every byte must reside on high-end NVMe. It does not follow that every 6-8 PB AIStore deployment must be expensive.
## Sources
- [OCI compute shapes](https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm#bm-dense)
- [OCI pricing](https://www.oracle.com/asean/a/ocom/docs/corporate/pricing/oracle-paas-and-iaas-global-price-list.pdf)
- [Seagate Exos X24 specifications](https://www.seagate.com/support/internal-hard-drives/enterprise-hard-drives/exos-x24/)
- [AIStore MLPerf Storage v3 results](https://docs.nvidia.com/aistore/blog/mlperf-storage-v3)
- [August 2026 HDD street-price snapshot](https://hddhunt.com/blog/hdd-price-per-tb-august-2026/) 