AIStore at Multi-Petabyte Scale: NVMe versus HDD
AIStore at Multi-Petabyte Scale: NVMe versus HDD
When we consider future multi-petabyte deployments for AIStore, there’s certainly the hyper-converged option. But there’s also one other option that never gets old: today’s enterprise-grade HDDs.
Here is a brief back-of-the-envelope comparison. All prices below are USD, and the entire analysis rests on information in the public domain (references below).
An OCI BM.DenseIO.E5.128 provides 12 x 6.8 TB NVMe (81.6 TB raw), a single 100 Gb/s interface, and costs approximately $11.91/hour at list price — $104,296/year.
We have measured this shape. In MLPerf Storage v3, twelve E5 nodes delivered 136.54 GiB/s of checkpoint-recovery read throughput against 139.7 GiB/s of nominal aggregate NIC bandwidth — 97.7%. On this platform the network interface, not the NVMe behind it, is the ceiling.
A conventional HDD server with 24 x 24 TB drives provides 576 TB raw. Seagate rates the Exos X24 at up to 285 MB/s, which is an outer-track, single-stream figure - so let’s assume 200-230 MB/s sustained. Twenty-four drives at that derated rate give roughly 4.8-5.5 GB/s, or 38-44 Gb/s, of aggregate sequential media bandwidth. A 25 GbE link per storage node therefore stays comfortably inside what the media can deliver even at the pessimistic end — deliberately so, since the E5 side of this comparison is measured and this side is arithmetic.
For simplicity, we further assume a 24-HDD node with a single 25 GbE interface - 4 nodes providing 100 Gbps in aggregate.
This also assumes enough concurrent object activity for AIStore to spread reads across disks and nodes. A single or low-concurrency stream remains bounded by one HDD, and small random reads are bounded by seek time rather than by any of the numbers above. The comparison that follows is a large-object, high-concurrency comparison and nothing else.
At a current bulk-market asking price of approx. $16-$17.5/TB for 24TB enterprise drives, the installed HDD media comes to:
Servers, networking, spares, support, and power must of course be added. Erasure coding and free-space watermarks are not on that list: both apply to either architecture at the same overhead, so they cancel out of the ratio. This is also CAPEX versus OCI list-price OPEX, so it is not yet a proper TCO comparison. But the order-of-magnitude difference is difficult to miss.
The Bandwidth You May Never Need
At matched capacity, the two configurations are nowhere near matched on bandwidth:
That is the honest headline - and the question it raises is not whether 275 Gb/s equals 7,400 Gb/s. It is why the 7,400 Gb/s was purchased at all. At 6 PB, DenseIO is the shape that holds the capacity; the bandwidth is what came attached.
And if a deployment does need a tenth of that nominal figure, size for it directly: thirty HDD servers provide a 750 Gb/s aggregate network ceiling — enough for 740 Gb/s — along with 17 PB of raw capacity and $276K-$302K of media. The E5 side does not move — the same 74 nodes are required by the 6 PB capacity floor. On the cost basis used here, that is $276K-$302K of one-time HDD media versus $7.7M/year for complete E5 nodes.
Where throughput genuinely is the requirement, size for it directly: one OCI E5 node provides 100 Gb/s and 81.6 TB; four 25 GbE HDD nodes provide 100 Gb/s and 2.3 PB; five provide 125 Gb/s and 2.9 PB. For concurrent large-object workloads, a one-to-four or one-to-five ratio can plausibly match application-level aggregate throughput while providing 28-35 times more raw capacity.
Isolating the storage
Oracle itemizes the E5 NVMe component at $0.0612/TB-hour.
For 81.6 TB, that is $43.7K/year — about 42% of the full-node price. The remaining 58% includes CPU and memory, which an HDD storage node also needs and which is priced into neither column below.
Amortizing HDD media over a five-year service life puts both sides in $/year:
The last column divides each cost basis by its network ceiling: for four HDD nodes, $7.4K-$8.1K/year over 12.5 GB/s gives approximately $0.6K/(GB/s)/year. The two bases are not the same kind of cost, though — the E5 figure is rent that continues as long as the node runs, while the HDD figure is a purchase spread across an assumed five-year life.
Roughly six to one on price per unit of delivered throughput, at 28 to 35 times the capacity. The five-year life is an assumption; at three years the media column rises to $12.3K-$13.4K and the ratio narrows to about 3.5x, which does not change the conclusion.
Six to eight petabytes is expensive when every byte must reside on high-end NVMe. It does not follow that every 6-8 PB AIStore deployment must be expensive.