How they use Nebius AI Cloud

WEKA and Nebius tested distributed KV caching for long-context inference on Nebius AI Cloud B300 infrastructure. Two eight-hour runs replayed captured coding-agent sessions, comparing GPU memory alone with WEKA’s Augmented Memory Grid. The purpose was to test whether shared storage could support longer or busier inference sessions. The runs kept 512 sessions active from more than 5,000 captured sessions. The open-sourced corpus was recorded through wekai’s capture router and contains over half a million requests, with contexts approaching 900,000 tokens. Scheduler concurrency was 32.

The serving setup used one eight-GPU HGX B300 node and vLLM v0.23.0 with nvidia/DeepSeek-V4-Pro-NVFP4. Augmented Memory Grid pooled local NVMe storage across eight B300 nodes over GPUDirect RDMA and InfiniBand. Nebius reported 55,694 requests served with the shared cache versus 23,015 with GPU memory alone, and cache hit rates of 93% versus 40%. Error rates were 0.64% versus 2.32%. Median time to first token fell from 5.68 seconds to 1.05 seconds. Nebius also reported 169,000 output tokens per GPU-hour versus 46,000, and p95 time to first token of 3.82 seconds versus 23.12 seconds. WEKA’s benchmark article reported 3.1 times fewer tokens recomputed. Its LinkedIn post reported 3.2 times fewer tokens recomputed. WEKA published the benchmark and identified the instances used. Nebius separately described the experiment.

WEKA also supplies storage technology to Nebius. Their June 2025 partnership announcement described integrating WEKA storage into Nebius AI Cloud. A solution brief describes NeuralMesh as a first-party Nebius service. WEKA’s case study describes 2 PB of NeuralMesh deployed alongside Nebius compute for an unnamed research institution. WEKA also describes work with Nebius on memory tiering for Nebius Token Factory. An AMG product job listing names Nebius as a GPU cloud partner for gathering inference requirements, validating benchmarks and prioritizing improvements to cost per token and SLAs. These supplier and technology collaborations are distinct from WEKA’s benchmark use.