LMSYS · SGLang
Nonprofit open-source inference research · Infrastructure
How they use Nebius AI Cloud
The SGLang team used Nebius-sponsored GPU machines to develop DeepSeek-V3.2 support, according to its September 2025 technical article. This work included Lightning Indexer support, a Native Sparse Attention backend and dynamic cache management for contexts up to 128K tokens. The article outlined planned multi-token prediction, FP8 KV caching and TileLang support. In a recorded discussion, an SGLang team member thanked Nebius for an H200 node. SGLang is hosted by LMSYS, an independent nonprofit. In April 2026, Narek T. reported Nebius support for SGLang's Blackwell adoption in a post describing donated GPU hours for researchers. He said the adoption work was open-sourced to benefit the wider community. LMSYS lists Nebius among its sponsors and partners. SGLang Summit 2026 lists Nebius as a Gold sponsor.
Nebius describes SGLang using on-demand AI Cloud clusters for DeepSeek R1 inference optimisation experiments. The stated goal was to maximize single-node inference throughput and reduce end-user latency without compromising accuracy. The teams deployed R1 to establish a performance baseline and benchmarked prompts from 100 to 10,000 tokens with 16–80 concurrent requests. SGLang worked with Nebius solution architects to identify attention, arithmetic and kernel bottlenecks. Container support and custom drivers enabled experiments with custom CUDA kernels. Experiments combined FlashAttention-3, DeepGEMM FP8 arithmetic, larger decode batches for CUDA Graphs and cache tuning. The team also implemented MLA-to-MHA switching for long prefixes, kernel fusion, partial KV cache reuse and data-parallel attention. Nebius says iteration took days and throughput stayed nearly flat across 16, 32 and 64 concurrent requests.
Nebius reports roughly twice the average throughput and lower latency on one node. Its customer-case post reports more than triple the throughput in some heavy-load scenarios, with responsive latency at 80 concurrent queries and multi-thousand-token contexts. These are completed research and optimisation projects. The teams were considering DeepSeek-V3, advanced parallelism and model pruning as next steps.
Sources
7 sourcesNebius SGLang cluster-use customer case
27 Sept 2026 · Nebius · Web page
Nebius describes SGLang's DeepSeek R1 inference experiments on on-demand Nebius AI Cloud clusters. Optimizations included FlashAttention-3, DeepGEMM FP8 arithmetic, CUDA Graphs batch tuning, MLA-to-MHA switching, kernel fusion, partial KV cache reuse and data-parallel attention. Nebius says iteration took days and throughput stayed nearly flat across 16, 32 and 64 concurrent requests. The teams were considering DeepSeek-V3, advanced parallelism and model pruning as next steps.
Open original sourceNebius SGLang customer-case post
19 May 2025 · LinkedIn · Web page
Nebius describes the LMSYS SGLang team using Nebius AI Cloud to optimize DeepSeek R1 inference on one node. It reports roughly twice the average throughput and more than triple throughput in some heavy-load scenarios, with responsive latency at 80 concurrent queries and multi-thousand-token contexts.
Open original sourceNebius SGLang · H200 compute sponsorship
20 Mar 2025 · Nebius · Video
33:30: an audience member identifies themselves as part of the SGLang team and thanks Nebius for sponsoring an H200 node. The speaker acknowledges sponsored compute.
Open original source at 33:30Blackwell research compute statement
4 Apr 2026 · LinkedIn · Web page
Narek T. describes Nebius donating GPU hours to researchers and helping SGLang adopt Blackwell hardware. He says the adoption work was open-sourced to benefit the wider community.
Open original sourceCompany context
lmsys.orgLMSYS is a nonprofit organization developing open-source AI models, datasets, evaluation tools and systems for researchers and developers. Its SGLang project provides an inference engine for serving large language and multimodal models, from a single GPU to distributed clusters. It uses prefix caching, parallel execution and other optimizations to improve response latency and throughput, with support for multiple model families and hardware platforms. Other LMSYS projects include FastChat for chatbot development and evaluation, RouteLLM for model routing, and the Vicuna language models.
LMSYS official About and Projects pages; SGLang official documentationRecent updates
20 updatesSGLang adds native Beam support and powers Reflection’s API
SGLang announced native support for Reflection’s Beam model and disclosed that it powers Reflection’s production API. The integration includes multiple precision formats, speculative decoding and context caching for agent workloads.
LMSYS OrgSGLang and Miles add launch-day support for Muse Spark Open
SGLang added inference support for Meta’s multimodal Muse Spark Open model across NVIDIA, AMD and Intel hardware. Miles added distributed reinforcement learning support, including full-parameter training and token-consistent multi-turn rollouts.
LMSYSSGLang adds launch-day support for NVIDIA Nemotron 3.5 Super VL
SGLang added serving support for NVIDIA’s new multimodal model, including continuous batching, prefix caching and speculative decoding. The announcement provides deployment configurations for hardware ranging from DGX Spark to H200 and Blackwell GPUs.
LMSYS