InfrastructureCompute, chips & cloud
NVIDIA Shows GLM-5.3 Running Across Two DGX Stations with vLLM, Peaking Near 608 Tokens per Second
NVIDIA's tutorial splits the 753-billion-parameter GLM-5.3, quantized to NVFP4, across two desk-side DGX Stations using vLLM tensor parallelism. The pair serves the model through one OpenAI-compatible endpoint and peaked near 608 output tokens per second at 16 concurrent users.
From the NVIDIA Generative AI article of November 13, 2026 and cited sources: The NVFP4 checkpoint is about 464 GB, which is more than the 252 GB on one GB300 GPU. [1] [5] [2]
SOURCES[1] NVIDIA Technical Blog – When One GPU Isn't Enough developer.nvidia.com[5] nvidia/GLM-5.3-NVFP4 model card huggingface.co[2] Linked video: When One GPU Isn’t Enough: Serving GLM-5.3… youtube.com
AI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- The NVFP4 checkpoint is about 464 GB, which is more than the 252 GB on one GB300 GPU. [1] [5] [2]
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 13, 2026 article published
- Covered
- Nov 14 edition
- Evidence
- 8 sources across 8 sites
- 1NVIDIA Technical Blog – When One GPU Isn't Enoughdeveloper.nvidia.com
- 2Linked video: When One GPU Isn’t Enough: Serving GLM-5.3…youtube.com
- 3Z.ai – Wikipediaen.wikipedia.org
- 4Artificial Analysis – GLM-5.3artificialanalysis.ai
- 5nvidia/GLM-5.3-NVFP4 model cardhuggingface.co
- 6NVIDIA DGX Station product pagenvidia.com
QuietHow it’s measured
YouTube video · 42 views · 2 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
The NVFP4 checkpoint is about 464 GB, which is more than the 252 GB on one GB300 GPU. The guide uses tensor parallelism of two, with one GPU in each Station. The GPUs exchange partial results for every token over a 400 Gb/s RoCE link. Station A runs the API and Station B runs only a worker. [1] [5] [2]
In NVIDIA's own runs, which no one else has repeated, throughput rose from 107.4 tokens/s with one user to 607.8 with 16 users. At 32 users it fell to 385 tokens/s, and the average request took 453.5 seconds. The test used max reasoning effort on 80 coding conversations, and token counts include reasoning tokens. The model's above-median output length helps explain the long request times. [1] [4]
Getting a stable setup took specific workarounds. The guide pins NCCL 2.30.7 because older versions can deadlock when CUDA graphs are replayed across nodes, and it turns off custom and FlashInfer all-reduce. It also pins the container and model versions and checks both checkpoint copies by hash. Owners of two-node DGX Spark setups have reported similar all-reduce hangs. [1] [7]
The second Station is a design choice, not a strict requirement. One Station's 748 GB of combined CPU and GPU memory could hold the checkpoint if part of it is offloaded to CPU memory. Using two Stations keeps all weights in GPU memory. Retail listings put a single GB300 Station near $91,100. [6] [1] [8]
a frontier-class open-weight coding model ran fully on-premises behind a standard API, with pinned settings that others can reproduce. Implication: teams with data-residency needs gain a local option.
Read the full assessment
The concurrency ceiling points to team or batch use, not public-facing services.
Executive brief
NVIDIA has published a tutorial showing a 753-billion-parameter open-weight model running on two desk-side DGX Stations. Each machine holds about half of the model, and together they serve it through one OpenAI-compatible endpoint. The model is GLM-5.3, quantized to NVFP4. Its weights take about 464 GB, and one GB300 GPU has 252 GB. In NVIDIA's own benchmark the pair peaked at 607.8 output tokens/s with 16 concurrent users. At 32 users, throughput dropped to 385 tokens/s and the average request took 453.5 seconds (NVIDIA blog). No one else has reproduced these results.
What changed and event timeline
Cyber capability flagged
The US Center for AI Standards and Innovation (CAISI) reportedly judged GLM-5.3 the strongest open-weight model at cybersecurity tasks ().
GLM-5.2 base released
Z.ai released a 743B-parameter model with a 1M-token context window. GLM-5.3 is built on this base ().
GLM-5.3 announced
Z.ai improved post-training on the GLM-5.2 base without retraining it from scratch (). Artificial Analysis lists August 18 as the release date ().
NVFP4 checkpoint published
NVIDIA released a version quantized with its Model Optimizer tool on Hugging Face, under the NVIDIA Open Model Agreement plus the GLM-5.3 license ().
Two-Station serving guide
NVIDIA published the tutorial and a companion video covering setup, a test request, benchmarking and shutdown (;).
Capabilities and access
- Model:
nvidia/GLM-5.3-NVFP4, pinned to revisione3b8f9f…. 753B total parameters, about 40B active per token. Supports up to 1M tokens of context (model card). - Serving stack: container
vllm/vllm-openai:v0.30.0-aarch64, AIPerf 0.12.0 for benchmarking. The endpoint supports tool calling and three reasoning-effort levels: low, high and max (NVIDIA blog). - Hardware per Station: 252 GB HBM3e GPU memory plus 496 GB LPDDR5X CPU memory, and a ConnectX-8 network card rated up to 800 Gb/s (NVIDIA DGX Station).
Read the full section
- Model:
nvidia/GLM-5.3-NVFP4, pinned to revisione3b8f9f…. 753B total parameters, about 40B active per token. Supports up to 1M tokens of context (model card). The tutorial caps context at 98,304 tokens. - Serving stack: container
vllm/vllm-openai:v0.30.0-aarch64, AIPerf 0.12.0 for benchmarking. The endpoint supports tool calling and three reasoning-effort levels: low, high and max (NVIDIA blog). - Hardware per Station: 252 GB HBM3e GPU memory plus 496 GB LPDDR5X CPU memory, and a ConnectX-8 network card rated up to 800 Gb/s (NVIDIA DGX Station).
Technical analysis for researchers and developers
- How the model is split: tensor parallelism across the two machines (TP=2, one GPU per node).
- Throughput features: multi-token prediction drafts 4 tokens per step, which the full model then checks (a form of speculative decoding).
- Stability workarounds: custom all-reduce and FlashInfer all-reduce are disabled.
Read the full section
- How the model is split: tensor parallelism across the two machines (TP=2, one GPU per node). Every token requires the two GPUs to exchange partial results over a single 400 Gb/s RoCE link. The second Station runs a worker only, with no API server. After loading, each GPU holds about 218 GiB of weights and shows about 231 GiB used in total.
- Throughput features: multi-token prediction drafts 4 tokens per step, which the full model then checks (a form of speculative decoding). The KV cache is stored in FP8, prefix caching is on, and CUDA graphs are captured for decode only.
- Stability workarounds: custom all-reduce and FlashInfer all-reduce are disabled. NCCL 2.30.7 is pinned because older versions can deadlock when CUDA graphs are replayed across nodes over RoCE (the post cites vLLM issue #52504) (NVIDIA blog).
- Reproducibility: the post pins the container digest and model revision, hash-checks both copies of the checkpoint, confirms traffic uses GPU RDMA by reading the NCCL logs, and fixes the benchmark seed at 42.
Claims and evidence
All four rows come from NVIDIA's own runs on 80 coding conversations from SPEED-Bench, at max reasoning effort with prefix caching on. The video repeats the peak of about 608 tokens/s (video). On quality, NVIDIA reports NVFP4 roughly matching its FP8 baseline: GPQA Diamond 92.68% vs 91.79%, LiveCodeBench v6 53.55% vs 52.95% (model card).
Read the full section
| Concurrency | Throughput (tok/s) | Mean latency | Inter-token latency |
| 1 | 107.4 | 49.8 s | 8.8 ms |
| 8 | 386.0 | 114.2 s | 19.0 ms |
| 16 | 607.8 | 136.5 s | 23.8 ms |
| 32 | 385.0 | 453.5 s | 77.9 ms |
All four rows come from NVIDIA's own runs on 80 coding conversations from SPEED-Bench, at max reasoning effort with prefix caching on. Token counts include reasoning tokens (NVIDIA blog). The video repeats the peak of about 608 tokens/s (video). On quality, NVIDIA reports NVFP4 roughly matching its FP8 baseline: GPQA Diamond 92.68% vs 91.79%, LiveCodeBench v6 53.55% vs 52.95% (model card). No independent check exists for either the speed or the accuracy figures.
Context and prior work
- Artificial Analysis rates GLM-5.3 (max) well above average among open-weight models. That verbosity helps explain the long request times above.
- NVIDIA's own reference test for the checkpoint used SGLang with TP=8 across two 4-GPU nodes, not vLLM on Stations (model card).
- Owners of two-node DGX Spark setups have reported similar NCCL all-reduce hangs (NVIDIA forums).
Read the full section
- Artificial Analysis rates GLM-5.3 (max) well above average among open-weight models. It also measured about 62 seconds to first token on hosted APIs and above-median output length (Artificial Analysis). That verbosity helps explain the long request times above.
- NVIDIA's own reference test for the checkpoint used SGLang with TP=8 across two 4-GPU nodes, not vLLM on Stations (model card).
- Owners of two-node DGX Spark setups have reported similar NCCL all-reduce hangs (NVIDIA forums).
Limitations, safety and contested findings
- Parameter count disagreement: Wikipedia says GLM-5.3 uses the same 743B base as GLM-5.2 (Z.ai – Wikipedia). NVIDIA and Artificial Analysis say 753B.
- Is the second Station needed? It is not the only way to run the model.
- Security: the API runs without authentication and listens on every interface unless firewalled (NVIDIA blog).
Read the full section
- Parameter count disagreement: Wikipedia says GLM-5.3 uses the same 743B base as GLM-5.2 (Z.ai – Wikipedia). NVIDIA and Artificial Analysis say 753B.
- Is the second Station needed? NVIDIA markets one Station as handling models up to 1T parameters, using its 748 GB of combined CPU and GPU memory (NVIDIA DGX Station). The 464 GB checkpoint fits within 748 GB, so the two-Station setup is a choice to keep all weights in GPU memory with CPU offload disabled. It is not the only way to run the model.
- Security: the API runs without authentication and listens on every interface unless firewalled (NVIDIA blog).
- Video caveats: the video's live run used only a small sample of the benchmark (video). Its automatic captions also contain errors, such as "TP four" where TP1 is meant (video).
Business and practitioner implications
- Where it fits: a frontier-class open-weight coding and agent model can run fully on-premises behind a familiar API.
- Concurrency limit: throughput peaks around 16 concurrent users. Lowering reasoning effort will change the numbers above.
- Cost: check hardware prices yourself. Retail listings put a single GB300 DGX Station near $91,100 (TechRadar).
Read the full section
- Where it fits: a frontier-class open-weight coding and agent model can run fully on-premises behind a familiar API. That matters for data-residency requirements and for the cyber-capability concerns noted above.
- Concurrency limit: throughput peaks around 16 concurrent users. Beyond that, latency becomes unworkable, so this suits a team or a batch pipeline, not a public service. Lowering reasoning effort will change the numbers above.
- Cost: check hardware prices yourself. Retail listings put a single GB300 DGX Station near $91,100 (TechRadar).
- Before relying on it: rerun AIPerf with your own prompts, as NVIDIA itself advises.
The source trail.
Sources (8)
NVIDIA Technical Blog – When One GPU Isn't Enough
developer.nvidia.comLinked video: When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM
linked video auto captions
www.youtube.com