NVIDIA positions AIPerf as GenAI-Perf successor for high-concurrency LLM inference benchmarking

Access. AIPerf is available as a Python package named aiperf; PyPI lists version 0.12.0, released August 6, 2026, with Python requirement >=3.11,<3.14, author NVIDIA Inc., and license Apache-2.0. The article also shows installation through uv tool install aiperf or uv pip install aiperf. aiperf · PyPI

Endpoint and workload scope. NVIDIA says AIPerf supports 15+ endpoint types, including chat, responses, NIM rankings, image generation, and more, plus public datasets such as ShareGPT and trace replay formats from Mooncake, Baseten, WEKA AgentX, and others. This is vendor-reported; no independent test matrix confirming all endpoint types were found in the reviewed sources. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

Traffic shaping. The blog says AIPerf supports constant, Poisson, and gamma arrival patterns; tunable burstiness; gradual ramping for concurrency and request rate; and synthetic distributions including vLLM/SGLang range-ratio style input/output sequence length variation. Similar load-shaping ideas are already present in vLLM’s own benchmark tooling, whose documentation describes finite request-rate tests, Poisson traffic at burstiness 1.0, gamma-distributed burstiness controls, and max-concurrency controls. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

Telemetry and exports. AIPerf documentation says it can collect server metrics from Prometheus-compatible endpoints and GPU telemetry through DCGM, pynvml, or AMD SMI depending on setup. GPU telemetry can be shown in console/dashboard modes and exported to files; server metrics collection defaults to a scrape interval of 333 ms according to the docs. Server Metrics Collection | NVIDIA AIPerf Documentation

Architecture

The main technical claim is that AIPerf avoids the single-process bottlenecks that NVIDIA says affected prior benchmark clients, including GenAI-Perf. The article describes a multiprocessed system: worker processes generate load, record-processor services handle results, and coordination occurs over ZMQ. A project-internal GitHub file also describes AIPerf as a Python 3.11+ async benchmarking tool in which multiple services communicate over a ZMQ message bus. Treat this as project evidence, not an external performance proof. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

For developers, this architecture matters because a load generator that cannot saturate the server produces misleading server-side conclusions. If Python GIL contention, event-loop saturation, tokenizer overhead, result aggregation, or client networking becomes the bottleneck, the benchmark measures the client. AIPerf’s design appears aimed at separating request issuance, record handling, and metric computation to reduce that risk. However, without independent stress tests showing client overhead versus server throughput at different concurrency levels, the degree of improvement remains unverified.

Evaluation methodology

The article’s “maiden benchmark” pins input and output lengths to 128 tokens each and uses min_tokens:128 plus ignore_eos:true to force the model to emit the intended number of tokens when the server supports those parameters. This is important: otherwise, output token length is a target, not a guarantee, and throughput comparisons can be invalid if one run produces fewer tokens. PyPI’s known-issues section makes the same caveat: output sequence length constraints cannot be guaranteed unless compatible server-side controls such as ignore_eos or min_tokens are used. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

The article also emphasizes streaming. Without streaming, a server can return a completed response as one chunk, which prevents accurate measurement of time to first token and inter-token latency from client-observed events. This is consistent with broader LLM benchmark practice: LLMPerf measures inter-token latency and generation throughput under concurrent requests, and LLM-Inference-Bench defines TTFT and ITL as core inference metrics. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical Blog

Metrics and reproducibility

AIPerf’s docs divide metrics into record metrics, aggregate metrics, and derived metrics. Record metrics are computed per request or response stream; examples include request latency, time to first token, inter-token latency, output token count, and input sequence length. Request latency is defined as the end-to-end time from request send to final response, including network time, queuing, prompt processing, token generation, and response transmission. Metrics Reference | NVIDIA AIPerf Documentation

That definition is useful but also constraining: AIPerf’s client-observed latency is exactly what an application user may experience, but it does not by itself isolate server-internal prefill, decode, scheduling, KV-cache, or network effects unless combined with server metrics and traces. AIPerf’s Prometheus scraping and GPU telemetry help close that gap, but practitioners still need synchronized clocks, stable network paths, fixed model/server versions, fixed tokenizer behavior, explicit warmup policy, and repeat runs.

Implementation implications

For developers maintaining benchmarking pipelines, AIPerf’s most practical value is a common CLI and output schema across workload types. NVIDIA’s migration guide says many GenAI-Perf options map directly to AIPerf, but it also warns that some metrics change semantics for reasoning-capable models. For example, it says GenAI-Perf TTFT measures time to the first non-reasoning output token, while AIPerf distinguishes TTFT and TTFO to expose more of the token-generation timeline. Migrating from GenAI-Perf | NVIDIA AIPerf Documentation

This matters for longitudinal dashboards: teams migrating from GenAI-Perf should not blindly plot old and new TTFT/OSL fields on the same axis without checking definitions. A benchmark migration can create artificial regressions or improvements if “first token,” “first output token,” “reasoning token,” and “output sequence length” are not aligned.

AIPerf fits into a maturing LLM-serving benchmark ecosystem. LLMPerf from the Ray project is an earlier open benchmark library for LLM APIs that measures inter-token latency and generation throughput under concurrent requests, while explicitly warning that provider backend variation, time of day, load, and workload mismatch can affect results. GitHub - ray-project/llmperf: LLMPerf is a library for validating and benchmarking LLMs · GitHub

vLLM helped popularize serving-oriented LLM benchmarking using ShareGPT- and Alpaca-derived workloads and Poisson arrivals. The vLLM paper notes that ShareGPT and Alpaca lack timestamps, so arrivals are synthesized using Poisson distributions at different request rates; it also emphasizes that latency can “explode” once request rate exceeds serving capacity. Efficient Memory Management for Large Language Model Serving with PagedAttention

MLPerf Inference offers a more standardized industry benchmark track. MLCommons’ September 2026 discussion of MLPerf Inference v6.1 lists Datacenter and Edge categories, scenarios such as Offline, Server, Interactive, SingleStream, and MultiStream, and Closed/Open divisions. AIPerf is not a replacement for MLPerf; rather, it appears aimed at practitioner-controlled endpoint benchmarking and production workload replay, whereas MLPerf is designed for comparable submissions under defined benchmark rules. Where the Industry Is Investing: A Look at MLPerf Inference v6.1 - MLCommons

The largest limitation is evidentiary: no independent coverage of the specific September 18 AIPerf story was found. NVIDIA’s claims are coherent and technically plausible, but independent practitioners should not treat them as externally verified performance facts.

Known operational caveats include output-length control, high-concurrency port exhaustion, startup hangs under invalid configuration, and dashboard text-copy issues listed on PyPI. The article also notes an aarch64 installation issue: the crick dependency may require a C toolchain because it ships source-only on that platform. aiperf · PyPI

There are also benchmark-safety issues: load tests can overload shared inference services, incur cloud cost, trigger provider rate limits, or produce misleading capacity plans if run from an unrepresentative network location. For regulated or sensitive deployments, trace replay must be checked for prompt/response data leakage before traces are moved into benchmark artifacts.

For infrastructure leaders, AIPerf’s main value proposition is repeatable capacity planning: test SLOs under realistic request arrival patterns, monitor p95/p99 tail behavior, and correlate latency with GPU and server metrics. That is more useful for procurement and autoscaling than isolated tokens-per-second numbers.

For developers, the migration risk is metric semantics. A GenAI-Perf-to-AIPerf migration should include a side-by-side baseline, pinned package versions, fixed model/server/container versions, fixed tokenizers, and documented changes to TTFT, TTFO, OSL, reasoning-token, and output-token definitions.

For researchers, AIPerf may become a useful harness if its workload, telemetry, and export formats are transparent enough for reproduction. But vendor-maintained tooling should be paired with independent scripts or MLPerf-style methods when publishing comparative claims.