---
type: "post"
id: "11-840-tps-on-a-single-gpu-d7e41f"
slug: "11-840-tps-on-a-single-gpu-d7e41f"
title: "11,840 TPS on a single GPU"
description: "Cline reports reaching 11,840 aggregate output tokens per second on a single NVIDIA GB300 while serving 256 concurrent requests to Nemotron 3.5 Super VL."
retrieval_nugget: "Cline reports reaching 11,840 aggregate output tokens per second on a single NVIDIA GB300 while serving 256 concurrent requests to Nemotron 3.5 Super VL. The useful result is not a universal speed record: it is a concrete reminder that self-hosted agent capacity and cost depend on traffic shape, scheduling, kernel choices, and acceptable quality trade-offs."
published_at: "2026-10-27"
updated_at: "2026-10-27"
record_date: "2026-10-27"
date_kind: "published_at"
topics: ["agent-economics","model-routing"]
entities: ["Cline","Nemotron 3.5 Super VL","NVIDIA","GB300","DGX Station","vLLM","FlashInfer"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Cline now reports 11,840 output tokens/sec on one GB300 with 256 concurrent requests for Nemotron 3.5 Super VL, detailing inference optimizations."
prior_refs: ["memory:fab35c20-3577-863c-a393-602c718d08dd","memory:5c2cf5da-0c2e-8e6e-ac05-8ca4fcb50d82"]
source_url: "https://cline.ghost.io/11-840-tps-on-a-single-gpu"
source_urls: ["https://cline.ghost.io/11-840-tps-on-a-single-gpu"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:11-840-tps-on-a-single-gpu-d7e41f"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/11-840-tps-on-a-single-gpu-d7e41f/","markdown":"https://newruntime.com/posts/11-840-tps-on-a-single-gpu-d7e41f.md","json":"https://newruntime.com/posts/11-840-tps-on-a-single-gpu-d7e41f.json"}
---

# 11,840 TPS on a single GPU

## Retrieval answer

Cline reports reaching 11,840 aggregate output tokens per second on a single NVIDIA GB300 while serving 256 concurrent requests to Nemotron 3.5 Super VL. The useful result is not a universal speed record: it is a concrete reminder that self-hosted agent capacity and cost depend on traffic shape, scheduling, kernel choices, and acceptable quality trade-offs.

## Summary

Cline reports reaching 11,840 aggregate output tokens per second on a single NVIDIA GB300 while serving 256 concurrent requests to Nemotron 3.5 Super VL. The useful result is not a universal speed record: it is a concrete reminder that self-hosted agent capacity and cost depend on traffic shape, scheduling, kernel choices, and acceptable quality trade-offs.

## What happened

Cline published a walkthrough of an inference experiment run with NVIDIA on a DGX Station using a GB300 with a stated 252 GB GPU-memory budget. NVIDIA released Nemotron 3.5 Super VL on October 15; Cline says it used the model because it is designed as a fast, locally hosted model for tasks including document intelligence, coding, and financial accounting. The company’s target was maximum throughput for a single machine under a highly concurrent load, rather than a broad model-quality comparison.

The starting point was vLLM version `0.32.0rc1.dev190+g3f1bcfd82`. Cline says that base configuration consumed 67 GB of memory and produced 6,058 aggregate output tokens per second in a test with 512 input tokens, 512 output tokens, and 256 concurrent streams. Its final result was 11,840 aggregate output tokens per second with 256 concurrent requests. Those numbers describe aggregate server throughput under the specified benchmark shape, not the response speed or quality that one user should expect from a lightly loaded deployment.

The walkthrough identifies several changes used in the climb. First, Cline used the model’s multi-token prediction head, testing MTP3 and reaching roughly 7,000 tokens per second with an OMP1 configuration. It then enabled asynchronous scheduling, arguing that the CPU’s scheduling work otherwise blocks progress while the GPU runs kernels. Switching the model’s Mamba backend kernel to FlashInfer raised throughput to 7,800 tokens per second in this experiment. The company explicitly says it is not clear why that choice worked and that another GPU configuration may behave differently.

Cline then changed the lookahead from MTP3 to MTP1, reporting 9,500 tokens per second. Its explanation is operational: predicted draft tokens still need verification, and with 256 concurrent threads the additional verification overhead was not worthwhile. Finally, it changed Mamba state from FP32 to FP16 with stochastic rounding. Cline says this can reduce precision and potentially evaluation scores, but accepted that trade-off to maximize throughput. It also presents a conditional economic claim: for a fully occupied GPU, its rough comparison found self-hosting about 66% cheaper than inference purchased through a model router or API.

## What the archive adds

This is a new performance and configuration result in a known Cline-and-Nemotron thread. [Our Field Note on Cline’s harness result](https://newruntime.com/posts/cline-agents-harness-unlocks-benchmark-leading-performance-from-03fd56/) covered a different question: reported Terminal-Bench performance for Nemotron 3.5 Super VL across agent harnesses. This post instead concerns serving throughput and the quality-versus-capacity choices behind it. [A prior note on inference planning](https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638/) likewise focused on making workload structure schedulable, though in an AI-SQL setting.

## Trend signal

The result supports [Agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/), but only indirectly. Raw tokens per second are not the economic outcome; they become relevant when they determine how many useful concurrent agent tasks a team can serve at acceptable quality and utilization. Cline’s MTP1 finding and its FP16 caveat underline that a throughput maximum can conflict with verification or evaluation quality. The reported 66% saving is therefore a utilization-dependent infrastructure input, not a unit-cost conclusion on its own.

## Why now

The timing follows NVIDIA’s October 15 release of Nemotron 3.5 Super VL and Cline’s opportunity to test it with NVIDIA on a DGX Station. The company frames the work around a near-term operating premise: small, fast open-weight models may enable dozens of parallel local agent threads, making the deployment configuration itself a consequential part of agent product design.

## Connections to other events

Cline previously reported that the same Nemotron 3.5 Super VL model reached 61.8% on Terminal-Bench 2.1 in its harness with thinking mode off, compared with lower reported scores in three other open-source harnesses. Together, the two reports separate two layers that teams should evaluate independently: the harness and model configuration used to complete agent tasks, and the serving configuration used to supply enough concurrent capacity for those tasks.

## Why it matters

Teams running coding or workflow agents locally often treat model selection, hosting, and scaling as separate decisions. This walkthrough shows why they are coupled. The same model can be economically attractive under a dense, predictable queue and unattractive under sparse or latency-sensitive demand. It also shows that a configuration can improve aggregate throughput while reducing numerical precision or spending less effort on speculative-token verification.

The practical implication is to benchmark against the traffic that actually reaches the endpoint: input and output lengths, concurrency, queue behavior, latency target, and quality evaluation. A full GPU is not automatically cheaper than an API, and a high aggregate token figure does not establish that an agent completes more verified work. Still, teams with sustained parallel workloads now have a more specific test case for comparing a self-hosted open-weight deployment with routed inference.

## What happens next

- **More self-hosting teams will publish workload-specific serving recipes rather than single throughput figures** (6 months, 65%). Watch for: benchmarks that report concurrency, token lengths, quality checks, and GPU utilization together.
- **Cline or comparable operators will test automated configuration search for inference endpoints** (12 months, 55%). Watch for: held-out capacity tests tied to production-metric monitoring and controlled traffic migration.

## Limits and uncertainty

The source is Cline’s own technical account, not an independent benchmark. It does not provide full reproduction commands, latency distributions, token acceptance rates, power consumption, hardware cost, or evaluation results before and after the FP16 change. The 11,840 figure applies to a stated 256-request workload and should not be read as per-request generation speed. Cline itself says some observed gains, including the FlashInfer change, may not transfer to another GPU configuration or load pattern. Its 66% cost comparison is explicitly based on a fully occupied GPU and rough API-versus-hardware arithmetic.

## Skill to build now

Create a small serving benchmark that mirrors one production agent queue: fix representative input and output lengths, sweep concurrency, and record aggregate throughput alongside p50 and p95 latency, GPU memory, utilization, and a task-quality check. Change one setting at a time—scheduler mode, speculative decoding lookahead, kernel, or precision—and retain the result only if it improves the metric your team actually needs. Compare the measured cost at observed utilization with the API route, rather than comparing list prices or peak TPS.

## Reader takeaway

Cline’s result is evidence that one modern GPU can supply substantial aggregate capacity for parallel local agents, not evidence that 11,840 tokens per second is a portable deployment promise. The winning inference configuration is a property of the workload. Measure capacity, quality, and utilization together before turning a benchmark into an architecture decision.

## Open question

How much quality, latency, and energy cost changes when this configuration is evaluated on real agent tasks rather than a fixed 512-token input/output load?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Economics](https://newruntime.com/topics/agent-economics.md): Explore the agent-economics topic hub.
- topic hub: [Model Routing](https://newruntime.com/topics/model-routing.md): Explore the model-routing topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)