---
type: "post"
id: "day-0-support-for-flint-on-vllm-2c6d85"
slug: "day-0-support-for-flint-on-vllm-2c6d85"
title: "Day-0 Support for Flint on vLLM"
description: "vLLM says its Flint-specific serving support is available from day one, covering Recursive’s NVFP4 and BF16 checkpoints."
retrieval_nugget: "vLLM says its Flint-specific serving support is available from day one, covering Recursive’s NVFP4 and BF16 checkpoints. On one NVIDIA GB200, the project reports roughly 485 output tokens per second at batch size one and a 100,000-token context, making this a concrete inference result for teams testing Flint on Blackwell hardware."
published_at: "2026-11-12"
updated_at: "2026-11-12"
record_date: "2026-11-12"
date_kind: "published_at"
topics: ["inference","ai-infrastructure","models"]
entities: ["vLLM","Flint","Recursive","NVIDIA GB200"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "The announcement reports about 500 tokens per second on one GB200, including with 100K tokens of context."
prior_refs: []
source_url: "https://vllm-project.github.io/2026/10/21/flint.html"
source_urls: ["https://vllm-project.github.io/2026/10/21/flint.html"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:day-0-support-for-flint-on-vllm-2c6d85"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/day-0-support-for-flint-on-vllm-2c6d85/","markdown":"https://newruntime.com/posts/day-0-support-for-flint-on-vllm-2c6d85.md","json":"https://newruntime.com/posts/day-0-support-for-flint-on-vllm-2c6d85.json"}
markdown_url: "https://newruntime.com/posts/day-0-support-for-flint-on-vllm-2c6d85.md"
---

# Day-0 Support for Flint on vLLM

## Retrieval answer

vLLM says its Flint-specific serving support is available from day one, covering Recursive’s NVFP4 and BF16 checkpoints. On one NVIDIA GB200, the project reports roughly 485 output tokens per second at batch size one and a 100,000-token context, making this a concrete inference result for teams testing Flint on Blackwell hardware.

## Summary

vLLM says its Flint-specific serving support is available from day one, covering Recursive’s NVFP4 and BF16 checkpoints. On one NVIDIA GB200, the project reports roughly 485 output tokens per second at batch size one and a 100,000-token context, making this a concrete inference result for teams testing Flint on Blackwell hardware.

## What happened

In an October 21, 2026 announcement, vLLM said its `vllm/vllm-openai:flint` Docker image supports Recursive’s Flint, with x86_64 and arm64 builds using CUDA 13.0. The image supports two separate checkpoints: an NVFP4 GGUF checkpoint and a BF16 checkpoint. For the NVFP4 version, the launch instructions point configuration and tokenizer loading to the BF16 repository. The announcement also gives equivalent `vllm serve` commands for a vLLM build that includes the support; it says the change is being merged as pull request #62938.

Flint is described as a 49-layer, 40-billion-parameter mixture-of-experts model, activating about 2.5 billion parameters per token, with a 262,000-token context. Thirty-five layers use sliding-window attention over 513 tokens, while 14 attend to the full context. The full-context layers alone retain key/value cache for the sequence, at a stated 28 KiB per token. The model also has 256 experts with sigmoid top-6 routing and a shared expert; its first two layers use dense MLPs.

The NVFP4 checkpoint uses a GGUF file with NVIDIA NVFP4 blocks in a different byte order. vLLM says it repacks the weights losslessly when loading and runs them weight-only, with BF16 activations and KV cache. The project reports that its implementation matches Recursive’s reference implementation on GSM8K, GPQA-Diamond, long-context MRCR and a tool-calling suite.

The performance work targets small batches. vLLM says its initial implementation used stock layers and ran 1,096 kernels per forward, many tuned for larger batches. It replaced those with Flint-specific kernels, including fused normalization, rotary-position and cache operations; small-batch MoE and dense GEMV kernels; and a low-latency decode path for up to eight requests. On one GB200, batch-one NVFP4 decode improved from 4.81 ms to 1.78 ms in the reported model-forward comparison; BF16 decode went from 3.53 ms to 2.34 ms. An 8,192-token prefill fell from 207 ms to 145 ms. End-to-end batch-one decode at 100,000-token context, including the model forward, language-model head and sampling, is reported at 2.06 ms per token, or about 485 tokens per second.

## What the archive adds

New Runtime’s earlier coverage on October 22 described Flint’s release and local coding capabilities ([Flint’s model release](https://newruntime.com/posts/recursive-released-flint-a-40b-total-2-5b-37dd1a/)). This is a new serving event for the same model: vLLM adds implementation details and performance measurements, rather than announcing a new Flint version.

## Trend signal

This supports the pattern that [harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/): a model’s practical performance depends on the serving stack and its model-specific execution paths, not only on the checkpoint. The announcement is evidence of an optimized implementation, not a comparison showing that vLLM’s approach generalizes to other models or runtimes.

## Why now

The release is timed to Flint’s arrival: vLLM calls the support day-zero and provides a dedicated image alongside the model’s two checkpoints. That can shorten the gap between a model release and teams being able to test it through a serving stack they may already use.

## Why it matters

For teams considering Flint, the announcement makes the deployment path and the hardware boundary clearer. Both checkpoints can be served with vLLM’s Flint-specific image, but the specialized kernels target a single Blackwell GPU with tensor parallelism set to one and BF16 KV cache. Other configurations fall back to standard vLLM layers, so the reported speedups should not be treated as a general performance guarantee.

The figures are most useful as a test baseline: they give operators a reference for batch-one decode at long context, as well as separate model-forward measurements for NVFP4, BF16 and prefill. The implementation details also show where serving work went—small-batch expert dispatch and decode overhead, rather than a change to the model itself. A team can therefore decide whether to reproduce the setup, measure its own request mix, and weigh throughput against its own latency and hardware requirements.

## Limits and uncertainty

The performance figures are vLLM’s own measurements on one GB200, not an independent benchmark or a comparison across serving systems. The announcement does not establish results for multi-GPU serving, other accelerators, larger batches, or production traffic. It says configurations outside the single-Blackwell target use standard layers, but does not quantify their performance. The reported accuracy match is limited to the named evaluation sets and tool-calling suite; it does not establish equivalent behavior across all tasks or prompts.

## Skill to build now

If your team has access to a Blackwell GPU and is evaluating Flint, run both the NVFP4 and BF16 checkpoints through the documented vLLM path against representative prompts. Record decode latency at the context lengths and batch sizes you actually expect, and separately measure prefill; the announcement’s 100,000-token, batch-one result is a useful comparison point, not a substitute for workload-specific testing.

## Reader takeaway

vLLM has made Flint immediately serveable with a dedicated path and reports strong batch-one speed on a single GB200, including at 100,000 tokens of context. Treat the numbers as a hardware-specific baseline: the practical result for your team depends on the checkpoint, request pattern and whether your deployment matches the optimized configuration.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- topic hub: [Ai Infrastructure](https://newruntime.com/topics/ai-infrastructure.md): Explore the ai-infrastructure topic hub.
- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)