---
type: "raw_signal"
id: "hitting-a-billion-tokens-per-minute-on-one-d93ea7"
slug: "hitting-a-billion-tokens-per-minute-on-one-d93ea7"
title: "Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine"
description: "Modal and Carnegie Mellon’s Full Stack Data Lab released Quail, an open-source query-aware inference layer for AI-SQL workloads."
retrieval_nugget: "Modal and Carnegie Mellon’s Full Stack Data Lab released Quail, an open-source query-aware inference layer for AI-SQL workloads."
observed_at: "2026-09-27"
record_date: "2026-09-27"
date_kind: "observed_at"
why_it_matters: "Source: https://modal.com/blog/quail-billion-tpm"
novelty: "notable"
verification_level: "source-linked"
signal_type: "research"
source_platform: "modal.com"
topics: []
entities: []
related_patterns: []
source_url: "https://modal.com/blog/quail-billion-tpm"
source_urls: ["https://modal.com/blog/quail-billion-tpm"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "signal:hitting-a-billion-tokens-per-minute-on-one-d93ea7"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/signals/hitting-a-billion-tokens-per-minute-on-one-d93ea7/","markdown":"https://newruntime.com/signals/hitting-a-billion-tokens-per-minute-on-one-d93ea7.md","json":"https://newruntime.com/signals/hitting-a-billion-tokens-per-minute-on-one-d93ea7.json"}
---

# Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine

## Retrieval answer

Modal and Carnegie Mellon’s Full Stack Data Lab released Quail, an open-source query-aware inference layer for AI-SQL workloads.

Modal and Carnegie Mellon’s Full Stack Data Lab released Quail, an open-source query-aware inference layer for AI-SQL workloads. Rather than treating every LLM request as an independent, client-driven serving event, Quail uses the SQL query plan to schedule large sets of related prompts. Modal reports more than one billion processed tokens per minute per H100 on a planning-sensitive multi-join query, over 10× its vLLM baseline on the same hardware, and a 1.84× geometric-mean speedup across its newly released AI-SQL benchmark.

The claimed advantage comes from workload structure: the engine can order requests around shared prefixes, manage and evict KV cache entries with known future demand, and target Boolean AI filters and joins where a next-token probability can remove the need for iterative decoding. These are vendor-published results and should be read as evidence for this workload design, not as a general throughput ranking for LLM serving.

Source: https://modal.com/blog/quail-billion-tpm

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Raw Signals](https://newruntime.com/signals.json): Open every source-linked raw signal.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)