---
type: "post"
id: "hitting-a-billion-tokens-per-minute-on-one-f36638"
slug: "hitting-a-billion-tokens-per-minute-on-one-f36638"
title: "Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine"
description: "Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior."
retrieval_nugget: "Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior."
published_at: "2026-09-27"
updated_at: "2026-09-27"
record_date: "2026-09-27"
date_kind: "published_at"
topics: []
source_urls: ["https://modal.com/blog/quail-billion-tpm"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:hitting-a-billion-tokens-per-minute-on-one-f36638"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:f46c6dcd74a7db636f123ea1ffa41d5c0bff7f9b6e76699dd6ba885c58d83933","reviewed_at":"2026-09-27T08:01:00.818Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.","observed_facts":[{"text":"Modal reports that Quail exceeded one billion processed tokens per minute per H100 GPU on one planning-sensitive multi-join query, more than 10× its vLLM baseline on the same hardware.","source_urls":["https://modal.com/blog/quail-billion-tpm"]},{"text":"The authors also report a 1.84× geometric-mean speedup over vLLM on their newly released AI-SQL benchmark.","source_urls":["https://modal.com/blog/quail-billion-tpm"]},{"text":"The reported workload uses AI functions inside SQL to construct prompts from database records and apply LLM-based filters or joins.","source_urls":["https://modal.com/blog/quail-billion-tpm"]}],"mechanism":"The mechanism is that a full SQL plan exposes the future set and structure of requests before execution: Quail can reorder related prompts, reuse shared-prefix KV state, prefetch where useful, and evict cache entries when the plan shows they are no longer needed.","why_now":"Why now is that AI-SQL is emerging across analytical platforms while inference infrastructure remains primarily optimized for chatbots and agents, whose request order and cache reuse are much less predictable.","implications":["The implication is that high-volume LLM transformations over enterprise data may need database-aware execution engines rather than generic serving layers.","Boolean AI filters can use the probability of a single next token, avoiding the iterative decode stage that often constrains GPU utilization in agent-style inference."],"evidence_boundary":"The evidence boundary is that the throughput, cost, and comparative gains are vendor-published results on specified hardware and benchmarks; they do not establish equivalent gains for general agentic, generative, or latency-critical workloads.","watch_conditions":["Watch whether independent users reproduce the reported gains across production datasets, models, and query shapes, and whether AI-SQL interfaces converge enough to make query-aware scheduling a reusable systems layer."],"related_records":[],"new_branch_reason":"No prior New Runtime record with usable editorial detail was supplied; this release starts a distinct record on database-aware LLM inference architecture."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638/","markdown":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638.md","json":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638.json"}
---

# Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine

## Retrieval answer

Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.

Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.

Modal reports that Quail exceeded one billion processed tokens per minute per H100 GPU on one planning-sensitive multi-join query, more than 10× its vLLM baseline on the same hardware. The authors also report a 1.84× geometric-mean speedup over vLLM on their newly released AI-SQL benchmark. The reported workload uses AI functions inside SQL to construct prompts from database records and apply LLM-based filters or joins.

The mechanism is that a full SQL plan exposes the future set and structure of requests before execution: Quail can reorder related prompts, reuse shared-prefix KV state, prefetch where useful, and evict cache entries when the plan shows they are no longer needed. Boolean AI filters can use the probability of a single next token, avoiding the iterative decode stage that often constrains GPU utilization in agent-style inference.

Why now is that AI-SQL is emerging across analytical platforms while inference infrastructure remains primarily optimized for chatbots and agents, whose request order and cache reuse are much less predictable. The implication is that high-volume LLM transformations over enterprise data may need database-aware execution engines rather than generic serving layers. The evidence boundary is that the throughput, cost, and comparative gains are vendor-published results on specified hardware and benchmarks; they do not establish equivalent gains for general agentic, generative, or latency-critical workloads.

Watch whether independent users reproduce the reported gains across production datasets, models, and query shapes, and whether AI-SQL interfaces converge enough to make query-aware scheduling a reusable systems layer.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)