{"type":"post","id":"hitting-a-billion-tokens-per-minute-on-one-f36638","slug":"hitting-a-billion-tokens-per-minute-on-one-f36638","title":"Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine","description":"Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.","retrieval_nugget":"Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.","published_at":"2026-09-27","updated_at":"2026-09-27","record_date":"2026-09-27","date_kind":"published_at","topics":[],"source_urls":["https://modal.com/blog/quail-billion-tpm"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:hitting-a-billion-tokens-per-minute-on-one-f36638","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:f46c6dcd74a7db636f123ea1ffa41d5c0bff7f9b6e76699dd6ba885c58d83933","reviewed_at":"2026-09-27T08:01:00.818Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Quail makes the query plan part of the inference system, turning AI-SQL from a stream of arbitrary LLM requests into a schedulable batch workload with unusually controllable cache behavior.","observed_facts":[{"text":"Modal reports that Quail exceeded one billion processed tokens per minute per H100 GPU on one planning-sensitive multi-join query, more than 10× its vLLM baseline on the same hardware.","source_urls":["https://modal.com/blog/quail-billion-tpm"]},{"text":"The authors also report a 1.84× geometric-mean speedup over vLLM on their newly released AI-SQL benchmark.","source_urls":["https://modal.com/blog/quail-billion-tpm"]},{"text":"The reported workload uses AI functions inside SQL to construct prompts from database records and apply LLM-based filters or joins.","source_urls":["https://modal.com/blog/quail-billion-tpm"]}],"mechanism":"The mechanism is that a full SQL plan exposes the future set and structure of requests before execution: Quail can reorder related prompts, reuse shared-prefix KV state, prefetch where useful, and evict cache entries when the plan shows they are no longer needed.","why_now":"Why now is that AI-SQL is emerging across analytical platforms while inference infrastructure remains primarily optimized for chatbots and agents, whose request order and cache reuse are much less predictable.","implications":["The implication is that high-volume LLM transformations over enterprise data may need database-aware execution engines rather than generic serving layers.","Boolean AI filters can use the probability of a single next token, avoiding the iterative decode stage that often constrains GPU utilization in agent-style inference."],"evidence_boundary":"The evidence boundary is that the throughput, cost, and comparative gains are vendor-published results on specified hardware and benchmarks; they do not establish equivalent gains for general agentic, generative, or latency-critical workloads.","watch_conditions":["Watch whether independent users reproduce the reported gains across production datasets, models, and query shapes, and whether AI-SQL interfaces converge enough to make query-aware scheduling a reusable systems layer."],"related_records":[],"new_branch_reason":"No prior New Runtime record with usable editorial detail was supplied; this release starts a distinct record on database-aware LLM inference architecture."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638/","markdown":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638.md","json":"https://newruntime.com/posts/hitting-a-billion-tokens-per-minute-on-one-f36638.json"},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}