Modal reports that Quail exceeded one billion processed tokens per minute per H100 GPU on one planning-sensitive multi-join query, more than 10× its vLLM baseline on the same hardware. The authors also report a 1.84× geometric-mean speedup over vLLM on their newly released AI-SQL benchmark. The reported workload uses AI functions inside SQL to construct prompts from database records and apply LLM-based filters or joins.
The mechanism is that a full SQL plan exposes the future set and structure of requests before execution: Quail can reorder related prompts, reuse shared-prefix KV state, prefetch where useful, and evict cache entries when the plan shows they are no longer needed. Boolean AI filters can use the probability of a single next token, avoiding the iterative decode stage that often constrains GPU utilization in agent-style inference.
Why now is that AI-SQL is emerging across analytical platforms while inference infrastructure remains primarily optimized for chatbots and agents, whose request order and cache reuse are much less predictable. The implication is that high-volume LLM transformations over enterprise data may need database-aware execution engines rather than generic serving layers. The evidence boundary is that the throughput, cost, and comparative gains are vendor-published results on specified hardware and benchmarks; they do not establish equivalent gains for general agentic, generative, or latency-critical workloads.
Watch whether independent users reproduce the reported gains across production datasets, models, and query shapes, and whether AI-SQL interfaces converge enough to make query-aware scheduling a reusable systems layer.