Rather than treating every LLM request as an independent, client-driven serving event, Quail uses the SQL query plan to schedule large sets of related prompts. Modal reports more than one billion processed tokens per minute per H100 on a planning-sensitive multi-join query, over 10× its vLLM baseline on the same hardware, and a 1.84× geometric-mean speedup across its newly released AI-SQL benchmark.
The claimed advantage comes from workload structure: the engine can order requests around shared prefixes, manage and evict KV cache entries with known future demand, and target Boolean AI filters and joins where a next-token probability can remove the need for iterative decoding. These are vendor-published results and should be read as evidence for this workload design, not as a general throughput ranking for LLM serving.