Field note
Gimlet's funding story is really a claim that inference has become a heterogeneous scheduling problem. The a16z announcement says Gimlet builds an execution plan that balances latency, throughput, and cost for each workload. It says the system can separate prefill from decode, target different processors through a compiler, and coordinate execution through a runtime behind one inference API.
The mechanism is workload decomposition: models, tools, stages, and operations can be matched to GPUs, CPUs, or purpose-built accelerators instead of forcing every step onto one architecture. This matters now because power, memory, data-center capacity, and accelerator availability are becoming binding constraints while inference workloads diversify.
If the approach works, the orchestration layer becomes as strategically important as the chosen chip because it decides where each unit of work runs. The evidence boundary is an investor announcement: claims about up to tenfold gains and large customers require independent workload-level benchmarks before generalization. Watch for reproducible results across named models, hardware pools, latency objectives, and failure modes when work crosses processor and facility boundaries.