Field note
The Fast Gemma work around FINAL-Bench uses a stronger definition of optimization than tokens per second. A speedup is valid only if the resulting server still satisfies a quality contract on a private prompt set. Changes that move perplexity are discarded even when they improve throughput.
That design prevents benchmark-specific shortcuts from becoming production regressions. Kernel changes, batching, cache behavior, quantization, and scheduling can be evaluated as candidate transformations; each candidate must pass both the performance measurement and an independent behavioral check.
The same pattern applies outside model serving. Any agent optimization should carry a private regression set and an outcome threshold. Cost, latency, and token reductions are secondary objectives constrained by task quality. If the check changes with the implementation, the proof has been weakened at exactly the moment it is needed.