What happened
In a November 5, 2026 engineering post, Uber describes its continuing effort to reduce the cost of a large Apache Spark fleet. It says data-lake ingestion consumes more than 200,000 CPU cores, with additional compute used by ETL. Across its broader Spark infrastructure, Uber reports more than 100 PB of data read and written each day, over 80 PB of daily shuffle data, more than 59,000 workload fingerprints and over 500,000 applications daily spanning more than a million compute cores. Drogon, its Spark-as-a-Service platform, controls application submission, configuration and rollout, making fleet-wide changes possible.
The post covers two distinct optimization efforts. First, Uber adopted Apache Gluten and Velox to offload Spark SQL operators from JVM execution to a native vectorized engine. For a 1 TB TPC-DS benchmark, it reports an average improvement of about 3.4× across tested scenarios, with a best case of 4× and a worst case of 2.7×. The post also describes selected production workloads where Gluten/Velox delivered runtime improvements of up to 79% and cost reductions of up to 98%. Uber says it contributed more than 30 patches and over 50 commits upstream, including fallback mechanisms, encryption support for native Parquet readers and writers, and additional operators and functions.
The new evaluation concerns Onehouse Quanton, which Uber calls a commercial Spark engine. Alongside native vectorized SQL execution, Quanton includes query-planning and storage-aware optimizations for reading, writing and managing Apache Hudi and Apache Iceberg tables. Its stated mechanisms include native readers for table-format features, pipelined input/output, planner changes to reduce repeated scans, and optimizations to writes, merges, indexing and table maintenance. Quanton is designed to run existing Scala, PySpark and SQL applications without application-code or SQL changes. Uber’s evaluation focused on the write path; it did not benchmark Quanton’s SQL or read-side acceleration.
The benchmark used synthetic data intended to reflect production table scale, schema shape and workload patterns. Uber tested an insert-heavy fact table with 60 billion records on 1,080 vCPUs and an update-heavy dimension table with 10.16 billion records on 3,300 vCPUs. Both sides used Spark 3.5.9 and Hudi 1.1.1, with identical table properties within each comparison, on an isolated HDFS cluster and on-premises Kubernetes. The reported median steady-state write-round speedups were 5.5× for the fact table and 4.7× for the dimension table.
Latency was measured from the start of one write round to the start of the next, reconstructed from Hudi commit instants and cross-checked against Spark History Server. Metadata-table compaction occurred roughly every six to seven commits. Uber excluded compaction-affected rounds from its steady-state figures; it reports one such fact-table round took 60.42 minutes on OSS Spark and 21.27 minutes on Quanton, a 2.8× difference. The post also describes scaling tests on the fact-table workload at 141, 564 and 1,128 vCPUs, with two measured commits per configuration. The supplied account does not state the scaling results.
What the archive adds
The archive’s earlier notes on speculative decoding energy accounting and durable harnesses share broad infrastructure themes, but do not cover this Spark benchmark. This is a new event in that sense, not a new fact on a known story or a restatement.
Trend signal
No supplied pattern directly addresses storage-aware Spark execution or this benchmark. The figures therefore do not materially support or weaken a listed SITE_PATTERNS thesis; they are best treated as a bounded infrastructure result rather than evidence of a broader agent trend.
Why it matters
For teams operating large data platforms, the benchmark points to a potentially important distinction: accelerating SQL operators is not necessarily enough when ingestion and table maintenance dominate compute. A storage-aware engine may improve the write path by reducing work in reading, merging, writing and maintaining tables, rather than merely executing the same operators faster. That makes the workload and table format central to evaluating an engine.
The practical implication is not to infer a 5× improvement for an arbitrary Spark estate. Uber used synthetic workloads, specific Hudi versions, large clusters and steady-state rounds that excluded metadata compaction. Teams should identify where their Spark compute goes, reproduce their own table and write patterns, include maintenance costs in total-run accounting, and compare equivalent resource allocations. Uber’s separate Gluten/Velox production results show that native execution can also help selected SQL workloads, but they are not a direct comparison with Quanton.
Limits and uncertainty
These are vendor-evaluation results reported by Uber, not an independently replicated benchmark. The data was synthetic, and the test environment was isolated from production; the post says it was designed to resemble production patterns but does not establish that it captures every production condition. Quanton’s read-side and SQL acceleration were not benchmarked in this evaluation. Steady-state figures exclude periodic metadata-compaction rounds, which can materially change individual round latency. The supplied account also omits the outcomes of the scaling measurements, and does not provide enough detail here to calculate total workload cost, licensing cost, or a fleet-wide savings estimate. The reported Gluten/Velox production gains concern different workloads and should not be attributed to Quanton.
Skill to build now
Before evaluating an alternative Spark engine, build a workload inventory that separates SQL-heavy processing from ingestion, updates, merges and table maintenance. Select representative tables for each major write pattern, record table format and maintenance cadence, and compare engines with the same data, configuration and resource allocation. Track both steady-state and maintenance-affected rounds so a faster commit is not mistaken for lower end-to-end cost.
Reader takeaway
Uber’s Quanton result is a substantial but narrowly scoped write-path benchmark: 5.5× median speedup on an insert-heavy Hudi table and 4.7× on an update-heavy one, against OSS Spark/Hudi with matched resources. The useful lesson is to benchmark the data and maintenance work that actually consumes your compute—not to assume the headline multiplier transfers to every Spark job.