---
type: "post"
id: "overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc"
slug: "overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc"
title: "Overhauling DoorDash’s Runtime Configuration Value Distribution with SquashFS Engineering Experimentation and Testing"
description: "DoorDash replaced eager extraction of runtime-configuration archives on Kubernetes nodes with mounted SquashFS images."
retrieval_nugget: "DoorDash replaced eager extraction of runtime-configuration archives on Kubernetes nodes with mounted SquashFS images. The company says the change cut a major distribution bottleneck by more than 100 times and reduced average EBS IOPS per node by more than five times, while preserving file-based reads for services."
published_at: "2026-10-31"
updated_at: "2026-10-31"
record_date: "2026-10-31"
date_kind: "published_at"
topics: ["api-design"]
entities: ["DoorDash","SquashFS","Kubernetes","Amazon S3","Amazon EBS","Amazon EC2"]
related_patterns: ["agent-ready-software-exposes-capabilities"]
novelty_label: "new_fact"
what_changed: "A distinct DoorDash infrastructure case study details SquashFS-based runtime config distribution, cutting a bottleneck’s latency by over 100-fold and average node EBS IOPS by more than fivefold."
prior_refs: ["memory:c186961c-23ab-8239-ad73-53af3f40671d","memory:7e7967a1-df34-8967-a5a9-6ca20b1fc93d"]
source_url: "https://careersatdoordash.com/blog/overhauling-doordashs-runtime-configuration-value-distribution-with-squashfs"
source_urls: ["https://careersatdoordash.com/blog/overhauling-doordashs-runtime-configuration-value-distribution-with-squashfs"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc/","markdown":"https://newruntime.com/posts/overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc.md","json":"https://newruntime.com/posts/overhauling-doordash-s-runtime-configuration-value-distribution-with-780ebc.json"}
---

# Overhauling DoorDash’s Runtime Configuration Value Distribution with SquashFS Engineering Experimentation and Testing

## Retrieval answer

DoorDash replaced eager extraction of runtime-configuration archives on Kubernetes nodes with mounted SquashFS images. The company says the change cut a major distribution bottleneck by more than 100 times and reduced average EBS IOPS per node by more than five times, while preserving file-based reads for services.

## Summary

DoorDash replaced eager extraction of runtime-configuration archives on Kubernetes nodes with mounted SquashFS images. The company says the change cut a major distribution bottleneck by more than 100 times and reduced average EBS IOPS per node by more than five times, while preserving file-based reads for services.

## What happened

DoorDash published an engineering account of changing how runtime configuration values reach services. The values support feature flags, gradual rollouts, experiments, and operational controls without requiring a deployment or service restart. Kubernetes is the company’s primary service-orchestration platform, and services consume configuration as files under `/srv/config-values`. That file interface is important: from an application’s point of view, reading a current value is a Linux filesystem read, accelerated by the node-local page cache and a pod-local cache.

The previous distribution path used a Kubernetes DaemonSet on every node. It polled Amazon S3 for a versioned `.tar.gz` archive, extracted that archive to a hostPath, and switched a symlink to the latest completed directory. HostPath-backed volumes then exposed the values to pods. The design had useful failure behavior. A download first landed in a temporary location and was committed only after completion, so a partial transfer could not replace a good dataset. If the DaemonSet could not obtain a new version, the symlink continued to point to the prior version; the worst case was stale values, not missing ones. A garbage collector removed older versions but not the active dataset.

That architecture assumed it was acceptable for every node to receive every value. It made scheduling simple because any pod could run on any node without considering configuration availability, but most pods read only a small relevant subset. As configuration use spread, the archive grew from hundreds to tens of thousands of distinct values. Extracting all of those files on every update overloaded the underlying Amazon EC2 Elastic Block Store volume. DoorDash first raised the S3-download-and-extraction timeout from 15 seconds to 30 seconds, then found the new limit was also regularly reached within weeks. Slow propagation of fresh configuration was a concern because it could delay incident mitigation.

The team evaluated approaches that could avoid eagerly materializing the entire dataset while retaining near-native file-read speed, virtually zero downtime, and existing access patterns. It selected SquashFS. The distribution service uploads a `.squashfs` image to S3 alongside the existing archive. The DaemonSet still polls S3, but it mounts the image as a loop device rather than extracting it. Linux then treats the image as a read-only filesystem. DoorDash prioritized read latency over image size, benchmarked its real configuration values, and selected LZ4 compression with 128 KB blocks. It reports that raw EXT4 reads were only a few microseconds faster in first-byte-latency tests, alongside more than a 100-fold reduction in the bottleneck latency and more than a fivefold drop in average node EBS IOPS.

## What the archive adds

This is a new fact on a known DoorDash infrastructure story. The earlier [dispatch-platform Field Note](https://newruntime.com/posts/how-doordash-built-a-next-generation-real-time-eb6790/) focused on separating orchestration from computation in a time-critical system; this account concerns the lower-level distribution of runtime controls. A related [dispatch objective-weighting note](https://newruntime.com/posts/aligning-dispatch-to-reality-using-reinforcement-learning-to-d0b2f6/) shows those controls operating in a separate production optimization context.

## Trend signal

The case supports [agent-ready software exposes capabilities](https://newruntime.com/patterns/agent-ready-software-exposes-capabilities/), although it is not itself an agent product. A stable, permissioned file interface insulated services from a change in how configuration was packaged and delivered. The useful boundary was the observable capability—read the current configuration value—not the storage or transfer mechanism behind it.

## Why it matters

Teams building AI systems often treat configuration delivery as incidental infrastructure, until feature gates, experiment parameters, routing controls, and operational switches become numerous enough to be on the incident path. DoorDash’s account gives a concrete design test: distinguish the cost of publishing an entire configuration universe from the cost of serving the small fraction each workload actually reads.

The result also illustrates an operationally conservative migration. The service contract stayed file-based, the prior archive continued to exist alongside the new image, and the older last-known-good behavior remained the reference for failure handling. That is more reusable than adopting SquashFS by name. A team can change a distribution implementation without forcing every consumer to change its configuration client or weakening its rollback properties.

## What happens next

- **Large configuration platforms will increasingly benchmark image mounting or on-demand access against archive extraction.** (12 months, 65%). Watch for: published migration designs that retain existing filesystem interfaces.
- **Configuration systems will expose storage-efficiency metrics alongside propagation latency.** (6 months, 55%). Watch for: reports separating update work from the subset of values actually read.

## Limits and uncertainty

This is a DoorDash engineering account, not an independent evaluation or a general performance benchmark. The reported reductions apply to one unspecified major bottleneck and the company’s own fleet, data distribution, storage volumes, and read patterns. The supplied material does not give absolute latency, image size, rollout duration, fleet size, cost, failure-rate comparisons, or a comparison with remote configuration reads and other alternatives. It also does not establish that SquashFS will outperform extraction for smaller datasets, frequently changing datasets, or workloads that read most files.

## Skill to build now

Inventory one configuration path this week. Measure the number of files or keys published in an update, the number actually read by a representative workload, update propagation time, and storage IOPS during rollout. Then document the failure contract: what consumers see after a partial download, a failed refresh, and cleanup of older versions. That evidence will tell you whether packaging is the bottleneck before you change formats.

## Reader takeaway

DoorDash’s key move was not merely compressing configuration better. It stopped doing work for every file before any consumer needed it, while keeping a fast local read interface and a last-known-good fallback. For a growing control plane, preserve the consumer contract first, then measure whether eager materialization is the hidden scaling cost.

## Open question

How much of the reported gain came from avoiding file creation and metadata operations, versus the different transfer and caching behavior of a single SquashFS image?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Api Design](https://newruntime.com/topics/api-design.md): Explore the api-design topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)