{"type":"post","id":"taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4","slug":"taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4","title":"Taming the ML Firehose: Scaling Feature Consistency","description":"Uber’s design makes the online inference record, rather than a reconstructed offline approximation, the basis for the next training set.","retrieval_nugget":"Uber’s design makes the online inference record, rather than a reconstructed offline approximation, the basis for the next training set.","published_at":"2026-09-23","updated_at":"2026-09-23","record_date":"2026-09-23","date_kind":"published_at","topics":[],"source_urls":["https://www.uber.com/us/en/blog/taming-ml-firehose"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:40db74c2abee06acb7d6792172f03a4cc26a62ae0e5f00f4cb24b18277c4590d","reviewed_at":"2026-09-23T08:00:54.992Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":2,"watch_condition_count":2,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Uber’s design makes the online inference record, rather than a reconstructed offline approximation, the basis for the next training set.","observed_facts":[{"text":"Uber reports that its recommendation systems can reach 8 million QPS and estimates that logging all predictions could require about 1.7 PB per day.","source_urls":["https://www.uber.com/us/en/blog/taming-ml-firehose"]},{"text":"Uber says a feature allow list reduces logging payload size by 4–5×, while Flink joins prediction features with client impression events.","source_urls":["https://www.uber.com/us/en/blog/taming-ml-firehose"]}],"mechanism":"The mechanism is selective capture: allow-listed features are serialized with compact integer aliases, then prediction records are joined to client impression events in Flink.","why_now":"Why now: transformer-backed recommendation traffic makes a canonical record desirable but economically infeasible without aggressive filtering and stream-state discipline.","implications":["The immediate implication is that feature consistency becomes an operational data-contract problem: retain the serving values and the outcome together, while avoiding the cost of retaining every scored candidate.","Selective logging shifts optimization from storing all model activity to preserving the subset tied to observed user exposure and labels."],"evidence_boundary":"The evidence boundary is one company’s engineering description; it reports architecture and internal estimates, not an independently audited model-lift result.","watch_conditions":["Watch whether Uber publishes measured changes in model quality, training freshness, or incident rates after deployment.","Watch whether the state-management approach remains reliable as traffic and join windows grow."],"related_records":[],"new_branch_reason":"This starts a distinct analytical record because the supplied material documents a specific inference-to-training logging architecture and its scaling trade-offs, while no prior New Runtime record is supplied for comparison."},"routes":{"html":"https://newruntime.com/posts/taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4/","markdown":"https://newruntime.com/posts/taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4.md","json":"https://newruntime.com/posts/taming-the-ml-firehose-scaling-feature-consistency-analysis-1b9ea4.json"},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}