---
type: "post"
id: "castle-training-natural-language-search-before-the-first-6a8432"
slug: "castle-training-natural-language-search-before-the-first-6a8432"
title: "CASTLE: training natural language search before the first real query"
description: "Why CASTLE changes the cold-start data problem Thesis The useful unit in this search cold start is a query paired with a behavior-grounded relevance distinction not a plausible query alone. Airbnb reports CASTLE generates queries from contrastive listing pairs based"
retrieval_nugget: "Why CASTLE changes the cold-start data problem Thesis The useful unit in this search cold start is a query paired with a behavior-grounded relevance distinction not a plausible query alone. Airbnb reports CASTLE generates queries from contrastive listing pairs based"
published_at: "2026-10-13"
updated_at: "2026-10-13"
record_date: "2026-10-13"
date_kind: "published_at"
topics: []
source_urls: ["https://airbnb.tech/ai-ml/castle-training-natural-language-search-before-the-first-real-query"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:castle-training-natural-language-search-before-the-first-6a8432"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:b9288fd187382114ff9a4a121d0ea0bcea7d2bae697078f0629ae149ca635cc8","reviewed_at":"2026-10-13T20:07:20.881Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The useful unit in this search cold start is a query paired with a behavior-grounded relevance distinction, not a plausible query alone.","observed_facts":[{"text":"Airbnb says CASTLE generates queries from contrastive listing pairs derived from de-identified booking sessions, selected according to privacy permissions.","source_urls":["https://airbnb.tech/ai-ml/castle-training-natural-language-search-before-the-first-real-query"]},{"text":"Airbnb reports KL divergence of 1.01 against real query distributions versus 9.33 for its best baseline.","source_urls":["https://airbnb.tech/ai-ml/castle-training-natural-language-search-before-the-first-real-query"]},{"text":"Airbnb reports 91–93% expert agreement across three raters reviewing 200 triplets.","source_urls":["https://airbnb.tech/ai-ml/castle-training-natural-language-search-before-the-first-real-query"]}],"mechanism":"The system conditions a language model on both listings and asks for a query satisfied by the positive but not the negative, so the relevance label follows the observed choice rather than a separate model judgment. For pairs without booking history, Airbnb adds an LLM Virtual Judge.","why_now":"Airbnb says natural-language search had to be trained before real user queries existed, while synthetic-query methods risk mismatching actual guest phrasing and require labels.","implications":["Search teams facing a traffic cold start can treat existing choice behavior as structured supervision for both query generation and relevance, rather than relying only on generated text plus post-hoc judgments.","This may reduce circularity in synthetic labeling, while leaving gaps where behavior signals are unavailable."],"evidence_boundary":"These metrics and implementation details are Airbnb's own published account; the supplied source does not provide independent replication or enough detail to assess all deployment effects.","watch_conditions":["Whether later evaluations show the same alignment and label quality on new markets, listing types, or independently reproduced datasets."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied for this specific joint query-and-label construction."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/castle-training-natural-language-search-before-the-first-6a8432/","markdown":"https://newruntime.com/posts/castle-training-natural-language-search-before-the-first-6a8432.md","json":"https://newruntime.com/posts/castle-training-natural-language-search-before-the-first-6a8432.json"}
---

# CASTLE: training natural language search before the first real query

## Retrieval answer

Why CASTLE changes the cold-start data problem Thesis The useful unit in this search cold start is a query paired with a behavior-grounded relevance distinction not a plausible query alone. Airbnb reports CASTLE generates queries from contrastive listing pairs based

## Why CASTLE changes the cold-start data problem

Thesis: The useful unit in this search cold start is a query paired with a behavior-grounded relevance distinction, not a plausible query alone. Airbnb reports CASTLE generates queries from contrastive listing pairs based on de-identified booking sessions, with the booked listing positive and a viewed-but-not-booked alternative negative. Its mechanism: The system conditions a language model on both listings and asks for a query satisfied by the positive but not the negative, so the relevance label follows the observed choice rather than a separate model judgment. For pairs without booking history, Airbnb adds an LLM Virtual Judge.

Why now: Airbnb says natural-language search had to be trained before real user queries existed, while synthetic-query methods risk mismatching actual guest phrasing and require labels. The company reports KL divergence of 1.01 against real query distributions versus 9.33 for the best baseline, and reports 91–93% expert agreement across three raters evaluating 200 triplets. First implication: Search teams facing a traffic cold start can treat existing choice behavior as structured supervision for both query generation and relevance, rather than relying only on generated text plus post-hoc judgments. This may reduce circularity in synthetic labeling, while leaving gaps where behavior signals are unavailable.

Evidence boundary: These metrics and implementation details are Airbnb's own published account; the supplied source does not provide independent replication or enough detail to assess all deployment effects. First watch condition: Whether later evaluations show the same alignment and label quality on new markets, listing types, or independently reproduced datasets. This record starts a distinct analytical branch because no prior New Runtime record is supplied for this specific joint query-and-label construction.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)