---
type: "post"
id: "zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d"
slug: "zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d"
title: "Zyphra introduces ZHA, a hierarchical sparse attention with subquadratic cost that can be distilled into existing models"
description: "Zyphra introduced Zyphra Hierarchical Attention (ZHA), a sparse-attention method that searches a tree of compressed key summaries instead of indexing every token in a long context."
retrieval_nugget: "Zyphra introduced Zyphra Hierarchical Attention (ZHA), a sparse-attention method that searches a tree of compressed key summaries instead of indexing every token in a long context. Its reported benchmarks suggest that existing pretrained models can retain strong task performance while attending to a small share of decode keys, but the headline speedups remain modeled rather than demonstrated with optimized kernels."
published_at: "2026-11-03"
updated_at: "2026-11-03"
record_date: "2026-11-03"
date_kind: "published_at"
topics: ["agent-economics"]
entities: ["Zyphra","Zyphra Hierarchical Attention","Qwen 3.6 35B MoE"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Zyphra introduces ZHA, a distinct tree-routed sparse attention method with subquadratic cost and distillation into pretrained models."
prior_refs: ["https://newruntime.com/posts/proxyattn-a-training-free-method-from-researchers-at-59312c/","https://newruntime.com/posts/this-ai-lab-takes-on-jev-with-12x-0746e5/"]
source_url: "https://zyphra.com/our-work/zyphra-hierarchical-attention"
source_urls: ["https://zyphra.com/our-work/zyphra-hierarchical-attention"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d/","markdown":"https://newruntime.com/posts/zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d.md","json":"https://newruntime.com/posts/zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d.json"}
markdown_url: "https://newruntime.com/posts/zyphra-introduces-zha-a-hierarchical-sparse-attention-with-967a3d.md"
---

# Zyphra introduces ZHA, a hierarchical sparse attention with subquadratic cost that can be distilled into existing models

## Retrieval answer

Zyphra introduced Zyphra Hierarchical Attention (ZHA), a sparse-attention method that searches a tree of compressed key summaries instead of indexing every token in a long context. Its reported benchmarks suggest that existing pretrained models can retain strong task performance while attending to a small share of decode keys, but the headline speedups remain modeled rather than demonstrated with optimized kernels.

## Summary
Zyphra introduced Zyphra Hierarchical Attention (ZHA), a sparse-attention method that searches a tree of compressed key summaries instead of indexing every token in a long context. Its reported benchmarks suggest that existing pretrained models can retain strong task performance while attending to a small share of decode keys, but the headline speedups remain modeled rather than demonstrated with optimized kernels.

## What happened
In a research post dated November 2, 2026, Zyphra introduced ZHA, a method intended to reduce the cost of attention over very long sequences. Standard attention compares each new query with all preceding tokens, so the attention computation grows quadratically with context length. Zyphra argues that existing sparse-attention methods do not fully solve this scaling problem when their indexer must still score every token or chunk to find a smaller set for attention.

ZHA’s proposed alternative is a hierarchy over the key-value cache. The source describes recursively compressed representations organized as a tree, with the original keys retained at the leaves. For each query, the method routes through the hierarchy using adaptive top-p selection, narrowing the search to relevant branches rather than scanning the full set of keys. Zyphra says that, with suitably chosen routing thresholds, this produces subquadratic complexity. The approach is query-dependent: it does not discard a fixed set of tokens in advance, and the actively retrieved set can grow with sequence length at a sublinear rate.

The post also describes adapting existing pretrained models through a distillation phase. Zyphra says this can be done with only a few billion tokens and reports an example configuration using 81,920 trainable parameters for Qwen 3.6 35B MoE. The post’s summary characterizes the adaptation as a way to retrofit models, rather than requiring a complete model retraining. It does not present ZHA as a generally available hosted service, provide a price, or specify a released production package in the supplied material.

Zyphra reports results on long-context reasoning, coding and retrieval benchmarks. On SWE-bench Verified, it reports 67.4% for the deep configuration and 68.4% for the wide configuration, compared with 67.4% for dense attention. On RULER-13 at 256K context, it reports 93.3% with a six-level tree and 96.2% for a wide tree, against 96.0% for dense attention. On LongBench v2, the reported result is 63.4% versus 58.0% for dense attention, while attending to about 1.1% of decode keys. The headline speed comparisons come from a performance model: Zyphra projects crossover points against dense and other sparse attention approaches and very large gains at ultra-long context lengths. Those are projections, not measured production speedups.

## What the archive adds
This is a new fact in a continuing line of long-context attention work. Our earlier [SubQ Snap note](https://newruntime.com/posts/this-ai-lab-takes-on-jev-with-12x-0746e5/) covered a product proposition centered on keeping agent decisions bounded as evidence grows; [ProxyAttn](https://newruntime.com/posts/proxyattn-a-training-free-method-from-researchers-at-59312c/) explored more precise attention-block selection without training a separate estimator. ZHA differs by proposing a hierarchical index and a distillation route for pretrained models.

## Trend signal
ZHA supports [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/) only indirectly: it targets the compute cost of processing long contexts, a possible input to the cost of completing a task. The release does not establish lower end-to-end task cost, since its speed gains are modeled and no optimized kernels are described in the supplied material.

## Why it matters
For teams whose agents repeatedly inspect large codebases, document collections or long histories, attention cost can constrain how much evidence they can afford to provide to a model. A method that makes retrieval itself cheaper, rather than merely reducing the final attention set after a full scan, would address an important source of long-context compute. The distillation claim is also relevant because it suggests a path to testing this kind of attention change without training a foundation model from scratch.

The practical implication is not yet to assume a cost or latency reduction. Benchmarks indicate a promising quality profile, but usable throughput depends on implementation and hardware behavior. Teams should treat ZHA as a research candidate, wait for optimized kernels or reproducible measurements, and then compare it against dense attention on the specific workloads, context lengths and quality requirements they operate.

## Limits and uncertainty
The results are reported by Zyphra, the method’s developer; the supplied material does not include independent replication. The source distinguishes benchmark performance from projected speed and compute costs, and no optimized kernels are identified in the supplied evidence. The reported task scores do not establish that the method will preserve quality across all models, languages, context types or production workloads. Nor does the release establish an end-to-end cost reduction: serving overhead, hardware, batching, and the distribution of real context lengths all affect that outcome. Availability, licensing and pricing are not specified here.

## Reader takeaway
ZHA is a research proposal for making sparse attention’s search hierarchical, not a ready-made long-context speedup to adopt today. Its reported benchmark results are worth tracking, especially for teams extending existing models, but the modeled performance advantage needs measured kernels and independent workload testing before it can inform a production cost or latency plan.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Economics](https://newruntime.com/topics/agent-economics.md): Explore the agent-economics topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)