---
type: "post"
id: "making-open-models-work-for-long-horizon-agentic-f8bdac"
slug: "making-open-models-work-for-long-horizon-agentic-f8bdac"
title: "Making open models work for long-horizon agentic workloads"
description: "Ollama and Together AI explain how coding-agent harnesses and inference engines can preserve prefix-cache reuse across long sessions."
retrieval_nugget: "Ollama and Together AI explain how coding-agent harnesses and inference engines can preserve prefix-cache reuse across long sessions. Their practical guidance focuses on stable prompts, bounded tool output, selective subagent context, and cache-aware routing—choices that can affect latency, GPU memory, and the cost of repeated turns."
published_at: "2026-11-19"
updated_at: "2026-11-19"
record_date: "2026-11-19"
date_kind: "published_at"
topics: ["open-models","claude","claude-code"]
entities: ["Ollama","Together AI","Claude Code","Codex","Pi","DeepSeek-V4.1-Flash","GLM-5.3"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: null
prior_refs: []
source_url: "https://ollama.com/blog/open-models-long-horizon-agents"
source_urls: ["https://ollama.com/blog/open-models-long-horizon-agents"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:making-open-models-work-for-long-horizon-agentic-f8bdac"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/making-open-models-work-for-long-horizon-agentic-f8bdac/","markdown":"https://newruntime.com/posts/making-open-models-work-for-long-horizon-agentic-f8bdac.md","json":"https://newruntime.com/posts/making-open-models-work-for-long-horizon-agentic-f8bdac.json"}
markdown_url: "https://newruntime.com/posts/making-open-models-work-for-long-horizon-agentic-f8bdac.md"
---

# Making open models work for long-horizon agentic workloads

## Retrieval answer

Ollama and Together AI explain how coding-agent harnesses and inference engines can preserve prefix-cache reuse across long sessions. Their practical guidance focuses on stable prompts, bounded tool output, selective subagent context, and cache-aware routing—choices that can affect latency, GPU memory, and the cost of repeated turns.

## Summary
Ollama and Together AI explain how coding-agent harnesses and inference engines can preserve prefix-cache reuse across long sessions. Their practical guidance focuses on stable prompts, bounded tool output, selective subagent context, and cache-aware routing—choices that can affect latency, GPU memory, and the cost of repeated turns.

## What happened
Ollama’s November 16, 2026 announcement, by Parth Sareen and Zain Hasan of Together AI, describes the traffic pattern common to coding agents: each turn resends the system prompt, tool definitions, the user’s request, files read, and commands run, while the response is often a short tool call. The authors call this large input sequence length and small output sequence length. They argue that an agent turn is therefore dominated by prefill—the engine reading the prompt—rather than the token-by-token decode that dominates many ordinary chat responses.

The post explains that prefix caching reuses computation for a matching prompt prefix across requests. Its idealized example says ten turns with equal additions process 55 units of prompt without caching versus 10 with caching; at 100 turns, the comparison is 5,050 versus 100. The authors caution that real turns vary and cached tokens still require attention. A cache only matches until the first differing token, so changing content near the beginning can force the engine to process everything after it again.

On the harness side, Ollama says it shipped the `ollama launch` command to connect harnesses including Claude Code, Codex, and Pi to open models including DeepSeek-V4.1-Flash and GLM-5.3. The post advises keeping the system prompt identical from turn to turn and placing changing information, such as timestamps or working directories, in the task message once. It also describes a trade-off in late tool loading: putting a newly searched-for tool definition at the end can preserve the earlier prefix, but some model chat templates do not support definitions inserted mid-conversation. For GLM-5.3-Flash, the authors say reasoning effort is written into the first system block; changing it mid-session can invalidate most cache reuse. They recommend choosing the level at the start, or starting a fresh session if a higher level is needed.

Tool output and subagent context also matter. The authors’ example warns that returning only the first 100 lines of a 640-line file without saying so can mislead an agent; they recommend marking the range and telling the model it can search or request another range. For subagents, inheriting a parent session may reuse its cached prefix, but also makes the subagent read through irrelevant history and consumes GPU memory. A concise brief can be more appropriate when a reviewer only needs a few files and a checklist.

On the engine side, Together AI describes routing requests to a node holding the session cache, using a client-supplied `prompt_cache_key` or matching cached prefixes. It says cache-aware routing must balance a warm node against an idle one, and describes offloading idle cache to CPU memory while tools run. The post also names PagedAttention, continuous batching, and separate prefill and decode clusters connected over RDMA. It says a GPU costs around $50,000 per year to rent, using that figure to explain pressure to keep GPUs busy; it does not present a measured before-and-after cost or latency result for the described practices.

## What the archive adds
New Runtime’s earlier coverage, [“Open Models have crossed a threshold”](https://newruntime.com/posts/open-models-have-crossed-a-threshold-analysis-8bf2ef/) (October 16, 2026), noted that open-weight models were becoming viable for defined agent workloads while parity remained task- and harness-specific. This is a new engineering fact on that broader story: performance depends not only on model choice, but also on whether the harness and inference engine preserve cache reuse over a long run.

## Trend signal
This supports the pattern [“Harness architecture outlives model choice”](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/). The post’s examples make harness behavior—prompt stability, tool-output handling, and subagent context—part of the operational performance boundary, while the engine must route and retain the matching cache. The guidance does not establish that one harness or model is universally faster; it shows how architecture around the model can determine whether long sessions repeatedly pay to process old context.

## Why it matters
For teams running agents over substantial tasks, a seemingly minor prompt change can turn accumulated context into repeated prefill work. That has consequences for latency and inference cost, but also for capacity: the authors say oversized tool results and irrelevant inherited history occupy cache memory that could serve other sessions. The source does not quantify savings in production, so the useful implication is a design and measurement agenda, not a guaranteed cost reduction.

The advice also connects efficiency to correctness. Truncating a file result without identifying the omitted range can lead an agent to act as if it inspected the whole file. Keeping output bounded is useful only when the harness makes the boundary legible and offers a way to retrieve missing context. Likewise, subagents should inherit history only when the task benefits from it. Teams can treat cache behavior, context selection, and visible truncation as properties of the workflow to test—not hidden details that can be left entirely to the model.

## Limits and uncertainty
The post is a joint technical explanation from Ollama and Together AI, whose work spans harness integrations and inference infrastructure. Its arithmetic examples are explicitly idealized, and it reports no controlled benchmark, production latency distribution, or quantified cost savings. Cache behavior also depends on the model’s chat template, the harness’s request construction, engine routing, workload, and available memory. The described mechanisms do not guarantee a cache hit: a node can be overloaded, a session cache can be unavailable, or a prompt change can invalidate the matching prefix. The announcement does not specify pricing or general availability terms for the described integrations.

## Skill to build now
Instrument one long agent workflow this week. Log the stable system prompt, the first changing token between turns, tool-output size and truncation notices, and which context each subagent inherits. Use a test task with a long file and a delayed tool call to check whether the next turn reuses the session cache and whether the agent can locate omitted lines. Keep reasoning effort fixed during the run; compare a fresh higher-effort session separately rather than changing the setting mid-session.

## Reader takeaway
Long-horizon agent efficiency is a joint property of the harness and the inference engine. Keep early prompt content stable, make truncation explicit, and give subagents only the history they need; then verify cache behavior in the actual model and serving setup rather than assuming prefix caching will work automatically.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Open Models](https://newruntime.com/topics/open-models.md): Explore the open-models topic hub.
- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Claude Code](https://newruntime.com/topics/claude-code.md): Explore the claude-code topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
