---
type: analysis
slug: why-tool-call-reduction-not-bigger-models-will-be-the-decisive-lever-for-fast
title: "Why tool‑call reduction, not bigger models, will be the decisive lever for fast AI assistants"
description: "Cerebras shows that shaving minutes off personal‑assistant latency comes from re‑architecting the orchestration layer—parallel checks, reusable navigation procedures, and fewer tool calls—rather than from raw model speed. Builders should focus on execution‑path engineering first, treating tool‑call topology as a core performance budget."
published_at: 2026-09-25T08:00:00.000Z
topics: ["orchestration"]
source_urls: ["https://www.cerebras.ai/blog/the-rise-of-slow-personal-assistants","https://z.ai/blog/glm-built-its-inference-infrastructure","https://arxiv.org/pdf/2406.11695"]
---
# Why tool‑call reduction, not bigger models, will be the decisive lever for fast AI assistants

Cerebras shows that shaving minutes off personal‑assistant latency comes from re‑architecting the orchestration layer—parallel checks, reusable navigation procedures, and fewer tool calls—rather than from raw model speed. Builders should focus on execution‑path engineering first, treating tool‑call topology as a core performance budget.

## Summary

Cerebras benchmarked several consumer AI assistants on a dinner‑reservation task. Meta Muse took 4 min 36 s, Claude Cowork 6 min 25 s (57 tool calls), and Grok Bot 7 min 40 s. Their own Qwen 3.8 27B agent, using the Pi harness, achieved a 22‑second median after cutting tool calls by >80 % and parallelising availability checks. The company attributes the speedup to execution‑path changes—parallel checks, reusable site‑navigation procedures, and reduced inference pauses—rather than to a faster model alone. The evidence is internal and bundles multiple changes, but it highlights a clear pattern: orchestration overhead dominates latency in today’s consumer agents.

## What happened

In a controlled experiment, Cerebras ran the same dinner‑reservation request across four agents. The three public agents relied on sequential tool calls, resulting in multi‑minute latencies. Cerebras’ own configuration used the Qwen 3.8 27B model with the Pi agent harness. By restructuring the workflow—parallelising independent availability checks, persisting a reusable navigation routine, and trimming the number of tool calls from dozens to a handful—they reduced end‑to‑end latency to a median of 22 seconds across two successful runs. The browser/API segment dropped from 4 min 31 s (an earlier Grok trace) to 6.8 seconds after the same optimisations. The reported 80 % cut in tool calls suggests that most of the latency was spent waiting on external services, not on model inference. Cerebras warns that the benchmark mixes model, harness, and execution‑path changes, so the exact contribution of each is unclear, but the magnitude of the gain points to orchestration as the primary bottleneck.

## What the archive adds

Prior work on AI‑driven agents has repeatedly highlighted the cost of tool‑call orchestration. GLM‑5.3’s rollout required building a custom inference service and an Infra Agent to manage hardware constraints, showing that infrastructure layers can dominate performance. Separately, research on automated prompt optimization (e.g., Databricks’ MIPROv2, SIMBA, GEPA) demonstrates that pipeline‑agnostic improvements can close gaps between open‑source and closed‑source models, reinforcing the idea that non‑model factors are critical for real‑world latency. These trends indicate a broader industry shift: performance budgets now include tool‑call topology, procedural memory, and inference latency as a combined target.

## Trend signal

The pattern mirrors a maturing AI‑assistant ecosystem where raw model scaling yields diminishing returns for user‑facing latency. Early assistants focused on model quality, but as they move from demos to transactional use‑cases, orchestration overhead becomes the limiting factor. Companies are now engineering end‑to‑end pipelines—parallelising API calls, caching navigation steps, and reducing unnecessary tool invocations—to meet user expectations of sub‑minute response times. This mirrors the broader trend of “systems‑first” AI, where infrastructure, caching, and workflow engineering are as valuable as model improvements.

## Connections to other events

Cerebras’ benchmark intersects with two notable industry movements. First, the GLM‑5.3 infrastructure build shows that large‑scale inference services are often built by the models themselves, hinting at a future where agents can self‑optimise their orchestration. Second, the Databricks prompt‑optimization research proves that pipeline‑agnostic techniques can dramatically improve performance without changing the underlying model. Together, these suggest that the next wave of fast assistants will be built on three pillars: (1) a model that meets quality thresholds, (2) an execution engine that parallelises and caches tool calls, and (3) automated optimisation of prompts and workflow steps. The Cerebras result is a concrete data point that the second pillar alone can slash latency by an order of magnitude.

## Why it matters

For builders, the immediate takeaway is to treat the tool‑call graph as a first‑class performance budget. Practical steps include: • Identify independent external checks (e.g., availability, pricing) and run them in parallel. • Extract repeatable web‑navigation sequences into reusable procedures or cached state, updating only dynamic data. • Instrument the agent to count tool calls and set targets (e.g., <10 calls for a typical transaction). • Evaluate harnesses that support asynchronous execution and procedural memory. For buyers, vendors that advertise faster response times solely on model size may be overpromising; the real differentiator will be how efficiently the agent orchestrates external services.

## Limits and uncertainty

The evidence comes from a single vendor’s internal tests on a narrow use‑case (dinner reservation) and mixes multiple variables (model, harness, execution path). We lack third‑party replication across diverse domains, websites, and hardware configurations. The reported 22‑second median is based on only two successful runs, so statistical significance is uncertain. Additionally, the benchmark does not isolate the impact of each optimisation, making it hard to quantify the exact ROI of parallelisation versus procedure reuse versus model choice.

## Reader takeaway

If you are building a consumer AI assistant, stop chasing ever larger models as the primary latency fix. Instead, audit your tool‑call workflow, parallelise independent calls, and codify reusable navigation steps. Treat the orchestration layer as a performance budget equal to the model’s inference budget, and you’ll likely achieve order‑of‑magnitude speed gains before needing to upgrade the underlying model.

## Sources

- [Why AI Assistants Are Slow—and How to Make Them Faster](https://www.cerebras.ai/blog/the-rise-of-slow-personal-assistants)

- [Toward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure](https://z.ai/blog/glm-built-its-inference-infrastructure)

- [Building State-of-the-Art Enterprise Agents 90x Cheaper with Automated Prompt Optimization](https://arxiv.org/pdf/2406.11695)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json