---
type: analysis
slug: local-gguf-tests-cannot-verify-liquid-ai-s-agent-benchmark-claim
title: "Local GGUF tests cannot verify Liquid AI’s agent benchmark claim"
description: "Liquid AI says LFM2.5-24B-A2B is competitive with Qwen3.6-35B-A3B on five of eight agentic benchmarks. Downloadable GGUF files make local trials possible, but they do not independently confirm that comparison."
published_at: 2026-10-20T08:00:00.000Z
topics: ["benchmarks"]
source_urls: ["https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF","https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF","https://www.liquid.ai/blog/lfm2-5-24b-a2b","https://huggingface.co/LiquidAI/LFM2.5-24B-A2B"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
# Local GGUF tests cannot verify Liquid AI’s agent benchmark claim

Liquid AI says LFM2.5-24B-A2B is competitive with Qwen3.6-35B-A3B on five of eight agentic benchmarks. Downloadable GGUF files make local trials possible, but they do not independently confirm that comparison.

## Summary

Liquid AI’s launch blog says LFM2.5-24B-A2B is competitive with Qwen3.6-35B-A3B on five of eight agentic benchmarks. That is a company-reported result, not independent validation. The Hugging Face repository offers multiple GGUF variants and llama.cpp instructions, making local testing more accessible; whether the reported advantage carries over to a particular quantization, runtime, hardware setup and agent task remains an open question.

## What happened

LFM2.5-24B-A2B is described in the release as a 24B mixture-of-experts model with 2.45B active parameters per token. Its GGUF repository lists BF16, F16, Q4_K_M, Q5_K_M, Q6_K, Q8_0 and QAD-Q4_0 files, as well as commands for serving and terminal inference with llama.cpp. The launch blog says the model is competitive with Qwen3.6-35B-A3B on five of eight agentic benchmarks; that comparison comes from Liquid AI, not an independent evaluation. According to the dossier, the Q4_K_M file is 14.5 GB. That is a concrete download and storage constraint, not a measure of runtime memory.

## What the archive adds

The archive includes Liquid AI’s launch blog, which gives the five-of-eight agentic benchmark comparison, and the model’s Hugging Face page, which contains evaluation-setting excerpts. These add a company-reported performance claim to the repository’s packaging details. They do not amount to independent validation of the claim or establish that a GGUF quantization reproduces benchmark results under a local runtime.

## Trend signal

Model releases increasingly arrive with both benchmark claims and artifacts that developers can run themselves. Those are different kinds of evidence: a company’s benchmark comparison describes reported performance under its evaluation conditions, while a local GGUF trial measures one quantization on one setup and workload. The gap matters especially for agentic tasks, where outcomes can depend on the task, runtime and evaluation procedure. Availability lowers the barrier to scrutiny; it does not make results directly comparable.

## Why now

First, Liquid AI’s launch blog reports a comparison on agentic benchmarks, giving the release a performance claim to evaluate. Second, the repository publishes several GGUF variants and llama.cpp instructions, so practitioners have a concrete route to local testing. Third, quantization imposes practical trade-offs: according to the dossier, Q4_K_M is a 14.5 GB file, although that figure does not establish runtime memory use.

## Connections to other events

The useful read is not that local access proves Liquid AI’s benchmark result, but that it gives practitioners a way to test whether the model is relevant to their own agent work. The five-of-eight comparison is a company claim about benchmarks, while a local Q4_K_M run tests a particular artifact and setup. The dossier lists that file at 14.5 GB; this signals a substantial download, not the memory required to serve it. To compare evidence, pin the quantization, runtime, hardware, agent task and scoring criteria.

## Why it matters

Builders should treat the benchmark result as a reason to run a focused comparison, not as a deployment decision. Teams considering local agent use can test one GGUF variant against their current baseline on a representative task, then record quality and serving behavior under the same conditions. Buyers should ask which benchmark settings support the company’s comparison and whether those settings resemble their own workflows. Liquid AI has made evaluation more actionable, but the practical case for the model still depends on results users can reproduce or obtain on their own workloads.

## What happens next

- **At least one independent public evaluation will compare LFM2.5-24B-A2B with a named baseline on an agentic task and specify the model variant and test conditions.** (3 months, 65%). Watch for: A public report names the LFM2.5 variant, baseline, task, runtime and hardware, and gives comparable results.
- **Independent matched evaluations will not establish a universal winner across all agentic tasks and local configurations.** (6 months, 70%). Watch for: Published comparisons show different outcomes across tasks or configurations rather than confirming one model as consistently superior.

## Limits and uncertainty

The supplied evidence does not provide an independent matched evaluation of the five-of-eight claim, nor does it establish that local GGUF results will match the launch blog’s benchmark conditions. The repository documents available files and instructions; the 14.5 GB Q4_K_M size does not reveal peak runtime memory, speed or quality. This take could be wrong if independent tests already reproduce the claim under comparable conditions, but those results are not established by the evidence here.

## Skill to build now

Build a matched agentic-task evaluation: run one named LFM2.5 GGUF quantization and your current baseline on the same representative task, with fixed scoring criteria and runtime settings, then record quality and serving behavior.

## Reader takeaway

This week, choose one named GGUF quantization and compare it with your current baseline on a representative agentic task. Keep the task, scoring criteria, runtime, hardware and settings fixed; log quality and serving behavior for both runs. Treat the launch blog’s five-of-eight comparison as a company-reported hypothesis to investigate, not a result your local test has already confirmed.

## Open question

Will independent, matched tests reproduce Liquid AI’s five-of-eight agentic benchmark comparison for a local GGUF variant, or will the result depend too heavily on the benchmark and deployment setup?

## Sources

- [Liquid AI released LFM2.5-24B-A2B , a 24B MoE with only 2.45B active parameters per token.](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF)

- [ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF · Hugging Face](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)

- [LFM2.5-24B-A2B: Built for Fast On-Device Agents | Blog | Liquid AI](https://www.liquid.ai/blog/lfm2-5-24b-a2b)

- [LiquidAI/LFM2.5-24B-A2B · Hugging Face](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B)

- [LFM2.5-24B-A2B-Q4_K_M.gguf](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF/blob/main/LFM2.5-24B-A2B-Q4_K_M.gguf)

- [README.md · unsloth/LFM2.5-24B-A2B-GGUF at main](https://huggingface.co/unsloth/LFM2.5-24B-A2B-GGUF/blob/main/README.md)

- [google/gemma-4-26B-A4B-it · Hugging Face](https://huggingface.co/google/gemma-4-26B-A4B-it)

- [NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · Hugging Face](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)

- [LFM2.5-24B-A2B on M5 Max: 184 tok/s measured, RAM & Setup](https://llm-speed.com/r/r_j5tgvia-kjk)

- [GitHub - GuideboardLabs/agon-arena: An open benchmark for local language models on consumer hardware. Three events. Hidden test grading. Consumer GPUs only.](https://github.com/GuideboardLabs/agon-arena)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Benchmarks](https://newruntime.com/topics/benchmarks.md): Explore the benchmarks topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)