---
id: "nr-mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653"
slug: "mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653"
title: "Mozilla.ai benchmark finds build flags beat server choice for local LLMs"
observed_at: "2026-09-18T13:36:42.507001+00:00"
record_date: "2026-09-18"
date_kind: "observed_at"
novelty: "new"
source_platform: "blog.mozilla.ai"
topics: ["local-llm","benchmarks","inference"]
entities: ["Mozilla.ai","llama.cpp","Ollama"]
related_patterns: []
source_url: "https://blog.mozilla.ai/benchmarking-local-llm-servers-llama-cpp-llamafile-lm-studio-and-ollama"
source_urls: ["https://blog.mozilla.ai/benchmarking-local-llm-servers-llama-cpp-llamafile-lm-studio-and-ollama"]
basket: {"id":"0325e6b8-f527-4f9b-bfa3-3f30e57f380b","revision":1,"cluster_id":"eae65321-27da-41d9-a27b-5b1ebf73b555"}
type: "post"
description: "Mozilla.ai compared llama.cpp, llamafile, LM Studio, and Ollama on three machines and found configuration, not engine choice, drove the largest throughput gains."
why_it_matters: "Worth watching is whether LM Studio changes its prebuilt runtime, since the authors attribute its deficit to that runtime rather than to its engine."
verification_level: "source-inspected"
signal_type: "field_note"
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653"
retrieval_nugget: "Mozilla.ai compared llama.cpp, llamafile, LM Studio, and Ollama on three machines and found configuration, not engine choice, drove the largest throughput gains. Mozilla.ai benchmarked four local LLM servers, llama.cpp, llamafile, LM Studio, and Ollama, on a Mac Studio M4 Max using Metal, a Linux server with an NVIDIA L40S, and a Steam Deck using Vulkan. The tests used Qwen models"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:f8f7266d1b86adcc5139049809f412cd7570470494e8e686b7669d8066ae93dc","reviewed_at":"2026-09-24T13:28:46Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The central finding is that build and configuration choices moved local inference throughput more than the choice of server.","observed_facts":[{"text":"All four servers serve GGUF models through llama.cpp under the hood.","source_urls":["https://blog.mozilla.ai/benchmarking-local-llm-servers-llama-cpp-llamafile-lm-studio-and-ollama"]},{"text":"Enabling CUDA graphs in llamafile improved L40S decoding by 16.8% on the 0.8B model.","source_urls":["https://blog.mozilla.ai/benchmarking-local-llm-servers-llama-cpp-llamafile-lm-studio-and-ollama"]},{"text":"A newer Vulkan shader toolchain raised Steam Deck prompt processing by 63.3% on the 9B model.","source_urls":["https://blog.mozilla.ai/benchmarking-local-llm-servers-llama-cpp-llamafile-lm-studio-and-ollama"]}],"mechanism":"The mechanism is shared code: all four serve GGUF through llama.cpp, so prompt processing lands within a few percent once weights and environment are fixed, while the real differences come from compile flags, shader toolchains, and host-side per-token overhead.","why_now":"This matters now because agents and large codebases push local servers toward long prompts, where prompt-processing throughput matters most.","implications":["One implication is that speculative-decoding draft length should be tuned per platform, since draft 2 won on Metal while draft 4 beat draft 2 on the L40S by up to 16%."],"evidence_boundary":"The study does not establish a general ranking of the servers, and its authors describe it as a snapshot of specific versions, models, and hardware.","watch_conditions":["Worth watching is whether LM Studio changes its prebuilt runtime, since the authors attribute its deficit to that runtime rather than to its engine."],"related_records":[],"new_branch_reason":"A controlled cross-platform benchmark of local LLM servers with pinned versions gives engineers reusable configuration evidence that deserves its own record."}
routes: {"html":"https://newruntime.com/posts/mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653/","markdown":"https://newruntime.com/posts/mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653.md","json":"https://newruntime.com/posts/mozilla-ai-benchmark-finds-build-flags-beat-server-choice-eae653.json"}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
---

# Mozilla.ai benchmark finds build flags beat server choice for local LLMs

## Retrieval answer

Mozilla.ai compared llama.cpp, llamafile, LM Studio, and Ollama on three machines and found configuration, not engine choice, drove the largest throughput gains. Mozilla.ai benchmarked four local LLM servers, llama.cpp, llamafile, LM Studio, and Ollama, on a Mac Studio M4 Max using Metal, a Linux server with an NVIDIA L40S, and a Steam Deck using Vulkan. The tests used Qwen models

Mozilla.ai benchmarked four local LLM servers, llama.cpp, llamafile, LM Studio, and Ollama, on a Mac Studio M4 Max using Metal, a Linux server with an NVIDIA L40S, and a Steam Deck using Vulkan. The tests used Qwen models at 0.8B, 9B, and 27B, the same GGUF files across runtimes, and prompt sizes from 1,024 to 8,192 tokens.

The central finding is that build and configuration choices moved local inference throughput more than the choice of server. The mechanism is shared code: all four serve GGUF through llama.cpp, so prompt processing lands within a few percent once weights and environment are fixed, while the real differences come from compile flags, shader toolchains, and host-side per-token overhead. Enabling CUDA graphs in llamafile raised L40S decode by 16.8% on the 0.8B model, and a newer Vulkan shader toolchain raised Steam Deck prompt processing by up to 63.3% on the 9B model.

This matters now because agents and large codebases push local servers toward long prompts, where prompt-processing throughput matters most. One implication is that speculative-decoding draft length should be tuned per platform, since draft 2 won on Metal while draft 4 beat draft 2 on the L40S by up to 16%. The study does not establish a general ranking of the servers, and its authors describe it as a snapshot of specific versions, models, and hardware.

Worth watching is whether LM Studio changes its prebuilt runtime, since the authors attribute its deficit to that runtime rather than to its engine.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Local Llm](https://newruntime.com/topics/local-llm.md): Explore the local-llm topic hub.
- topic hub: [Benchmarks](https://newruntime.com/topics/benchmarks.md): Explore the benchmarks topic hub.
- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)