---
type: "post"
id: "independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc"
slug: "independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc"
title: "Independent 4-bit run reproduces Nous Propaganda_Eval scores for Arete and Qwen"
description: "Sonny’s Labs reports that one 4-bit run of Arete-27B and Qwen3.8-27B reproduced Nous Research’s Propaganda_Eval results closely."
retrieval_nugget: "Sonny’s Labs reports that one 4-bit run of Arete-27B and Qwen3.8-27B reproduced Nous Research’s PropagandaEval results closely. The rerun supports the repeatability of those measurements, not the neutrality or completeness of the benchmark—a distinction that matters whenever teams compare models using a test its creator also shaped."
published_at: "2026-11-16"
updated_at: "2026-11-16"
record_date: "2026-11-16"
date_kind: "published_at"
topics: ["evals","models"]
entities: ["Sonny’s Labs","Nous Research","Arete-27B","Qwen3.8-27B","Propaganda_Eval","Grok 4.7","OpenRouter","Splash"]
related_patterns: ["scientific-agents-need-artifact-evidence"]
novelty_label: "new_fact"
source_url: "https://sonny.firetussin.com/reports/arete-vs-qwen-propaganda-eval"
source_urls: ["https://sonny.firetussin.com/reports/arete-vs-qwen-propaganda-eval"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc/","markdown":"https://newruntime.com/posts/independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc.md","json":"https://newruntime.com/posts/independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc.json"}
markdown_url: "https://newruntime.com/posts/independent-4-bit-run-reproduces-nous-propaganda-eval-b09bdc.md"
---

# Independent 4-bit run reproduces Nous Propaganda_Eval scores for Arete and Qwen

## Retrieval answer

Sonny’s Labs reports that one 4-bit run of Arete-27B and Qwen3.8-27B reproduced Nous Research’s PropagandaEval results closely. The rerun supports the repeatability of those measurements, not the neutrality or completeness of the benchmark—a distinction that matters whenever teams compare models using a test its creator also shaped.

## Summary
Sonny’s Labs reports that one 4-bit run of Arete-27B and Qwen3.8-27B reproduced Nous Research’s Propaganda_Eval results closely. The rerun supports the repeatability of those measurements, not the neutrality or completeness of the benchmark—a distinction that matters whenever teams compare models using a test its creator also shaped.

## What happened
Sonny’s Labs says it ran all 803 questions in Nous Research’s Propaganda_Eval on 11 November 2026. The models were Arete-27B, Nous’s fine-tune of Alibaba’s Qwen3.8-27B, and the base Qwen3.8-27B. Both were run as Q4_K_M, 4-bit GGUF files on a Mac mini with an Apple M6 chip and 32 GB of unified memory. The report says the two models used the same engine, settings and speculative draft model; only one model fit in memory at a time, so they were run sequentially.

The report used Splash 1.6.0, the Nous grader code without changes, and x-ai/grok-4.7 through OpenRouter as the judge. The evaluation ran at temperature zero, with thinking mode off, no system prompt, one question per conversation and a 32,768-token answer limit. The report says all 1,606 answers received valid verdicts. The dataset commit was 401223a2. The Mac generated the answers locally; judging incurred the reported $42.76 in OpenRouter fees. Sonny’s Labs reports that generating answers took 4.31 hours for Arete and 5.07 hours for Qwen.

On the report’s 0–5 mean fact score, Arete scored 2.932 and base Qwen 2.298. The share of answers assigned a propaganda score of 1 or 2—the report’s definition of adopting the state line—was 0.75% for Arete (6 of 803) and 22.04% for Qwen (177 of 803). Nous’s published figures were 2.98 and 0.9% for Arete, and 2.32 and 22% for Qwen. Sonny’s Labs says the rerun’s differences between the two models were also close to Nous’s: +0.634 fact-score points and −21.3 percentage points in state-line answers, compared with Nous’s +0.66 and −21.1.

The report describes the test as 803 English prompts on topics the People’s Republic of China censors or presents through an official narrative. Each question includes two to four required facts from a cited source. Grok 4.7 scores factual coverage from 0 to 5 and propaganda from 0 to 2. A score of 1 or 2 counts as a state-line answer; attributing an official position to the state, rather than adopting it as the answer’s own view, receives 0. The report says the questions comprise 564 named-fact, 222 frame-testing and 17 closed-exit prompts.

The results were not uniformly favorable to Arete. The judge marked at least one required fact as opposed in 175 Arete answers, compared with 293 for Qwen. Arete scored zero on 230 questions, and scored lower than Qwen on 89; it scored higher on 239 and tied on 475. One Arete answer repeated the same bullet until reaching the answer limit. These results sit alongside the substantially lower state-line rate, rather than replacing the need to examine what the benchmark rewards and how individual answers perform.

## What the archive adds
New Runtime covered Nous’s release of Arete-27B on [7 November 2026](https://newruntime.com/posts/nous-research-released-arete-27b-a-qwen3-8-a169fd/) and the public release of the [Propaganda_Eval benchmark and training pairs](https://newruntime.com/posts/held-out-propaganda-eval-and-training-pairs-are-07ab29/) on 8 November 2026. This is a new fact on that story: an outside team’s 4-bit rerun closely matches Nous’s reported scores, while making clear that score replication does not validate the test’s framing.

## Trend signal
This result supports the importance of inspectable evidence in agent and model evaluation, though it is a single, bounded replication rather than a general demonstration of reproducible research. The report identifies its model files, hashes, software, settings, dataset commit, grader and judge, and publishes results and code. That kind of procedural detail makes comparison more testable; the [scientific agents need artifact evidence](https://newruntime.com/patterns/scientific-agents-need-artifact-evidence/) pattern is relevant because independent scrutiny depends on being able to inspect how a result was produced.

## Why it matters
For teams choosing a model, a score is only useful when its measurement process is understood. This rerun is useful because it separates two questions often collapsed in benchmark coverage: can another setup reproduce the reported result, and does the evaluation measure the quality or behavior a team actually cares about? The first gets meaningful support here. The second remains unresolved: Nous created the evaluation and trained Arete, and Sonny’s Labs notes that some prompts suggest a particular answer.

Teams should also avoid treating the headline gap as a general capability ranking. The tested set covers English-language political questions about China, not Chinese-language responses, code, general capability or other governments’ official narratives. Even within the tested scope, Arete opposed a required fact in 175 answers. The useful lesson is therefore not simply that one model is safer; it is that model comparisons need disclosed test construction, scoring rules and error-level results alongside aggregate scores.

## Limits and uncertainty
The rerun comprised one run per model, while Nous reports averages across five runs. Sonny’s Labs says its results depend on the Grok 4.7 API judge and apply to the judge date, 11 November 2026; another judge may yield different scores. No person independently checked the judge’s verdicts. The report did not test other quantizations or inference engines, run the questions in Chinese, enable thinking mode, or evaluate general capability, code security or adversarial system prompts. The authors also note that some prompts may reward boilerplate. These boundaries mean the close match is evidence about this particular setup, not a validation of broader claims about political bias or model behavior.

## Reader takeaway
A reproducible benchmark result is not automatically a neutral or complete benchmark. Treat this rerun as evidence that Nous’s published Propaganda_Eval scores can be closely reproduced under a specified 4-bit setup. Before using those scores to choose a model, inspect the questions, scoring rubric and answer-level failures—and check whether the test represents the language, tasks and risks your own product will face.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
