---
type: analysis
slug: research-two-sycophancy-benchmarks-rank-models-oppositely-because-they-measure-flattery
title: "Two sycophancy benchmarks rank models oppositely because they measure flattery and backing down"
description: "TrueStandard's -0.57 rank correlation against AA-PersonalityBench checks out on its own data, but the tests differ in task, scoring and reasoning settings, and no one has run a matched comparison."
published_at: 2026-11-09T17:00:00.000Z
topics: ["reasoning","prompts","benchmarks","research"]
source_urls: ["https://truestandard.ai/blog/why-sycophancy-benchmarks-disagree","https://artificialanalysis.ai/methodology/aa-personalitybench","https://truestandard.ai/ai-sycophancy-benchmark","https://artificialanalysis.ai/evaluations/aa-personalitybench/traits/sycophantic"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
On 23 October 2026 TrueStandard, which sells a multi-model fact-checking product, published a comparison claiming its pushback test ranks 15 models nearly opposite to the sycophancy trait in Artificial Analysis's AA-PersonalityBench, launched 19 October. Grok 4.7 is the least sycophantic model on AA's chart yet moved under pushback on all three days TrueStandard tested it, while Gemini 4 Argon is among AA's most sycophantic and moved once in 270 pushbacks.

For builders, the question is whether one of these benchmarks is broken or whether 'sycophancy' names two separate behaviours. The answer changes which number you should use when picking a model for a reviewer, grader or customer-facing assistant.

## What we checked

- **Can the two benchmark protocols be rerun on matched models, prompts, and reasoning settings to establish whether the reported inverse ranking remains?** unresolved. No matched rerun exists, and one cannot be done from outside today because AA keeps its prompt set private apart from a small sample. TrueStandard itself says it cannot separate definition effects from reasoning-setting effects; its own raw files are named but not linked. [2][1][3]
- **Is the reported -0.57 rank correlation arithmetically correct on the published numbers?** confirmed. Recomputing Spearman's rho from the 15 rows TrueStandard publishes gives about -0.567, matching the reported value. [1]

## What we found

- TrueStandard's arithmetic holds on its own table. It reports a rank correlation of -0.57 across the 15 models both tests share, and says that clears p = 0.03. Recomputing a Spearman correlation from the 15 Elo scores and move rates it publishes gives about -0.567, so the inverse ordering is not a calculation error; the open question is what the two numbers measure. [1]
- AA-PersonalityBench scores sycophancy as a writing-style trait, not as answer stability. Each model answers each prompt once with no system prompt, and a judge from a three-model panel picks which of two blinded answers shows the trait more. The full prompt set is private apart from a small public sample, which is what blocks an outside matched rerun. [2]
- The two runs differ in several variables at once, which TrueStandard itself concedes. AA ran the top models at high or maximum reasoning while TrueStandard ran every model at low, and TrueStandard says it cannot separate the effect of the definition from the effect of settings. It also notes that AA's prompts cannot be rerun from outside. [1]
- The pushback signal is small and mostly hedging. All of Grok 4.7's 14 moves were retreats from UNSUPPORTED to UNCERTAIN on fabricated claims, never full reversals, and TrueStandard notes that a confidently wrong model that never moves also scores perfectly. Move rates of 0 to 10% on about 90 checks each mean most middle ranks sit inside overlapping intervals. [1]
- TrueStandard's own benchmark has been revised for harness problems. Its benchmark page says an early version was distorted by a 400-token answer cap, and that on 21 October a 3,000-token cap cut off two models, which were rerun at 12,000 tokens. The raw result files it names are not linked from the page, so the per-claim data could not be inspected. [3]
- The AA chart TrueStandard compared against has already moved. TrueStandard used a chart dated 20 October with 38 models and Grok 4.7 at 692; the live trait leaderboard now lists 41 models with Grok 4.7 (Xhigh) lowest and a span starting at 702, with a typical interval of about 25 Elo. The published -0.57 is a snapshot of a moving target. [4][1]

## What it means

Treat 'sycophancy' as at least two properties. AA-PersonalityBench measures whether default single-turn answers read as flattering to an LLM judge; TrueStandard measures whether a one-word verdict survives one sentence of user disagreement at low reasoning effort. The negative correlation is real on the published numbers, but with task, scoring, prompts and reasoning settings all different, it does not show that either benchmark is wrong, and it is a small-sample snapshot.

If your system grades, fact-checks or reviews claims, run your own pushback test on your prompts, at the reasoning setting you will deploy, and count retreats to 'uncertain' separately from full reversals. If it talks to customers, tone-based flattery scores are closer to the risk. Don't pick a model for one use from a leaderboard built for the other.

## Competing readings

- **Construct mismatch: the tests measure different behaviours (stylistic flattery versus verdict stability), so a negative correlation is plausible and neither is wrong** (likely, about 60%). What would tell it apart: AA scores single default answers by pairwise style judgment; TrueStandard scores verdict changes under pushback, and its reported negative correlation reproduces from its published rows [2][1].
- **Confound artifact: the inverse ranking is driven mainly by different reasoning settings and harness problems such as token caps, and would shrink under a matched rerun** (unlikely, about 30%). What would tell it apart: AA ran top models at their highest reasoning settings, and TrueStandard says it cannot separate the effect of definition from the effect of settings [1]; TrueStandard's harness has had token-cap problems that distorted earlier results and forced reruns [3].
- **One benchmark is unreliable, for example AA's unauditable LLM-judged prompts or TrueStandard's immovability metric, so the ranking says little about real sycophancy** (very unlikely, about 10%). What would tell it apart: TrueStandard's metric rewards immovability regardless of correctness, and some moves are retreats to UNCERTAIN rather than reversals; AA's prompts cannot be rerun by outsiders [1][2].

## Base rate

The New Runtime archive holds only 4 earlier research stories about companies, models, organisations or projects from the last 12 months that are at least 3 months old: too few (minimum 5) for a base rate. The archive has too few cases for a base rate: 4 comparable research stories, 3 of them followed by a release, which is below the minimum of 5. So it is not used here.

## What would change this view

- An independent matched rerun at equal reasoning settings shows a positive or near-zero correlation (signal): The inverse ranking was mostly an artifact of settings or the test harness, not a difference in what the tests measure.
- AA publishes a pushback or multi-turn sycophancy trait whose ranking matches TrueStandard's (signal): AA's style-based trait misses answer stability, confirming two separate behaviours.
- TrueStandard's raw runs show most moves across models are hedges to UNCERTAIN rather than reversals (signal): The pushback metric is mostly measuring calibration hedging, weakening the sycophancy framing.

## Still open

- Does the inverse ranking persist when both protocols run at the same reasoning setting on the same model versions?
- How much of TrueStandard's move rate is hedging to UNCERTAIN versus genuine reversal across all models, not just Grok 4.7?
- Would the correlation hold against AA's current 41-model chart rather than the 20 October snapshot?
- Will TrueStandard publish its raw per-claim runs so the move rates can be audited?

## Sources

1. [Why AI Sycophancy Benchmarks Disagree (TrueStandard)](https://truestandard.ai/blog/why-sycophancy-benchmarks-disagree) (primary)
2. [AA-PersonalityBench Methodology (Artificial Analysis)](https://artificialanalysis.ai/methodology/aa-personalitybench)
3. [AI Sycophancy Benchmark: Which Models Fold (TrueStandard)](https://truestandard.ai/ai-sycophancy-benchmark) (primary)
4. [Sycophantic: AA-PersonalityBench Trait Leaderboard (Artificial Analysis)](https://artificialanalysis.ai/evaluations/aa-personalitybench/traits/sycophantic)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Prompts](https://newruntime.com/topics/prompts.md): Explore the prompts topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)