---
type: "post"
id: "an-api-based-probe-reports-a-strong-signal-bcc3eb"
slug: "an-api-based-probe-reports-a-strong-signal-bcc3eb"
title: "An API-based probe reports a strong signal that some recent Claude models learned the order of obscure"
description: "An API-based probe reports a strong signal that some recent Claude models learned the order of obscure protein sequences: their next-amino-acid accuracy drops sharply when the same residues are shuffled."
retrieval_nugget: "An API-based probe reports a strong signal that some recent Claude models learned the order of obscure protein sequences: their next-amino-acid accuracy drops sharply when the same residues are shuffled. The result is a useful demonstration of how to test for possible training-data exposure, but it does not reveal how those sequences entered training."
published_at: "2026-11-19"
updated_at: "2026-11-19"
record_date: "2026-11-19"
date_kind: "published_at"
topics: ["claude","anthropic"]
entities: ["Xinming Tu","LLM-bioseq-probe","Claude Fable 5.2","Claude Opus 5.5","Anthropic","OpenRouter"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "An external probe finds strong evidence that recent Claude models reproduce biological sequence data, unlike the other tested models."
prior_refs: ["https://newruntime.com/posts/biologists-now-acknowledge-using-llms-almost-as-much-4bf81c/"]
source_url: "https://raw.githubusercontent.com/XinmingTu/LLM-bioseq-probe/main/README.md"
source_urls: ["https://raw.githubusercontent.com/XinmingTu/LLM-bioseq-probe/main/README.md"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:an-api-based-probe-reports-a-strong-signal-bcc3eb"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/an-api-based-probe-reports-a-strong-signal-bcc3eb/","markdown":"https://newruntime.com/posts/an-api-based-probe-reports-a-strong-signal-bcc3eb.md","json":"https://newruntime.com/posts/an-api-based-probe-reports-a-strong-signal-bcc3eb.json"}
markdown_url: "https://newruntime.com/posts/an-api-based-probe-reports-a-strong-signal-bcc3eb.md"
---

# An API-based probe reports a strong signal that some recent Claude models learned the order of obscure

## Retrieval answer

An API-based probe reports a strong signal that some recent Claude models learned the order of obscure protein sequences: their next-amino-acid accuracy drops sharply when the same residues are shuffled. The result is a useful demonstration of how to test for possible training-data exposure, but it does not reveal how those sequences entered training.

## Summary
An API-based probe reports a strong signal that some recent Claude models learned the order of obscure protein sequences: their next-amino-acid accuracy drops sharply when the same residues are shuffled. The result is a useful demonstration of how to test for possible training-data exposure, but it does not reveal how those sequences entered training.

## What happened
The LLM-bioseq-probe repository describes an outside-in test: give a model the beginning of a protein and ask it to predict the next amino acids. Its main test set contains 668 bacterial and archaeal proteins from 167 Pfam families classified as domains of unknown function. The README characterizes these proteins as having no solved structure, almost no papers, and no other UniProt sequence within 90% identity. The intent is to reduce the chance that a model can answer from general biological knowledge or close known examples.

For each protein, the authors also asked the model to continue a version with the residues shuffled. Shuffling preserves composition but disrupts order, so the difference in next-amino-acid accuracy measures sensitivity to the sequence’s actual order. The README reports a 45.1-point gap for Claude Fable 5.2, accessed through OpenRouter: 54.1% accuracy on real sequences versus 9.0% on shuffled ones (95% confidence interval for the gap: 42.5 to 47.6). Claude Opus 5.5 scored 34.9% versus 7.5%, a 27.4-point gap (24.7 to 30.0).

The listed gaps are smaller for earlier Claude versions: +14.9 points for Fable 5.1, +9.7 for Opus 5, +7.7 for Opus 4.8, +8.9 for Opus 4.7, and between +0.6 and +2.9 for Opus 4.5 and 4.6. The README says GPT and Gemini models stayed near zero on this test, as did seven open-weight models on raw next-token continuation. Results also rose with the number of sequences in a protein family. After adding 90 smaller families, Opus 5.5’s gap ranged from +8 to +37 points, while Fable 5.2’s rose from +20 and levelled near +45 at about 300 sequences per family. The repository reports that the gap was also present in 128 families with zero papers.

The repository also compares the two Claude results with open protein language models under the same input condition. It reports a +32.5-point gap for ProGen3-3B, +27.7 for ProtGPT3-10B, +13.0 for ESM C 600M, +11.6 for ProtGPT3-1.3B, +10.2 for ProtGPT2, +8.5 for ESM3-open, and +7.0 for ESM-2 650M. On this measure, Fable 5.2 exceeded every listed open protein model, while Opus 5.5 was level with ProtGPT3-10B. The README says ProtGPT3-10B followed the same family-size curve, which the authors present as a fingerprint expected when a system has learned from sequence data.

The README describes further checks: test-time thinking tokens and prompt format did not explain the result. It reports a distinct DNA result for Opus 5.5: exact continuations in 25 of 500 human coding-sequence windows, but not for promoters, introns, intergenic DNA, or the bacterial genes in the protein test. The repository includes code, test sets, results and model outputs under an MIT license. Its CLI can run a probe through Anthropic, OpenAI, Gemini or Fireworks APIs; a full run asks 6,680 questions and costs roughly $1–20 depending on model pricing and reasoning.

## What the archive adds
New Runtime's earlier coverage examined how data-mixture selection might become a causal estimation problem in [Data Mixing, Minus the Guesswork](https://newruntime.com/posts/data-mixing-minus-the-guesswork-analysis-9fc7d6/) (2026-10-21). This probe is a new event: it tests for sequence-order signals in deployed models, not how training mixtures should be selected.

## Trend signal
This does not yet fit a pattern New Runtime tracks. The repository describes a focused empirical test of possible sequence exposure, rather than a demonstrated change in agent systems, interfaces or deployment practice.

## Why it matters
Teams evaluating models for scientific or sensitive-data work need to distinguish capability from provenance. Strong continuation of a sequence absent from ordinary reference material may be a clue that related data appeared in training, and a shuffled control offers a more informative check than asking whether a model can recite a famous protein. The repository’s approach is actionable as an evaluation design: select obscure examples, compare real inputs with composition-preserving controls, and report uncertainty intervals rather than treating a successful completion as proof of memorization.

That still does not settle whether a particular model is safe to use with proprietary or sensitive sequences. A positive signal does not identify the training source or phase, and a negative result on a limited test cannot establish absence. Builders should treat this as a prompt to demand clearer provenance and broader testing, not as a universal verdict on a model family.

For a team making model-selection or data-governance decisions, the relevant operational question is narrower: can a repeatable probe detect unexpected overlap before a model is used in a sensitive workflow? The public CLI, controls and retained outputs make the method inspectable rather than a one-off demonstration. It can complement contractual assurances and model cards, but cannot replace them.

## Limits and uncertainty
The repository’s README says the experiment cannot show whether the sequences came from a deliberate training dataset or crawled database pages, nor at which training phase they entered. The reported comparisons also cover only the listed models and test sets; they do not establish that other models never saw such sequences. The README reports confidence intervals for the headline gaps, but the results have not been independently checked here. The DNA finding is limited to the tested human and mouse coding-sequence windows and the specific continuation task.

## Reader takeaway
The useful result is the test design as much as the headline number: compare continuation of obscure real sequences with shuffled controls, and interpret a gap as evidence of learned sequence order—not proof of where the data came from. Model builders should ask for provenance and treat this kind of probe as one bounded evaluation, not a complete audit.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Anthropic](https://newruntime.com/topics/anthropic.md): Explore the anthropic topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
