---
type: "post"
id: "built-on-rumodernbert-base-with-three-stage-training-9166f4"
slug: "built-on-rumodernbert-base-with-three-stage-training-9166f4"
title: "Built on RuModernBERT-base with three-stage training: RetroMAE, weakly supervised, then supervised fine-tuning."
description: "DeepVK’s RuModernBERT-base is a 150-million-parameter, encoder-only ModernBERT variant trained on roughly two trillion Russian, English, and code tokens."
retrieval_nugget: "DeepVK’s RuModernBERT-base is a 150-million-parameter, encoder-only ModernBERT variant trained on roughly two trillion Russian, English, and code tokens. For teams building Russian-language retrieval, classification, or entity-extraction systems, its long 8,192-token context and a tokenizer patch make it a concrete model to evaluate rather than a general-purpose chat-model replacement."
published_at: "2026-10-27"
updated_at: "2026-10-27"
record_date: "2026-10-27"
date_kind: "published_at"
topics: ["models","open-models"]
entities: ["DeepVK","RuModernBERT-base","RuModernBERT-small","ModernBERT","Hugging Face","Russian Super Glue","RuMTEB","Encodechka"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "Introduces RuModernBERT-base and its three-stage training method; none of the prior documents cover this model."
prior_refs: []
source_url: "https://huggingface.co/deepvk/RuModernBERT-base"
source_urls: ["https://huggingface.co/deepvk/RuModernBERT-base"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:built-on-rumodernbert-base-with-three-stage-training-9166f4"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/built-on-rumodernbert-base-with-three-stage-training-9166f4/","markdown":"https://newruntime.com/posts/built-on-rumodernbert-base-with-three-stage-training-9166f4.md","json":"https://newruntime.com/posts/built-on-rumodernbert-base-with-three-stage-training-9166f4.json"}
---

# Built on RuModernBERT-base with three-stage training: RetroMAE, weakly supervised, then supervised fine-tuning.

## Retrieval answer

DeepVK’s RuModernBERT-base is a 150-million-parameter, encoder-only ModernBERT variant trained on roughly two trillion Russian, English, and code tokens. For teams building Russian-language retrieval, classification, or entity-extraction systems, its long 8,192-token context and a tokenizer patch make it a concrete model to evaluate rather than a general-purpose chat-model replacement.

## Summary

DeepVK’s RuModernBERT-base is a 150-million-parameter, encoder-only ModernBERT variant trained on roughly two trillion Russian, English, and code tokens. For teams building Russian-language retrieval, classification, or entity-extraction systems, its long 8,192-token context and a tokenizer patch make it a concrete model to evaluate rather than a general-purpose chat-model replacement.

## What happened

DeepVK has published RuModernBERT-base on Hugging Face as a Russian version of the modernized bidirectional, encoder-only Transformer architecture ModernBERT. The listed base model has 150 million parameters, a hidden dimension of 768, 22 layers, a vocabulary of 50,368 tokens, and an 8,192-token context length. The repository also identifies a smaller RuModernBERT-small variant with 35 million parameters, 12 layers, and a hidden dimension of 384. Both are presented for masked-language-modeling use rather than text generation.

The model card says RuModernBERT was pre-trained on approximately two trillion tokens spanning Russian, English, and code. The stated sources include internet material, books, scientific sources, and social media. Training had three named phases: massive pre-training, context extension, and cooldown. Stage one used 1.7 trillion tokens with a context length of 1,024. Stages two and three used 250 billion and 50 billion tokens respectively, while extending the context length to 8,192. The card says these later stages used cleaner sources than the first stage. FineWeb Russian and English data appears in stage one only, while CulturaX-Ru-Edu appears in stage two only; Wikipedia, ArXiv, books, code, StackExchange, and Russian social data appear across the listed stages according to the table.

A new tokenizer was trained using the original configuration, retaining vocabulary size and the same special tokens. Its training mixture was Russian and English FineWeb data. DeepVK also supplies a `patched-tokenizer` revision. The stated motivation is that several lowercase Russian letters were split into multiple subword tokens, a behavior the authors flag as problematic for tasks such as named-entity recognition, where the first token of a word should have semantic meaning. The patch makes six common lowercase letters single tokens: a, e, i, n, o, and t in Russian transliteration. It uses unused vocabulary slots, adds BPE merges, and initializes each new token from its uppercase counterpart in both the token embedding and language-model head, with small Gaussian noise.

The card reports an evaluation of the patched version on 20 RuMTEB tasks and says it found no statistically significant difference from the original model. Still, it recommends the patched revision for work sensitive to tokenization granularity, including NER. For the reported Russian Super Glue development evaluation, RuModernBERT-base has an aggregate score of 0.737, compared with 0.683 for RuModernBERT-small and 0.678 for the listed ai-forever Russian BERT base comparator. On the displayed Encodechka table, the base model’s average score is 0.673 and its average S+W score is 0.611. The source frames these comparisons as comparisons with raw encoders, not models trained specifically for retrieval or sentence embeddings.

The repository is Apache-2.0 licensed and gives standard Transformers loading examples for masked-token prediction. It also documents use with FlashAttention 2 where the hardware supports it. The model page lists 11,110 downloads in the prior month, but does not make that figure a performance or adoption claim.

## What the archive adds

We have not covered this before; it opens a new thread.

## Trend signal

No supplied pattern genuinely fits this release closely enough to support a trend claim.

## Why it matters

Russian-language AI products often need an encoder alongside a generative model: to classify incoming requests, retrieve documents, rank candidates, detect entities, or create compact representations for downstream systems. RuModernBERT-base offers a specific starting point for those workloads with a documented multilingual training mix, long-context configuration, and an openly described tokenizer intervention.

The tokenizer revision is the operationally interesting part. Teams doing entity extraction should not assume that a model’s aggregate benchmark score reveals whether its segmentation behavior is appropriate for their labels and text. The source says the patch did not produce statistically significant differences across its 20-task RuMTEB evaluation, yet still recommends it where token-level granularity matters. That makes revision selection and task-level testing part of integration, not a cosmetic implementation detail.

## What happens next

- **More downstream Russian NLP fine-tunes will appear around the base model.** (6 months, 55%). Watch for: newly listed adapters or fine-tunes on the model repository.
- **Teams evaluating it for entity extraction will compare the original and patched tokenizer revisions on labeled data.** (3 months, 65%). Watch for: published task-specific NER evaluations that isolate tokenizer effects.

## Limits and uncertainty

The source is a model card, not an independent reproduction or a deployment study. Its benchmark figures are reported by the model publisher, and the card does not provide latency, throughput, hardware cost, safety behavior, production reliability, or results on a reader’s own domain. The comparisons shown are limited to the named benchmark tables and their stated encoder-only comparison policy. The reported lack of statistically significant change for the patched tokenizer does not establish that either revision will improve a particular NER dataset. The source also does not establish that the model is preferable to a retrieval-tuned encoder for search or embedding workloads.

## Skill to build now

Create a small Russian-language evaluation set from real work this week: include entity-bearing text, long documents, and the labels or ranking judgments your product uses. Run the original and `patched-tokenizer` revisions through the same pipeline, then inspect token boundaries as well as final task metrics. If the task is NER, measure span-level precision and recall rather than relying only on a model-wide benchmark average. For retrieval or classification, compare this raw encoder with the task-specific baselines already used by the team.

## Reader takeaway

RuModernBERT-base is a long-context Russian/English raw encoder with an Apache-2.0 license and a documented tokenizer patch for Russian token boundaries. Its reported benchmarks justify an evaluation, not a blanket replacement decision: select the tokenizer revision and the model only after testing the actual downstream task.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- topic hub: [Open Models](https://newruntime.com/topics/open-models.md): Explore the open-models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)