Field noteOct 24, 20261 min read1 source

Release candidate v1.0.0-rc.3 · huggingface/tokenizers

In one minute

Hugging Face has released Tokenizers v1.0.0-rc.3, a pre-release centered on new WebAssembly bindings.

What changed: A new tokenizers v1.0.0-rc.3 release adds WebAssembly bindings and fixes.

The bindings are published on npm as tokenizers-wasm; the project says release candidates use the next dist-tag, with installation via npm install tokenizers-wasm@next.

The release also adds Python support for encoding pairs of texts. Its fixes cover byte-level BPE vocabularies missing bytes that cannot occur in UTF-8, including a cited ModernBERT case; WordPiece vocabularies containing an empty-string token; and tokenizer.json files using regex Split pre-tokenizers or Replace normalizers, which had been rejected by Python, Node, and WASM bindings.

Other changes make a 64-bit vocabulary-hash collision an error rather than silently dropping a token, remove an unnecessary Rust/Python encoding copy, and document feature selection through the umbrella Rust crate. The release notes say feature parity with version 0.23 remains the main milestone before stable 1.0.0. They do not establish a stable-release date, browser performance figures, or compatibility guarantees beyond the listed fixes.

Evidence / sources

  1. 1github.com/huggingface/tokenizers/releases/tag/v1.0.0-rc.3
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicDeveloper ToolsExplore the developer-tools topic hub.
  2. 02topicOpen SourceExplore the open-source topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04source ledgerSource LedgerInspect the public source evidence graph.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract