{"type":"post","id":"hugging-face-tokenizers-v1-0-0-rc-4-f5dd41","slug":"hugging-face-tokenizers-v1-0-0-rc-4-f5dd41","title":"Hugging Face tokenizers v1.0.0-rc.4 restores Python APIs and splits Tokenizer from CompiledTokenizer","description":"Hugging Face’s tokenizers v1.0.0-rc.4 restores Python APIs used by libraries such as Transformers and vLLM, while changing the boundary between editable tokenizer configuration and the compiled object that encodes text.","retrieval_nugget":"Hugging Face’s tokenizers v1.0.0-rc.4 restores Python APIs used by libraries such as Transformers and vLLM, while changing the boundary between editable tokenizer configuration and the compiled object that encodes text. The release also reports performance and memory improvements, but it remains a release candidate, not a stable 1.0 release.","published_at":"2026-11-14","updated_at":"2026-11-14","record_date":"2026-11-14","date_kind":"published_at","topics":["models","inference","open-source"],"entities":["Hugging Face","tokenizers","Transformers","vLLM","PyTorch"],"related_patterns":[],"novelty_label":"new_fact","what_changed":"The v1.0.0-rc.4 release restores Python APIs and separates Tokenizer from CompiledTokenizer.","prior_refs":["https://newruntime.com/posts/release-candidate-v1-0-0-rc-3-huggingface-32da6a/"],"source_url":"https://github.com/huggingface/tokenizers/releases/tag/v1.0.0-rc.4","source_urls":["https://github.com/huggingface/tokenizers/releases/tag/v1.0.0-rc.4"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:hugging-face-tokenizers-v1-0-0-rc-4-f5dd41","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/hugging-face-tokenizers-v1-0-0-rc-4-f5dd41/","markdown":"https://newruntime.com/posts/hugging-face-tokenizers-v1-0-0-rc-4-f5dd41.md","json":"https://newruntime.com/posts/hugging-face-tokenizers-v1-0-0-rc-4-f5dd41.json"},"markdown_url":"https://newruntime.com/posts/hugging-face-tokenizers-v1-0-0-rc-4-f5dd41.md","next_reads":[{"type":"topic","path":"/topics/models/","url":"https://newruntime.com/topics/models/","title":"Models","reason":"Explore the models topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/models.md","json_url":"https://newruntime.com/topics/models.json"},{"type":"topic","path":"/topics/inference/","url":"https://newruntime.com/topics/inference/","title":"Inference","reason":"Explore the inference topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/inference.md","json_url":"https://newruntime.com/topics/inference.json"},{"type":"topic","path":"/topics/open-source/","url":"https://newruntime.com/topics/open-source/","title":"Open Source","reason":"Explore the open-source topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/open-source.md","json_url":"https://newruntime.com/topics/open-source.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/sources/","url":"https://newruntime.com/sources/","title":"Sources","reason":"See the original sources this archive cites.","media_type":"text/html"}]}