---
type: "post"
id: "code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c"
slug: "code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c"
title: "Code at github.com/klimzaporojets/emerge, dataset on Hugging Face."
description: "EMERGE is a benchmark for measuring how systems update knowledge graphs when new textual evidence arrives."
retrieval_nugget: "EMERGE is a benchmark for measuring how systems update knowledge graphs when new textual evidence arrives. Its GitHub repository releases a 3,500-instance test set, predictions from 13 benchmark models, and a reproducible evaluation pipeline, giving researchers a concrete way to test more than one-shot information extraction."
published_at: "2026-11-11"
updated_at: "2026-11-11"
record_date: "2026-11-11"
date_kind: "published_at"
topics: ["github"]
entities: ["EMERGE","Wikidata","Wikipedia","GPT-5.1","Hugging Face"]
related_patterns: ["scientific-agents-need-artifact-evidence"]
novelty_label: "new_fact"
what_changed: "The EMERGE repository shows a recent commit deprecating E-R in G_t."
prior_refs: []
source_url: "https://github.com/klimzaporojets/emerge"
source_urls: ["https://github.com/klimzaporojets/emerge"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c/","markdown":"https://newruntime.com/posts/code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c.md","json":"https://newruntime.com/posts/code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c.json"}
markdown_url: "https://newruntime.com/posts/code-at-github-com-klimzaporojets-emerge-dataset-on-206c7c.md"
---

# Code at github.com/klimzaporojets/emerge, dataset on Hugging Face.

## Retrieval answer

EMERGE is a benchmark for measuring how systems update knowledge graphs when new textual evidence arrives. Its GitHub repository releases a 3,500-instance test set, predictions from 13 benchmark models, and a reproducible evaluation pipeline, giving researchers a concrete way to test more than one-shot information extraction.

## Summary
EMERGE is a benchmark for measuring how systems update knowledge graphs when new textual evidence arrives. Its GitHub repository releases a 3,500-instance test set, predictions from 13 benchmark models, and a reproducible evaluation pipeline, giving researchers a concrete way to test more than one-shot information extraction.

## What happened
The EMERGE GitHub repository describes a dataset and evaluation framework for text-driven knowledge graph updating, or TKGU. The task is to update a knowledge graph from textual evidence rather than merely extract a fact from a passage. Each benchmark instance pairs a textual passage, a snapshot of a knowledge graph, and the update operations that the passage induces.

The project defines five update operations. Exists denotes a triple that is already in the graph and supported by the passage. Add is a new triple between entities already represented in the graph. Mint+Add covers a triple involving one or more entities not yet represented. Infer is a triple linking a newly introduced entity to an existing graph entity that is not explicitly stated in the passage. Deprecate represents an existing triple invalidated by updated information in the passage. The repository notes that its implementation uses different internal identifiers for those operation categories.

The released test set contains 3,500 instances drawn from seven annual Wikidata snapshots spanning 2019 to 2025. According to the repository, every instance includes a Wikipedia passage with temporal context, ground-truth TKGU triples with LLM and human assessments, and predictions from 13 benchmark models. The listed models include EDC+, KGGen, RAKG, REBEL, and ReLiK variants. EMERGE also includes a KG-aware baseline using GPT-5.1 with access to the knowledge-graph snapshot. Its stated purpose is to test whether supplying graph structure closes gaps left by information-extraction-only approaches.

The project provides a larger corpus reported as 1.19 million update operations: 608,000 Exists operations, 207,000 Add operations, 149,000 Mint+Add operations, 220,000 Infer operations, and 9,500 Deprecate operations. The construction pipeline combines Wikidata and Wikipedia histories, LLM annotation, iterative refinement of suspected bad annotations, deduplication, test-set sampling, and human annotation on a subset. The repository calls the evaluation set a frozen final artifact, so users can download it and reproduce the evaluation without rebuilding the full corpus.

The quick-start path uses Python 3.12, PyTorch, and the project’s core requirements, followed by a roughly 155 MB dataset download. The repository lists one GPU for evaluation, at least 16 GB of VRAM, and 32 GB of RAM as baseline requirements. One ReLiK evaluation path that loads full graph snapshots can require 180 GB or more of RAM. Optional downloads add relation indexes, a 2.3 GB corpus, and 3.8 GB of compressed graph snapshots. Recreating the corpus from the beginning requires large raw Wikidata and Wikipedia history dumps.

## What the archive adds
New Runtime's earlier coverage of EMERGE’s code, data, and pre-generated outputs described the benchmark’s release and EMNLP 2026 acceptance ([earlier note, November 3, 2026](https://newruntime.com/posts/code-data-and-pre-generated-outputs-available-at-ee47e8/)). This is a new fact on that known story: the repository documents a KG-aware GPT-5.1 baseline and substantially more detailed reproduction paths.

## Trend signal
This supports the idea that useful progress in scientific-agent evaluation depends on inspectable artifacts and reproducible evidence, rather than headline scores alone. The closest New Runtime pattern is [scientific agents need artifact evidence](https://newruntime.com/patterns/scientific-agents-need-artifact-evidence/), although EMERGE evaluates knowledge-graph updating rather than an end-to-end scientific-agent workflow.

## Why now
The repository identifies a v1.0.1 release with a KG-aware GPT-5.1 baseline and a camera-ready evaluation configuration. Its commit history also records changes through November 2026. That makes this relevant as an updated evaluation artifact, rather than simply a restatement of the original benchmark release.

## Why it matters
Teams maintaining knowledge graphs from changing documents face a different problem from extracting isolated facts. A system must distinguish between confirming what is already known, adding a relation between known entities, creating an entity, drawing an allowed inference, and retiring an outdated claim. A single extraction score can blur those failure modes.

EMERGE gives researchers a time-indexed test set and a specific taxonomy for testing those decisions. The included baseline predictions lower the cost of initial comparison, while the graph-conditioned GPT-5.1 baseline makes one design question explicit: whether a model performs differently when it can see the current graph state. That is a useful question for teams considering retrieval, graph context, or update queues around an LLM.

It remains a research and evaluation artifact, not a production deployment recipe. But its task definition can help builders make their own tests more realistic. A team can ask whether its pipeline preserves valid existing facts, introduces duplicates, invents entities, fails to invalidate stale relations, or relies on an inference the underlying text did not actually warrant.

## Limits and uncertainty
The repository’s dataset counts, annotation process, model results, and hardware requirements are project claims and have not been independently audited here. The test set uses Wikipedia passages and Wikidata snapshots, so benchmark performance may not transfer to private enterprise graphs, documents with different editorial standards, or live streams of conflicting updates. The scope of the human assessment is described only at a high level in the repository excerpt. The benchmark also cannot establish that a high score produces a safer or more accurate production knowledge base.

## Skill to build now
Take one document stream your system already uses and label a small sample with EMERGE-like update categories: confirmed existing fact, new relation, new entity, inferred link, or invalidated fact. Run the current pipeline against those labels and inspect errors by category. If you maintain a graph from changing text, do this before optimizing extraction accuracy alone.

## Reader takeaway
EMERGE turns knowledge-graph maintenance into a time-aware, reproducible benchmark with explicit update operations and runnable baselines. Its value is not a deployment claim; it is a sharper way to test whether a system can keep structured knowledge current as the underlying text changes.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)