---
type: "post"
id: "stanford-kaust-ut-austin-and-mit-release-muses-010c3e"
slug: "stanford-kaust-ut-austin-and-mit-release-muses-010c3e"
title: "Stanford, KAUST, UT Austin and MIT release Muses-Bench, the first systematic multi-user LLM agent benchmark."
description: "Kordi-AI’s public repository releases the code for “Multi-User Large Language Model Agents,” a benchmark and framework for evaluating agents that serve people with different roles, permissions, and preferences."
retrieval_nugget: "Kordi-AI’s public repository releases the code for “Multi-User Large Language Model Agents,” a benchmark and framework for evaluating agents that serve people with different roles, permissions, and preferences. Its four scenarios make multi-user behavior a testable engineering concern, from access control to shared inference queues, though the repository excerpt does not report model scores."
published_at: "2026-11-07"
updated_at: "2026-11-07"
record_date: "2026-11-07"
date_kind: "published_at"
topics: ["anthropic"]
entities: ["Kordi-AI","Muses-Bench","Multi-User Large Language Model Agents","vLLM","LiteLLM","OpenAI","Anthropic"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "A Stanford-led collaboration with KAUST, UT Austin and MIT released Muses-Bench, a systematic benchmark focused specifically on multi-user LLM agents."
prior_refs: ["https://newruntime.com/posts/stanford-collaborative-intelligence-benchmark-measures-ai-agents-as-33e6c6/"]
source_url: "https://github.com/Kordi-Lab/Multi-User-LLM-Agent"
source_urls: ["https://github.com/Kordi-Lab/Multi-User-LLM-Agent"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:stanford-kaust-ut-austin-and-mit-release-muses-010c3e"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/stanford-kaust-ut-austin-and-mit-release-muses-010c3e/","markdown":"https://newruntime.com/posts/stanford-kaust-ut-austin-and-mit-release-muses-010c3e.md","json":"https://newruntime.com/posts/stanford-kaust-ut-austin-and-mit-release-muses-010c3e.json"}
markdown_url: "https://newruntime.com/posts/stanford-kaust-ut-austin-and-mit-release-muses-010c3e.md"
---

# Stanford, KAUST, UT Austin and MIT release Muses-Bench, the first systematic multi-user LLM agent benchmark.

## Retrieval answer

Kordi-AI’s public repository releases the code for “Multi-User Large Language Model Agents,” a benchmark and framework for evaluating agents that serve people with different roles, permissions, and preferences. Its four scenarios make multi-user behavior a testable engineering concern, from access control to shared inference queues, though the repository excerpt does not report model scores.

## Summary

Kordi-AI’s public repository releases the code for “Multi-User Large Language Model Agents,” a benchmark and framework for evaluating agents that serve people with different roles, permissions, and preferences. Its four scenarios make multi-user behavior a testable engineering concern, from access control to shared inference queues, though the repository excerpt does not report model scores.

## What happened

The GitHub repository identifies itself as the official code for the paper “Multi-User Large Language Model Agents.” Its README frames the work around a gap in prior agent evaluation: many environments assume one user with one utility function, while real deployments may involve several users with differing roles, permissions, and preferences. The project provides a formal framework and benchmark for evaluating these interactions. The repository excerpt does not establish that this is the first benchmark of its kind.

The benchmark covers four capabilities. “Privacy & Access Control” uses secure credential management to test permission enforcement, privacy-aware summarization, and resistance to social engineering. “Sequential Coordination” centers on meeting scheduling and tests preference elicitation, conflict resolution, and scalable context management. “Resource Optimization” examines a shared LLM inference queue through fairness, queueing efficiency, and incentive compatibility. “Instruction Following” tests whether an agent can follow per-user constraints in settings with multiple stakeholders.

The README says test data is organized by scenario under `data/scenarios/`, with ready-to-use JSONL files and data-builder scripts for generating custom datasets. It documents running a scenario through `run.py`, with a model and provider specified as command-line arguments. Its example uses `gpt-4o-mini` with the OpenAI provider; configuration instructions also describe credentials for Anthropic and providers supported by LiteLLM. For trained models, the documented evaluation path uses the vLLM provider and supports an optional LoRA path.

The repository includes a training pipeline: scripts to generate seed scenarios, generate conversations using a teacher model, aggregate data, and train. The README describes submission through a batch script or direct invocation of the training script. It lists an Apache 2.0 license and links the associated paper as arXiv:2604.08567. No benchmark scores, model rankings, or measured improvements appear in the repository excerpt, so the release offers a way to run evaluations rather than evidence here of which systems perform well.

## What the archive adds

New Runtime’s earlier coverage of the Stanford Collaborative Intelligence Benchmark described a 240-task evaluation of agents as teammates, including weaknesses under interpersonal pressure and when taking over work from a weaker model ([earlier coverage, 2026-11-07](https://newruntime.com/posts/stanford-collaborative-intelligence-benchmark-measures-ai-agents-as-33e6c6/)). Muses-Bench is a new benchmark release, not a restatement: it focuses on multi-user permissions, coordination, shared resources, and instructions rather than teamwork tasks generally.

## Trend signal

This does not yet fit a pattern New Runtime tracks. The repository makes multi-user evaluation concrete, but its description alone does not establish a durable shift in how agent systems are built or deployed. In particular, a benchmark that tests access control is not evidence by itself that production security boundaries are moving to agent runtimes.

## Why now

The project’s motivation is the difference between single-user evaluation assumptions and deployments in which users have different roles and preferences. Its four scenarios turn that stated mismatch into testable cases. The repository does not explain a release timetable or identify a particular product launch that prompted publication.

## Why it matters

Teams building shared assistants often need to answer questions that single-user demos leave unresolved: whose instruction should govern a conflict, what information one person may see, and how requests compete for shared capacity. Muses-Bench gives those concerns separate evaluation scenarios instead of treating “multi-user support” as one vague capability. The supplied run path, data builders, and training scripts also make the repository a possible starting point for a team that wants to adapt test data to its own workflow.

The distinction between a test harness and a result matters. The repository description does not provide scores or show that any model reliably enforces permissions, coordinates preferences, or allocates a queue fairly. Builders can use the scenarios to structure their own evaluation, but should not treat the benchmark’s existence—or a future score on it—as a substitute for checking their own roles, data boundaries, and failure costs.

## Limits and uncertainty

The repository excerpt does not report model-level results, baseline comparisons, or measurement methodology beyond describing the capabilities each scenario is intended to test. It therefore does not establish how difficult the tasks are, whether scores predict behavior in deployed systems, or which model performs best. The README links to an arXiv paper, but the claims and results of that paper are not independently assessed here. Teams should inspect the paper and datasets before relying on the benchmark for purchasing or deployment decisions.

## Skill to build now

Choose one real workflow with more than one user—such as scheduling a meeting or sharing an assistant—and write down each person’s permissions, private information, and instruction priority. Then map those requirements to the repository’s closest scenario and adapt its data-builder approach to create cases that include both ordinary requests and conflicting or unauthorized ones. Keep an explicit pass/fail check for what each user may see; do not infer safety from a successful run alone.

## Reader takeaway

Muses-Bench gives builders an open, runnable starting point for testing agents in multi-user situations that ordinary single-user evaluations can miss. Its value today is the scenarios and tooling, not proven model rankings: the repository description supplies no scores. Treat it as a way to make permissions, coordination, and shared-resource behavior explicit in your own tests.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Anthropic](https://newruntime.com/topics/anthropic.md): Explore the anthropic topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)