Publications | Xinyue Zheng

UniCode: Augmenting Evaluation for Code Reasoning

Wed, 01 Jul 2026 00:00:00 +0000

ICML 2026 · · ·

UniCode is a generative evaluation framework for code reasoning. Instead of asking whether a model can solve a familiar static benchmark, UniCode asks whether the model can still reason when the problem is structurally rewritten, recomposed, scaled, and tested against robust corner cases.

Background: the evaluation paradox

Modern code LLMs score impressively on standard benchmarks, yet often fail when requirements shift in realistic coding settings. This is the evaluation paradox: high benchmark accuracy does not necessarily mean robust reasoning. Static benchmarks can be saturated, contaminated, or solved through familiar templates and statistical shortcuts rather than genuine algorithmic adaptation.

UniCode addresses this by moving from static evaluation to generative evaluation. Instead of repeatedly testing a fixed set of problems, UniCode expands a dynamic problem space by transforming high-quality seed tasks into new variants that preserve rigor while changing the reasoning path required to solve them.

Why existing benchmarks are still insufficient

Many recent benchmark variants improve coverage, but a large portion of them still rely on shallow perturbations—for example variable renaming, story rewriting, or surface-level rephrasing. As we argue in the paper, these settings often focus on superficial variations rather than changing the underlying reasoning logic.

That means a model can appear robust simply because the old solution template still works. But in realistic coding, the challenge is not just reading a different story — it is adapting when the constraint changes, when the complexity budget changes, or when multiple algorithmic ideas must be composed together.

UniCode is designed to test exactly this deeper form of reasoning.

Methodology

UniCode has three main components.

1. Multi-dimensional task augmentation

UniCode starts from high-quality seed programming problems and generates new tasks along five reasoning axes:

Axis	What changes	What it probes
Narrative perturbation	Theme, variable names, or irrelevant context	Whether the model is biased by surface tokens
Rule modification	Operational rules or boundary conditions	Whether the model follows new constraints instead of a memorized solution
Efficiency scaling	Input scale and computational budget	Whether the model switches to a more efficient algorithm when needed
Sequential composition	Multiple algorithmic steps are chained together	Whether the model can maintain long reasoning chains
Concept fusion	Multiple algorithmic concepts are merged into one task	Whether the model can perform combinatorial generalization

These augmentations intentionally change the reasoning graph, not just the wording.

2. Stress-driven test generation

For newly generated tasks, human-written reference solutions do not scale. UniCode therefore constructs reliable test suites through a stress-driven pipeline.

The pipeline combines random cases, adversarial cases, and corner cases. For ground-truth outputs, UniCode uses brute-force solvers on small inputs, filters optimized candidates through trusted stress tests, uses majority voting on larger inputs, and finally applies LLM adjudication when solver outputs disagree.

3. Fine-grained diagnosis

A single pass/fail score hides why a model fails. UniCode therefore categorizes failures into modeling errors, logic bugs, complexity errors, indexing/cache bugs, and other implementation mistakes. This turns the benchmark from a simple leaderboard into a diagnostic tool for reasoning analysis.

Experimental results

UniCode evaluates 19 LLMs across reasoning-focused, general-purpose, and open-source families. The benchmark is both challenging and discriminative: even the top model reaches only 70.3% average Pass@1, and the average performance drop from seed problems to UniCode variants is 31.2%.

Main takeaways

Large performance collapse: strong performance on seed benchmarks does not reliably transfer to structurally augmented variants.
Hard split is still very difficult: several models drop to 0.0% on the hard subset.
Narrative shifts are easier than structural shifts: models are relatively robust to surface changes, but often fail when the logic, scale, or composition changes.
Reliable automatic evaluation: the stress-driven pipeline achieves 94.5% correctness and 86.0% coverage in the test-generation quality study.

Where do models fail?

UniCode reveals that model failures are structured, not random. The paper highlights three key findings:

1. Large variance across reasoning dimensions

Aggregate benchmark scores hide substantial within-model variance. Even strong models can perform well on one reasoning axis and collapse on another. This means a single average score often masks important blind spots.

2. Structural changes are much harder than surface changes

Models are comparatively robust to narrative perturbations, but they deteriorate sharply once the underlying reasoning graph changes. The hardest settings are typically efficiency scaling, sequential composition, and concept fusion, which require the model to adapt its strategy rather than reuse a familiar template.

3. Seed-problem regression

The most revealing phenomenon is seed-problem regression: models frequently fall back to the logic of the original seed task even when the augmented problem invalidates that logic. In other words, the model recognizes the old template, but fails to integrate the new constraints.

Beyond accuracy, UniCode also shows that the dominant error types are not small implementation mistakes. Instead, the main failures are modeling errors and complexity errors, revealing weaknesses in conceptualization, algorithm selection, and computational scaling.

Figure 5 provides concrete case studies of these failures. The examples show that models often reuse a memorized modeling paradigm or a low-efficiency solution that was valid for the seed task, but becomes wrong once new structure or new constraints are introduced.

Together, these results suggest that current code LLMs are often strong at syntax and familiar templates, but still fragile in zero-shot adaptation to novel algorithmic requirements.

Resources

Paper:
Code:
Dataset:

from datasets import load_dataset

dataset = load_dataset("grandsmile/Generative_Coding_Dataset")

Citation

@article{zheng2026unicode,
 title = {UniCode: Augmenting Evaluation for Code Reasoning},
 author = {Zheng, Xinyue and Lin, Haowei and Cai, Shaofei and Zheng, Zilong and Yang, Yaodong and Liang, Yitao},
 journal = {arXiv preprint arXiv:2510.17868},
 year = {2026}
}

MCU: An Evaluation Framework for Open-Ended Game Agents

Thu, 03 Jul 2025 00:00:00 +0000

MineStudio: A Streamlined Package for Minecraft AI Agent Development

Thu, 01 May 2025 00:00:00 +0000

MineStudio is an open-source package for Minecraft AI agent development. It lowers the engineering barrier for embodied policy research by unifying environment customization, trajectory data processing, policy modeling, offline pretraining, online finetuning, distributed inference, and benchmark evaluation.

Authors: Shaofei Cai*, Zhancun Mu*, Kaichen He, Bowei Zhang, Xinyue Zheng, Anji Liu, Yitao Liang
Equal contribution

· · ·

Abstract

Minecraft is a valuable testbed for embodied intelligence and sequential decision-making, but developing and validating new Minecraft agents often requires substantial engineering work. MineStudio provides a streamlined, open-source software package that helps researchers build, train, evaluate, and benchmark agents in a unified workflow.

The package integrates seven core components: simulator, data, model, offline pretraining, online finetuning, inference, and benchmark. With this modular design, researchers can focus more on algorithmic innovation and less on reimplementing infrastructure.

Introduction

Minecraft offers a rich open-world environment: agents need to perceive high-dimensional observations, plan over long horizons, interact with objects, collect resources, craft tools, and complete diverse tasks. This makes it an ideal platform for studying embodied AI, reinforcement learning, imitation learning, and open-ended agent evaluation.

However, the Minecraft AI ecosystem has long suffered from fragmented engineering pipelines. Different works often customize environments, datasets, controllers, models, and evaluation protocols in incompatible ways. This makes it hard to reproduce results, compare agents fairly, or quickly test a new algorithmic idea.

MineStudio addresses this problem by providing a coherent development stack for Minecraft agents:

Simulator: a customizable Minecraft simulator wrapper with callback-based environment configuration.
Data: an efficient trajectory data structure for storing and retrieving long Minecraft demonstrations.
Models: a unified policy-model template and a gallery of baseline Minecraft agents.
Offline training: a practical pipeline for pretraining agents from offline gameplay data.
Online training: distributed reinforcement learning support with memory-based policies and crash recovery.
Inference: a parallel and distributed pipeline for running and evaluating agents at scale.
Benchmark: automated batch testing for diverse Minecraft tasks, connected to open-ended evaluation with MCU.

For my broader research agenda, MineStudio is closely related to my interest in open-ended agent evaluation: instead of evaluating agents only on fixed small task sets, we need infrastructure that can generate, run, and diagnose many diverse tasks in complex environments.

Quick Start

MineStudio requires Python 3.10+ and JDK 8. A minimal installation looks like this:

conda create -n minestudio python=3.10 -y
conda activate minestudio
conda install --channel=conda-forge openjdk=8 -y
pip install MineStudio

Test whether the simulator can launch:

python -m minestudio.simulator.entry

For GPU rendering with VirtualGL:

MINESTUDIO_GPU_RENDER=1 python -m minestudio.simulator.entry

Minimal Usage

Create a Minecraft simulator

from minestudio.simulator import MinecraftSim

sim = MinecraftSim(
 obs_size=(224, 224),
 render_size=(640, 360),
 callbacks=[]
)

obs, info = sim.reset()

Load a pretrained policy and interact with the environment

from minestudio.simulator import MinecraftSim
from minestudio.simulator.callbacks import RecordCallback
from minestudio.models import VPTPolicy

policy = VPTPolicy.from_pretrained(
 "CraftJarvis/MineStudio_VPT.rl_from_early_game_2x"
).to("cuda")
policy.eval()

env = MinecraftSim(
 obs_size=(128, 128),
 callbacks=[RecordCallback(record_path="./output", fps=30, frame_type="pov")]
)

memory = None
obs, info = env.reset()

for _ in range(1200):
 action, memory = policy.get_action(obs, memory, input_shape="*")
 obs, reward, terminated, truncated, info = env.step(action)

env.close()

Install from source

pip install git+https://github.com/CraftJarvis/MineStudio.git

Why MineStudio Matters

MineStudio makes Minecraft agent research easier in three ways:

Less engineering overhead. Researchers can reuse a common simulator, data format, model interface, training pipeline, and evaluation framework.
More reproducible comparisons. Standardized interfaces make it easier to compare agents under shared environments and benchmark protocols.
Better support for open-ended evaluation. The benchmark module enables batch testing over diverse Minecraft tasks, which is important for discovering agent weaknesses in complex open-world settings.

Citation

@article{cai2024minestudio,
 title={MineStudio: A Streamlined Package for Minecraft AI Agent Development},
 author={Cai, Shaofei and Mu, Zhancun and He, Kaichen and Zhang, Bowei and Zheng, Xinyue and Liu, Anji and Liang, Yitao},
 journal={arXiv preprint arXiv:2412.18293},
 year={2024}
}