UniCode: Augmenting Evaluation for Code Reasoning

Wed, 01 Jul 2026 00:00:00 +0000

ICML 2026 · · ·

UniCode is a generative evaluation framework for code reasoning. Instead of asking whether a model can solve a familiar static benchmark, UniCode asks whether the model can still reason when the problem is structurally rewritten, recomposed, scaled, and tested against robust corner cases.

Background: the evaluation paradox

Modern code LLMs score impressively on standard benchmarks, yet often fail when requirements shift in realistic coding settings. This is the evaluation paradox: high benchmark accuracy does not necessarily mean robust reasoning. Static benchmarks can be saturated, contaminated, or solved through familiar templates and statistical shortcuts rather than genuine algorithmic adaptation.

UniCode addresses this by moving from static evaluation to generative evaluation. Instead of repeatedly testing a fixed set of problems, UniCode expands a dynamic problem space by transforming high-quality seed tasks into new variants that preserve rigor while changing the reasoning path required to solve them.

Why existing benchmarks are still insufficient

Many recent benchmark variants improve coverage, but a large portion of them still rely on shallow perturbations—for example variable renaming, story rewriting, or surface-level rephrasing. As we argue in the paper, these settings often focus on superficial variations rather than changing the underlying reasoning logic.

That means a model can appear robust simply because the old solution template still works. But in realistic coding, the challenge is not just reading a different story — it is adapting when the constraint changes, when the complexity budget changes, or when multiple algorithmic ideas must be composed together.

UniCode is designed to test exactly this deeper form of reasoning.

Methodology

UniCode has three main components.

1. Multi-dimensional task augmentation

UniCode starts from high-quality seed programming problems and generates new tasks along five reasoning axes:

Axis	What changes	What it probes
Narrative perturbation	Theme, variable names, or irrelevant context	Whether the model is biased by surface tokens
Rule modification	Operational rules or boundary conditions	Whether the model follows new constraints instead of a memorized solution
Efficiency scaling	Input scale and computational budget	Whether the model switches to a more efficient algorithm when needed
Sequential composition	Multiple algorithmic steps are chained together	Whether the model can maintain long reasoning chains
Concept fusion	Multiple algorithmic concepts are merged into one task	Whether the model can perform combinatorial generalization

These augmentations intentionally change the reasoning graph, not just the wording.

2. Stress-driven test generation

For newly generated tasks, human-written reference solutions do not scale. UniCode therefore constructs reliable test suites through a stress-driven pipeline.

The pipeline combines random cases, adversarial cases, and corner cases. For ground-truth outputs, UniCode uses brute-force solvers on small inputs, filters optimized candidates through trusted stress tests, uses majority voting on larger inputs, and finally applies LLM adjudication when solver outputs disagree.

3. Fine-grained diagnosis

A single pass/fail score hides why a model fails. UniCode therefore categorizes failures into modeling errors, logic bugs, complexity errors, indexing/cache bugs, and other implementation mistakes. This turns the benchmark from a simple leaderboard into a diagnostic tool for reasoning analysis.

Experimental results

UniCode evaluates 19 LLMs across reasoning-focused, general-purpose, and open-source families. The benchmark is both challenging and discriminative: even the top model reaches only 70.3% average Pass@1, and the average performance drop from seed problems to UniCode variants is 31.2%.

Main takeaways

Large performance collapse: strong performance on seed benchmarks does not reliably transfer to structurally augmented variants.
Hard split is still very difficult: several models drop to 0.0% on the hard subset.
Narrative shifts are easier than structural shifts: models are relatively robust to surface changes, but often fail when the logic, scale, or composition changes.
Reliable automatic evaluation: the stress-driven pipeline achieves 94.5% correctness and 86.0% coverage in the test-generation quality study.

Where do models fail?

UniCode reveals that model failures are structured, not random. The paper highlights three key findings:

1. Large variance across reasoning dimensions

Aggregate benchmark scores hide substantial within-model variance. Even strong models can perform well on one reasoning axis and collapse on another. This means a single average score often masks important blind spots.

2. Structural changes are much harder than surface changes

Models are comparatively robust to narrative perturbations, but they deteriorate sharply once the underlying reasoning graph changes. The hardest settings are typically efficiency scaling, sequential composition, and concept fusion, which require the model to adapt its strategy rather than reuse a familiar template.

3. Seed-problem regression

The most revealing phenomenon is seed-problem regression: models frequently fall back to the logic of the original seed task even when the augmented problem invalidates that logic. In other words, the model recognizes the old template, but fails to integrate the new constraints.

Beyond accuracy, UniCode also shows that the dominant error types are not small implementation mistakes. Instead, the main failures are modeling errors and complexity errors, revealing weaknesses in conceptualization, algorithm selection, and computational scaling.

Figure 5 provides concrete case studies of these failures. The examples show that models often reuse a memorized modeling paradigm or a low-efficiency solution that was valid for the seed task, but becomes wrong once new structure or new constraints are introduced.

Together, these results suggest that current code LLMs are often strong at syntax and familiar templates, but still fragile in zero-shot adaptation to novel algorithmic requirements.

Resources

Paper:
Code:
Dataset:

from datasets import load_dataset

dataset = load_dataset("grandsmile/Generative_Coding_Dataset")

Citation

@article{zheng2026unicode,
 title = {UniCode: Augmenting Evaluation for Code Reasoning},
 author = {Zheng, Xinyue and Lin, Haowei and Cai, Shaofei and Zheng, Zilong and Yang, Yaodong and Liang, Yitao},
 journal = {arXiv preprint arXiv:2510.17868},
 year = {2026}
}

ICML 2026 | Xinyue Zheng