A task-semantic evaluation suite for LeWM

CLEAR-LeWM

Junhan Sun¹ Guofeng Zhang^1,† Hao Zhao^2,†

¹State Key Laboratory of CAD&CG, Zhejiang University | ²Tsinghua University | ^†Corresponding authors

Task completion, not proxy success.

Four corrected task contracts, two calibrated protocols, and complete provenance for measuring world-model control beyond permissive endpoint proxies.

Explore the protocol View on GitHub

4: task contracts
2: difficulty modes
16: audited v0.3 runs
37: automated tests

At a glance

One benchmark.Four task-semantic contracts.

See how CLEAR-LeWM replaces permissive proxies
with explicit, reproducible task rules.

Why CLEAR-LeWM

A high score should mean the task was actually solved.

The historical LeWM evaluator can assign stable success to random actions and can reward physically invalid or task-irrelevant endpoints. This weakens its ability to distinguish improvements to the model itself.

CLEAR-LeWM replaces those proxies with task-native geometry, temporal hold rules, route legality, and deterministic evaluation pairs. Every score remains traceable to a frozen goal, seed, solver budget, checkpoint, and protocol version.

Random-floor audit

Historical success can survive without a useful controller.

Random-policy success rate, 100 deterministic evaluation pairs per protocol. Lower is better.

PushT

Cube

47%

Reacher

16%

TwoRoom

30%

Moderate + Strict protocols

Choose the question your experiment needs to answer.

Moderate

Practical model-selection protocol with corrected semantics and attainable tolerances.

TaskSuccess contract

PushTBlock pose < 20 px / < 20 deg, held for 3 frames; displacement >= 25 px

CubePosition <= 4 cm; symmetry-aware angle <= 30 deg, held for 3 frames

ReacherWrapped joint error < 0.075 rad; first valid arrival

TwoRoomLegal swept-disk route across rooms; endpoint < 12 px, held for 3 frames

Task corrections

The metric follows the task, frame by frame.

Each evaluator isolates the object, geometry, or route that determines completion and exposes its decision trace for audit.

PushT trace showing the block pose metric over time

01 / PushT

Score the T-block, not the pusher.

Object pose, angular error, displacement, and temporal hold determine success. The pusher endpoint is task-irrelevant.

Read the task contract

Cube trace showing symmetry-aware position and orientation metrics

02 / Cube

Respect the cube's 24 rotational symmetries.

Orientation error is minimized over proper cube rotations, then combined with target position and a hold requirement.

Read the task contract

Reacher trace showing wrapped joint error and first arrival

03 / Reacher

Measure joint geometry without angle-wrap artifacts.

Wrapped joint distance defines arrival, while the trace keeps arrival and later stabilization conceptually separate.

Read the task contract

TwoRoom trace showing a collision-free cross-room route

04 / TwoRoom

Require a physically legal route through the door.

Swept-disk collision, eroded door clearance, cross-room topology, and endpoint accuracy jointly define success.

Read the task contract

Reference evaluation

A lower random floor reveals useful separation.

Official high-epoch LeWM checkpoints, seed 42, 100 deterministic pairs, 300 x 30 CEM, solver batch size 1. Cells report model / random SR (%).

Task	Historical	Moderate	Strict	Strict excess
PushT	89 / 7	93 / 7	79 / 2	+77 pp
Cube	62 / 47	43 / 4	18 / 3	+15 pp
Reacher	87 / 16	90 / 17	36 / 4	+32 pp
TwoRoom	85 / 30	61 / 2	24 / 0	+24 pp

Strict excess is model SR minus the matched random-policy SR.

Reproduce the score

Install, audit, evaluate.

CLEAR-LeWM is MIT-licensed and ships task contracts, deterministic protocols, provenance records, and automated tests in one public repository.

Quick start guide Protocol calibration Reproduction guide

git clone --recurse-submodules https://github.com/DavidSunok/CLEAR-LeWM.git
cd CLEAR-LeWM
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,lewm]'

Open evaluation infrastructure

Built for comparable world-model control research.

Versioned criteria keep future model changes comparable to the same physical and task-semantic standard. Cite the repository and protocol version used in your report.