01 / PushT
5.79×FAST loader vs Lance
Separate image matching from object completion.
Moderate keeps the full goal state. Strict scores only the T at 10 px / 10 deg for three steps.
Read the task contractA task-semantic evaluation suite for LeWM
CLEAR-LeWM
Task completion, not proxy success.
Versioned task contracts remove pre-solved pairs, repair demonstrated physics defects, and keep success aligned with each task's official meaning.
Official semantics
Physics-valid
Fixed manifests
No pre-solved pairs
- 4
- task contracts
- 2
- auditable v0.8 modes
- 84
- audited v0.8 runs
- 17.9×
- 4-task avg.
At a glance
One benchmark.Four task-semantic contracts.
See how one corrected benchmark supports
both compatibility and precise task semantics.
Why CLEAR-LeWM
A high score should mean the task was actually solved.
The historical LeWM evaluator retains initially solved pairs, can terminate Reacher inside dm-control before CLEAR scores the rollout, and includes a TwoRoom collision rewrite that can admit physically invalid transitions.
v0.8 Moderate keeps official task predicates while fixing termination, joint and collision topology, pre-solved pairs, and contaminated TwoRoom windows. Strict inherits those repairs and asks whether the task-relevant object or endpoint is precise.
Audited training I/O
17.87× geometric-mean loader speedup.
The generic FAST path decodes once into verified row-major memory maps while preserving episode boundaries and complete action chunks. Five to ten paired steady-state rounds ran serially after conversion; all exact-equivalence audits passed. Figures are loader-only.
Read the performance auditPushT · Lance5.79×672 → 3812 samples/s
Cube · HDF537.72×120 → 4427 samples/s
Reacher · HDF530.77×143 → 4362 samples/s
TwoRoom · HDF515.15×279 → 4291 samples/s
Random-floor audit
Historical success can survive without a useful controller.
Random-policy success rate, mean over three 100-pair fixed manifests per protocol. Lower is better.
v0.8 Moderate + Strict
Choose the claim your experiment makes.
01
Moderate
Minimal repair for LeWM-compatible comparison: remove trivial cases, fix Reacher topology, and restore TwoRoom physics.
TaskSuccess contract
PushTPusher + block position < 20 px; T angle < 20 deg; first hit
CubeCube position <= 4 cm; first hit; orientation and robot pose unscored
ReacherPeriodic shoulder + bounded wrist error < 0.05 rad; first hit
TwoRoomCross-room clean source window; swept-disk runtime; endpoint < 16 px
02
Strict
Task-semantic precision: isolate the physical object or endpoint and apply tighter geometry and persistence.
TaskSuccess contract
PushTT only < 10 px / < 10 deg, held for 3 steps; pusher unscored
CubeCube <= 3 cm + 24-fold angle <= 15 deg, held for 3 steps; robot unscored
ReacherPhysical fingertip endpoint <= 1 cm, held for 2 steps
TwoRoomValid legal crossing + goal side + endpoint < 8 px
Task corrections
The metric follows the task, frame by frame.
Each evaluator isolates the object, geometry, or route that determines completion and exposes its decision trace for audit.
02 / Cube
37.72×FAST loader vs HDF5
Move the object, then measure its precise pose.
Moderate follows OGBench position. Strict adds 24-fold cube orientation; robot pose is never scored.
Read the task contract
03 / Reacher
30.77×FAST loader vs HDF5
Repair joint topology, then score the endpoint.
Both modes own termination after reset or recompilation. Strict requires the physical fingertip within 1 cm for two steps.
Read the task contract
04 / TwoRoom
15.15×FAST loader vs HDF5
Repair collision, then demand a legal crossing.
Both modes use clean windows and swept disks. Strict adds route, crossing, goal-side, and 8 px gates.
Read the task contractMatched evaluation
Three LeWM variants. One v0.8 contract.
The model comparison uses seeds 0, 1, and 42, 100 episodes each, pure CEM 300 x 30, and solver batch size 1.
Canonical model SR
Success rate under identical evaluation.
Loading matched model comparison...
Only tasks submitted by all three models are shown. Reacher remains unreported for both community models.
Official reference
Three-seed LeWM baseline.
Means over seeds 0, 1, and 42. Each mode retains its own paired-random control.
| Task | ModerateLeWM / random | StrictLeWM / random |
|---|---|---|
| PushT | 86.67%random 4.00%+82.67 pp | 70.67%random 5.00%+65.67 pp |
| Cube | 50.33%random 15.67%+34.67 pp | 21.67%random 6.00%+15.67 pp |
| Reacher | 79.67%random 7.33%+72.33 pp | 87.00%random 8.00%+79.00 pp |
| TwoRoom | 83.00%random 6.67%+76.33 pp | 51.33%random 1.67%+49.67 pp |
Main values are official LeWM SR; secondary values are paired random and excess over random. Full per-seed JSON records are checked into results/v0.8.
Community submissions
More LeWM models, under the same contract.
Canonical entries use seed 42 and 100 episodes per task and mode. Each bundle is checked by CI but remains self-reported unless independently reproduced.
Loading validated community results...
Cells report model / paired random, followed by excess over random. Missing tasks are never imputed. Submit a result bundle.
Reproduce the score
Install, audit, evaluate.
CLEAR-LeWM is MIT-licensed and ships task contracts, deterministic protocols, provenance records, and automated tests in one public repository.
git clone --recurse-submodules https://github.com/DavidSunok/CLEAR-LeWM.git
cd CLEAR-LeWM
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,lewm]'Open evaluation infrastructure
Built for comparable world-model control research.
Versioned criteria keep future model changes comparable to the same physical and task-semantic standard. Cite the repository and protocol version used in your report.