Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 75 additions & 3 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Evaluation scripts

Two unattended scripts measure SpatialDSL Studio without a browser, an HTTP
server, or a database. Both call the backend engines in-process and write their
Three unattended scripts measure SpatialDSL Studio without a browser, an HTTP
server, or a database. All call the backend engines in-process and write their
results to `benchmarks/results/` (git-ignored).

## Prerequisites

```bash
cd backend && npm ci # the scripts reuse the backend's ts-node and engines
cd ../benchmarks
python3 -m venv .venv && .venv/bin/pip install usd-core # fidelity only; provides pxr
python3 -m venv .venv && .venv/bin/pip install usd-core # fidelity and twin loop; provides pxr
```

## Scalability
Expand Down Expand Up @@ -53,3 +53,75 @@ error or warning.

Outputs: `results/fidelity.json`, `results/fidelity.md`, and the generated
bundles in `results/fidelity/`. Expected runtime: under 10 seconds.

## Twin loop

```bash
cd benchmarks && npm run twin-loop
```

Generation fidelity compares the model with the generated files. The twin loop
continues from there: it compares the model with what the scene does when it
runs, and brings the result back in the model's own terms.

1. The bundled Smart Warehouse model is generated into
`results/twin-loop/bundle/`. Every generated prim and layout record carries
the ID of its model element (`spatialDsl:elementId`, `id`).
2. `observe.py snapshot` reads the composed stage with pxr and records the
authored pose of every stamped prim, with the layer that authors it, and a
fingerprint of the generated bundle (the layout JSON, the root scene layer,
and its sublayers).
3. A simulator runs the bundle and records an observation log: the world pose
of every stamped prim over time, plus which elements the simulator drives.
Every sample must cover every element of the snapshot; an incomplete log is
rejected rather than read as "nothing moved".
4. Each sample is lifted onto a copy of the model: the observed model pose is
the design pose plus the difference between the observed and the authored
scene pose. The design-time spatial rules are then evaluated, unchanged, on
the lifted model.

The report lists, per run:

- **realization offsets**: undriven elements that were not at their authored
pose in any sample;
- **runtime drifts**: undriven elements that moved away from where they
started, with the onset time and the displacement at onset and at the end
(zero at the end when the element came back);
- **rule verdicts that differ** between the model and the run, compared sample
by sample, with the first and last sample and the number of samples in
which they differ;
- a **reconciliation proposal**: the model changes that accepting the observed
state would make (`reconciled-model.json`, importable as a model), and the
rule issues that model would gain or lose.

Each finding names the model element, the prim, the generated file, and the
template that produced that file.

Without arguments the script runs four scenarios on the simulator stand-in
(`run_standin_sim.py`: the MAS brain drives the robots kinematically, with no
physics engine): an undisturbed run, a storage rack pushed into the aisle at
t = 20 s, the same rack pushed back at t = 40 s, and a dock resting 150 mm
above its authored pose. It exits non-zero if the findings differ from what was
injected. The stand-in checks the loop,
not the physics: deviations exist only where a scenario injects them.

To analyse a run recorded in Isaac Sim, run the bundle from
`results/twin-loop/bundle/` with the brain and the bridge (see
`examples/omniverse-assets/warehouse_sim/README.md`) and pass the log back:

```bash
python3 warehouse_brain.py --layout <bundle>/warehouse_layout.json # terminal 1
/isaac-sim/python.sh run_warehouse_sim.py --layout <bundle>/warehouse_layout.json \
--scene <bundle>/warehouse_simulation.usda \
--observe-out observations.jsonl --headless --max-seconds 60 # terminal 2
npm run twin-loop -- --observations observations.jsonl
```

The log records the fingerprint of the generated bundle it ran against; the
script refuses a log recorded against a different generation of the model and
names the generated files that differ. A log may be read from anywhere,
including `results/twin-loop/`, which the script cleans only after reading it.

Outputs: `results/twin-loop.json`, `results/twin-loop.md`, and per-scenario
`report.json`, `report.md`, `observations.jsonl`, and `reconciled-model.json`
in `results/twin-loop/`. Expected runtime: about 5 seconds.
Loading
Loading