Whole-program search: parts re-search, re-search the 3 best (0.7.0)

.github/workflows/ablation.yml

@@ -33,7 +33,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        config: [full, no-research, no-cleanup, no-exact-grouping, no-box-shadows, no-tag-rules,
+        config: [full, no-research, no-parts, research-1, no-cleanup, no-exact-grouping, no-box-shadows, no-tag-rules,
                  no-clip-reuse, top-only, no-corner-split, no-outline-polygons, no-length-aware, no-gradient-base,
                  no-soft-stops, no-motifs]
     steps:

.github/workflows/benchmark.yml

@@ -4,6 +4,10 @@ name: Benchmark
 
 on:
   workflow_dispatch:
+    inputs:
+      research:
+        description: "Re-search the N best results (compile_image research=N)"
+        default: "3"
 
 jobs:
   bench:
@@ -21,7 +25,7 @@ jobs:
       - run: npm ci
         working-directory: bench
       - run: python bench/fetch.py
-      - run: python bench/run.py
+      - run: python bench/run.py --research ${{ inputs.research }}
       - run: python bench/tradeoff.py
       - run: python bench/build_page.py
       - uses: actions/upload-artifact@v7

.github/workflows/heldout.yml

@@ -0,0 +1,46 @@
+# Held-out evaluation: generate one held-out set of structural families
+# (bench/families.py), compile it with each method one after another on the
+# same runner (so CPU times compare), render in Chromium and print the
+# summary (bench/evaluate.py). Run it by hand from the Actions tab, once per
+# version: a set is spent once it has been run (see docs/method.md).
+name: Held-out
+
+on:
+  workflow_dispatch:
+    inputs:
+      set:
+        description: "Held-out set, a key of HELDOUT_SEEDS in bench/families.py"
+        required: true
+      methods:
+        description: "Methods of bench/evaluate.py, space-separated"
+        required: true
+        default: "full"
+
+jobs:
+  evaluate:
+    runs-on: ubuntu-latest
+    timeout-minutes: 330
+    env:
+      SET: ${{ inputs.set }}
+      METHODS: ${{ inputs.methods }}
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 22
+      - run: pip install -e .
+      - run: npm ci
+        working-directory: bench
+      - run: python bench/families.py "$SET"
+      - run: for m in $METHODS; do python bench/evaluate.py --set "$SET" --method "$m"; done
+      - run: python bench/evaluate.py --set "$SET" --compare $METHODS | tee bench/eval/summary.txt
+      - uses: actions/upload-artifact@v7
+        with:
+          name: heldout
+          path: |
+            bench/${{ inputs.set }}/cases.json
+            bench/${{ inputs.set }}/*.html
+            bench/eval/

.gitignore

@@ -24,6 +24,7 @@ bench/train/
 bench/learn/
 bench/heldout2/*.png
 bench/heldout3/*.png
+bench/heldout4/*.png
 
 # the demo site, assembled locally (demo/README.md)
 demo/site/

README.md

@@ -40,10 +40,10 @@ Chromium and compared pixel by pixel:
 
 | | |
 |---|---|
-| Shorter than the human solution (minified) | **56 of 66** |
-| Total code length | **23,329** characters (human: 35,532) |
+| Shorter than the human solution (minified) | **60 of 66** |
+| Total code length | **21,175** characters (human: 35,532) |
 | Targets at ≥ 99% pixel match | **66 of 66** (lowest 99.21%) |
-| Mean pixel match | 99.82% |
+| Mean pixel match | 99.84% |
 
 The human solutions are readable code from two public repositories, minified
 but not golfed. Top CSSBattle players write far shorter code, so these
@@ -61,25 +61,28 @@ rows don't add up.
 
 | Switched off | Total J | Characters | Shorter than human |
 |---|---|---|---|
-| nothing (full compiler) | 60,441 | 23,329 | 56 / 66 |
-| Insertion at any depth (shapes go on top) | 64,637 (+6.9%) | 24,887 (+1,558) | 54 / 66 |
-| Polygons traced from outlines | 64,449 (+6.6%) | 25,791 (+2,462) | 52 / 66 |
-| Re-search of the best result | 64,444 (+6.6%) | 24,652 (+1,323) | 55 / 66 |
-| Gradient backgrounds | 63,265 (+4.7%) | 24,077 (+748) | 54 / 66 |
-| Length-aware shape choice (by pixels only) | 62,921 (+4.1%) | 24,017 (+688) | 54 / 66 |
-| Length-aware clean-up | 61,982 (+2.5%) | 24,914 (+1,585) | 52 / 66 |
-| Splitting regions that touch at corners | 61,705 (+2.1%) | 24,005 (+676) | 55 / 66 |
-| Nested-ring motifs | 61,309 (+1.4%) | 23,495 (+166) | 56 / 66 |
-| Box-shadows | 61,258 (+1.4%) | 24,150 (+821) | 56 / 66 |
-| Soft gradient stops | 61,106 (+1.1%) | 23,680 (+351) | 55 / 66 |
-| Per-tag rules (`<i>`, `<u>`, `<s>`) | 60,807 (+0.6%) | 23,583 (+254) | 55 / 66 |
-| `clip-path` reuse for copies and mirrors | 60,708 (+0.4%) | 23,544 (+215) | 55 / 66 |
-| Exact (CP-SAT) shadow grouping, greedy kept | 60,554 (+0.2%) | 23,442 (+113) | 56 / 66 |
-
-Inserting shapes below existing ones, outline polygons and the re-search
-matter most; gradient backgrounds and choosing shapes by length as well as
-pixels come next. Exact shadow grouping and `clip-path` reuse are small on
-average but decide a few targets.
+| nothing (full compiler) | 55,925 | 21,175 | 60 / 66 |
+| Re-search of the best results | 63,980 (+14.4%) | 24,086 (+2,911) | 56 / 66 |
+| Redrawing a color in parts (re-search) | 59,350 (+6.1%) | 22,692 (+1,517) | 57 / 66 |
+| Polygons traced from outlines | 59,032 (+5.6%) | 23,074 (+1,899) | 59 / 66 |
+| Insertion at any depth (shapes go on top) | 58,545 (+4.7%) | 21,819 (+644) | 59 / 66 |
+| Length-aware shape choice (by pixels only) | 57,824 (+3.4%) | 21,386 (+211) | 61 / 66 |
+| Gradient backgrounds | 57,462 (+2.7%) | 21,514 (+339) | 59 / 66 |
+| Re-searching the 3 best results (only the best) | 57,269 (+2.4%) | 21,691 (+516) | 60 / 66 |
+| Nested-ring motifs | 56,758 (+1.5%) | 21,260 (+85) | 60 / 66 |
+| Box-shadows | 56,674 (+1.3%) | 21,876 (+701) | 60 / 66 |
+| Length-aware clean-up | 56,567 (+1.1%) | 21,779 (+604) | 61 / 66 |
+| `clip-path` reuse for copies and mirrors | 56,184 (+0.5%) | 21,254 (+79) | 59 / 66 |
+| Per-tag rules (`<i>`, `<u>`, `<s>`) | 56,171 (+0.4%) | 21,293 (+118) | 59 / 66 |
+| Splitting regions that touch at corners | 55,990 (+0.1%) | 21,158 (−17) | 61 / 66 |
+| Exact (CP-SAT) shadow grouping, greedy kept | 55,971 (+0.1%) | 21,221 (+46) | 60 / 66 |
+| Soft gradient stops | 55,961 (+0.1%) | 21,167 (−8) | 60 / 66 |
+
+The re-search matters most, and its step that redraws a color in parts is
+worth 6.1% on its own; outline polygons and inserting shapes below existing
+ones come next. Three parts now move the total by 0.1% or less: splitting
+regions at corners, soft gradient stops and exact shadow grouping. Rows that
+show more targets shorter than human (61) do so at a higher total J.
 
 ### Beyond the development set
 
@@ -89,28 +92,30 @@ test it elsewhere (details in [docs/method.md](docs/method.md#beyond-the-develop
 **Held-out structural families.** 48 targets generated from six structures:
 mirrored shapes, repeats, concentric rings, petals, stripes and unrelated
 shapes. They are compiled once per version, never inspected, and scored in
-Chromium (`bench/evaluate.py`). The second baseline climbs 40% more starts,
-to spend at least the CPU time the motif search does:
+Chromium (`bench/evaluate.py`, run by the **Held-out** workflow). The fourth
+set was spent on 0.7.0, all three methods on one runner. The second baseline
+climbs 48 starts per step instead of 10, to spend at least the CPU time
+0.7.0 does:
 
-| | solved (≥ 99%) | characters, on the 40 all solve | CPU |
+| | solved (≥ 99%) | characters, on the 43 all solve | CPU |
 |---|---|---|---|
-| v0.5.0 | 40 / 48 | 11,944 | 2,634 s |
-| v0.5.0, 14 starts per step | 41 / 48 | **11,240** | 3,177 s |
-| with nested-ring motifs | **43 / 48** | 11,627 | **2,311 s** |
-
-The motifs solve the most with the least time. The baseline with more starts
-writes 3.4% less code on the targets all three solve, but that rests on three
-scatter and petals targets: it is shorter on 6 and longer on 7, and the 95%
-bootstrap interval runs from 2.1% shorter to 9.6% longer. Giving the default
-more starts makes its code longer, not shorter (+6.5% on the development
-split). (This is the third
-held-out set; the earlier two were spent on earlier versions, see
-[docs/method.md](docs/method.md#motifs).)
+| 0.6.0's search | 43 / 48 | 16,722 | **4,667 s** |
+| 0.6.0's search, 48 starts per step | 43 / 48 | 16,728 | 9,143 s |
+| 0.7.0 | 43 / 48 | **14,696** | 8,141 s |
+
+0.7.0 writes 12.1% less code on the targets all three solve: shorter than
+0.6.0's search on 9 of them and longer on 3, with a 95% bootstrap interval
+from 22.7% to 1.4% shorter. Rings (2,564 to 1,528 characters) and petals
+(4,333 to 3,292) gain the most; scatter is 70 characters longer. It solves
+no more targets and takes 1.7 times the CPU, and giving 0.6.0's search that
+time as more starts changes nothing (16,728 characters). (The earlier three
+sets were spent on earlier versions, see
+[docs/method.md](docs/method.md#beyond-the-development-set).)
 
 **Known optima.** On 40 tiny targets (up to four grid-aligned boxes and
 ellipses), an exact solver finds the shortest program that cssynth's own
 emitter can write. cssynth reproduces 35 exactly and is optimal on 33 of
-them (30 before this round); the other two are 40 characters over.
+them (30 before 0.6.0; the same in 0.7.0); the other two are 40 characters over.
 
 Petals (leaves turned about a center) remain the weak family: Chromium
 antialiases turned and curved edges differently from cssynth's model, so the
@@ -124,6 +129,7 @@ cssynth target.png           # prints the code; length and details on stderr
 cssynth -v target.png        # also logs the search
 cssynth --rho 2 target.png   # golf harder (see "Objective")
 cssynth --fast target.png    # skip the re-search: about 2x faster, longer code
+cssynth --research 1 target.png  # re-search only the best result: less CPU, longer code
 ```
 
 From Python:
@@ -158,7 +164,7 @@ flowchart LR
     E --> F[simplify for length]
     F --> G[code generation]
     G --> H{lowest J<br>of 16 variants}
-    H --> I[re-search parts<br>of the best]
+    H --> I[re-search parts<br>of the 3 best]
 ```
 
 1. **Palette** (`search.extract_palette`): flat colors with at least 40
@@ -243,12 +249,16 @@ flowchart LR
    but not all: it can pick a cheap shape that leaves more for later.
 8. **Re-search** (`improve.improve`): the greedy search commits to shapes in
    the order they first help, so an early polygon can take a region that a
-   shorter primitive would have drawn. The best result is taken apart and
-   refilled: one polygon at a time (refilled without polygons of its color),
-   then every shape of one color (refilled as before, and without polygons).
-   Refills use the winning variant's mode. Each is refined and simplified,
-   and kept when J drops. `--fast`
-   skips this step.
+   shorter primitive would have drawn. Each of the 3 best results (distinct
+   programs, in parallel) is taken apart and refilled: one polygon at a
+   time (refilled without polygons of its color),
+   then every shape of one color (refilled as before, and without polygons),
+   then every shape of a color whose region falls into parts, one shape per
+   part (two circles stacked on an oval). Refills use the winning variant's
+   mode. Each is refined and simplified, and kept when J drops; the best
+   finished program wins, which isn't always the one that started best.
+   `--fast` skips this step; `--research 1` runs it on the best result
+   only.
 
 ## Benchmark
 
@@ -303,7 +313,7 @@ ruff check .
 ```
 cssynth/
   compiler.py   compile_image: run the variants, simplify, keep the lowest J
-  improve.py    re-search: take the best result apart and refill it
+  improve.py    re-search: take the best results apart and refill them
   search.py     palette, base layer, greedy shape search, refine
   shapes.py     the Shape model and a rasterizer that predicts Chromium's
   simplify.py   length-aware clean-up moves
@@ -321,14 +331,16 @@ tests/
   turns as motifs rescued some petals but lengthened other targets. A filter
   learned from outcome-labeled proposals didn't transfer to structures it
   hadn't seen (docs/method.md). Otherwise the search still goes one shape at
-  a time, and the re-search only repairs its choices locally. Two light circles stacked on an oval (2024-02-27) become one tall
-  rounded box plus patches, because the box fixes more pixels than either
-  circle, and a climb from one circle grows back into the box. The lotus is
-  drawn with polygons where a golfer uses three rotated leaves. Comparing
-  whole programs (two circles plus a cheap shadow against a box plus
-  patches) is the next step. A beam over partial J is not enough on its
-  own: the two-circle branch trails for two steps before it pulls ahead,
-  and the climbs never propose a lone circle to start it.
+  a time. The re-search now compares whole programs in one case: a color
+  whose region falls into parts (two circles stacked on an oval, a cloud of
+  discs) is redrawn with one shape per part, finished, and kept if the
+  finished program scores better. On cssynth's own scores this lowered the
+  development total by 4.6% and made no target worse; measured in
+  Chromium, 59 targets instead of 56 come out shorter than the human code
+  (docs/method.md). Choices that don't show up as parts
+  of one color still go one step at a time, and a beam over partial J
+  doesn't fix that: on 2024-02-27 the two-circle branch trails for two
+  steps before it pulls ahead.
 - **Reuse.** Turned copies of a box share a rule and mirrored copies of a
   polygon share a `clip-path`, but nothing yet uses `::before`/`::after` to
   repeat a shape, or mirrors a rounded box.