Skip to content

feat(abi): generate the Zig and Rust wire enums from the normative Idris2 ABI (#120) - #811

Merged
hyperpolymath merged 2 commits into
mainfrom
feat/issue-120-abi-codegen
Sep 22, 2026
Merged

hyperpolymath merged 2 commits into
mainfrom
feat/issue-120-abi-codegen

Conversation

@hyperpolymath

@hyperpolymath hyperpolymath commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Closes #120.

The 16-connector wire contract existed in five hand-maintained copies that
agreed only by discipline. This makes three of them generated from the
normative Idris2 ABI, so drift becomes impossible rather than merely detected.

What generates what

src/Hypatia/ABI/Gen.idr + src/abi/hypatia-abi-gen.ipkg build an Idris2
executable that imports Hypatia.ABI.Types and evaluates it. It never
parses .idr text. Because Gen is the mainfile, --total is genuinely
re-checked for it (Idris2 only re-checks --total for the mainfile).

Generated From
ffi/zig/src/connector_generated.zig src/Hypatia/ABI/Types.idr
clients/rust/hypatia-client/src/connector_generated.rs same
ffi/connectors.json same

ffi/connectors.json previously called itself the "Golden source-of-truth",
which contradicts the ABI-authority ruling. That claim is gone.

zigIdent / rustIdent live in Gen, not in the ABI — the normative module
must not know about its consumers. Both are total with one clause per
constructor, so adding a connector without naming it is a build failure,
which is strictly stronger than a CI diff.

The hand-written unified-api-adapter.zig and connector.rs keep their own
tests. A generated test re-asserts its own source and proves nothing.

just abi-gen regenerates. Every generated file carries
// ABI source blob: <git hash-object src/Hypatia/ABI/Types.idr>.
⚠ Deliberate consequence: any byte change to Types.idr, comments
included, reddens the gate until just abi-gen is re-run and committed.
Touching the normative ABI should force you to look at what it generated.

Three things this PR found, stated plainly

1. Before this PR, neither "independent pin" on the wire contract executed
anywhere.

  • The Zig comptime 16-connector assertion and the four tests in
    unified-api-adapter.zig were reachable from no test root. build.zig
    rooted main.zig, json_writer.zig, file_ops.zig and
    test/integration_test.zig; none import the adapter. They had never run,
    locally or in CI. This PR adds the fourth root — zig build test goes from
    3 tests to 7.
  • cargo test --workspace is red on main for an unrelated pre-existing
    reason: integration/Cargo.toml requires testcontainers ^0.28 and
    testcontainers-modules ^0.15, which needs testcontainers ^0.27. That is
    unsatisfiable at the manifest level, so no lockfile can fix it and
    hypatia-client never gets compiled. Its five tests were verified against
    the generated enum in a standalone crate: 5 passed, 0 failed.

So abi-codegen-drift is currently the only executing pin on the wire
contract, and zig.yml makes the comptime assertion the second. That is an
improvement on nothing rather than "one of three independent pins".

2. The Elixir drift guard was reading a dead file. It bound its golden to
src/abi/Types.idr — a byte-identical copy that is compiled by nothing (the
live module is src/Hypatia/ABI/Types.idr; src/abi/hypatia-abi.ipkg sets
sourcedir = "..", which resolves to src/). The live ABI could drift and the
guard would stay green. This is instance 21 of the guard-asks-a-different-
question-than-its-consumer trap. Repointed, and a mutant proves the fix:
renaming a connector in the live module reddens the guard while the dead copy
is untouched.

The guard also gained a wire-id check. A mutant showed the name-only
version stayed green through an id swap — names and their order are
unchanged by a renumber, so a name-only guard sleeps through a C ABI break.

3. zig build test is red on main for pre-existing reasons unrelated to
this PR: three of four roots fail to compile against the Zig 0.16 std — 16 call
sites across six removed or relocated symbols (std.fs.cwd,
std.posix.getenv, std.fs.openFileAbsolute, std.fs.createFileAbsolute,
std.fs.deleteFileAbsolute, std.time.timestamp) plus an opaque type
declared with fields at main.zig:46. zig.yml therefore runs the maximal
green subset (zig test on the adapter), not the whole build. Filed as its own
issue rather than folded in here.

The gate

.github/workflows/abi-codegen-drift.yml, no paths: filter, so it fires
on every PR and can be made required later without the filter/required
deadlock. (verify-proofs.yml filters on src/abi/**, src/Hypatia/ABI/**
and verify/** and lists none of the generated targets, so a hand-edit to
a generated file would not have fired it.)

It regenerates into $RUNNER_TEMP and diffs — never in place with
git diff. Measured, in an isolated repo: a no-op generator + git diff
produces an empty diff, rc=0, reads GREEN. The temp-dir design on the same
no-op prints connectors emitted: 0 / files compared: 0 of 3 and fails.

It prints its denominator and fails on a zero one. It does not assert
N == 16 — that would be a sixth hand-written 16; the count is the proof's
job (connectorCount = Refl).

It reuses verify-proofs.yml's Idris2 install verbatim, including the same
cache key
so it shares that warm cache.

Mutants — a passing gate proves nothing until one dies

Every red below was verified to be a drift or assertion failure, not a compile
error.

# Mutant Result
1 hand-edit a generated Zig wire name gate RED, guard RED (independently)
2 swap wire ids 3↔4 in the live ABI, do not regenerate gate RED on all 3 files; guard initially GREEN → blind spot → guard strengthened → now RED
3a add a 17th connector, do not name it in Gen generator BUILD FAILS: rustFile,variant is not covering
3b …then name it and regenerate Zig comptime RED; Rust count_is_sixteen + from_id_round_trip RED
4 replace the generator with a silent no-op exiting 0 connectors emitted: 0 / files compared: 0 of 3, RED

Positive control, clean tree: gate rc=0 printing connectors emitted: 16 /
files compared: 3 of 3; guard 4/4; zig test 4/4; zig build test 7/7;
Rust 5/5; just abi-gen byte-identical and idempotent.

actions.lock

Hand-edited, purely additive: 6 insertions, 0 deletions — two workflow keys
in alphabetical position. Both actions already had dependency entries, so none
was added. Tags, not SHAs, because that is the format every one of the other 30
workflow entries uses; a lone SHA-pinned workflow would be inconsistent with
the lock in use. Fix/rewrite mode was not used — it has previously
de-pinned 24 SHAs and invented invalid uses: refs.

gh actions-lock --verify-local (offline, writes nothing) goes from
2 of 32 workflows failed: 4 not-pinned to
All 32 workflows have complete lockfile coverage.

This gate lands advisory

rules/branches/main on this repo returns [] — zero effective rules, proven
with a positive control (a repo returning 8 rule types) and a 404 negative
control. Optimus-Branch exists but is enforcement: disabled. So nothing
here can block anything until a ruleset requires it. Giving it teeth is an
owner action, tracked separately.

⚠ Do not enable Optimus-Branch for this — it carries five required
contexts whose reporting on hypatia PRs is unmeasured, and a required context
that never reports is how a default branch deadlocks.

Follow-ups (issues, not blockers)

  • Delete the six duplicated src/abi/*.idr modules and correct the ~15 prose
    references — the agreed immediately-following PR.
  • Zig 0.16 std migration (the 16 call sites above).
  • The integration/ testcontainers conflict. Note DEBT-REGISTER CI-1 blames
    "a stale dtolnay/rust-toolchain@stable pin … 13 jobs die at Set up job" —
    that is now wrong; the jobs start and die at cargo resolution.
  • mise.toml pins zig = "latest" — a floating pin.
  • verification/PROOF-STATUS.adoc is stale.
  • IDApTIK docs/abi/ABI-AUTHORITY.md should repoint to
    src/Hypatia/ABI/Types.idr.

🤖 Generated with Claude Code

https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV


Residue, filed as issues (added post-merge)

Per the standing rule that a new finding is an issue, not a merge blocker —
each with explicit acceptance criteria:

Discovered after merge:

…ris2 ABI (#120)

The 16-connector wire contract existed in five hand-maintained copies that
agreed only by discipline. This makes three of them generated from the
normative Idris2 ABI, so drift becomes impossible rather than merely detected.

New:
  - src/Hypatia/ABI/Gen.idr + src/abi/hypatia-abi-gen.ipkg -- an Idris2
    executable that IMPORTS Hypatia.ABI.Types and EVALUATES it. It never
    parses .idr text. Because Gen is the mainfile, --total is genuinely
    re-checked for it.
  - ffi/zig/src/connector_generated.zig
  - clients/rust/hypatia-client/src/connector_generated.rs
  - ffi/connectors.json is now generated (its "Golden source-of-truth" claim
    contradicted the ABI-authority ruling and is gone)
  - `just abi-gen`
  - .github/workflows/abi-codegen-drift.yml -- regenerates into a temp dir and
    diffs. Never regenerate-in-place-and-`git diff`: measured, that reads GREEN
    when the generator silently no-ops.
  - .github/workflows/zig.yml

zigIdent/rustIdent live in Gen, not in the ABI -- the normative module must not
know about its consumers. Both are total with one clause per constructor, so
adding a connector without naming it is a BUILD failure, not a CI diff.

The hand-written files keep their tests. A generated test re-asserts its own
source and proves nothing.

Also closes instance 21 of the guard/consumer trap: the Elixir guard read
src/abi/Types.idr, a byte-identical copy compiled by nothing, so the live ABI
could drift and the guard would stay green. It now reads
src/Hypatia/ABI/Types.idr. A mutant proved the fix: renaming a connector in the
live module reddens the guard while the dead copy is untouched.

The guard also gained a wire-id check. A mutant showed the name-only version
stayed GREEN through an id swap -- names and their order are unchanged by a
renumber, so a name-only guard sleeps through a C ABI break.

ffi/zig/build.zig now roots unified-api-adapter.zig. It was reachable from no
test root, so its comptime 16-connector assertion and its four tests had never
executed anywhere. `zig build test` goes from 3 tests to 7.

Measurements worth recording:
  - `zig build test` is RED on main for pre-existing reasons: three of four
    roots fail to compile against the Zig 0.16 std (16 call sites, six removed
    or relocated symbols) plus an `opaque` type with fields. zig.yml therefore
    runs the maximal green subset, not the whole build.
  - `cargo test --workspace` is RED on main for an unrelated pre-existing
    reason: integration/Cargo.toml requires testcontainers ^0.28 AND
    testcontainers-modules ^0.15, which needs testcontainers ^0.27. That is
    unsatisfiable at the manifest level, so no lockfile can fix it and
    hypatia-client never gets compiled. Its five tests were verified against
    the generated enum in a standalone crate: 5 passed, 0 failed.
  - Consequently, before this PR NEITHER "independent pin" on the wire contract
    executed anywhere. The drift gate is currently the only executing pin, and
    zig.yml makes the comptime assertion the second.

actions.lock: two workflow keys added by hand, purely additive (6 insertions,
0 deletions). Both actions were already present, so no new dependency entry.
`gh actions-lock --verify-local` goes from "2 of 32 workflows failed:
4 not-pinned" to "All 32 workflows have complete lockfile coverage". Fix mode
was not used -- it has previously de-pinned SHAs and invented invalid refs.

Every gate was mutation-tested; all reds were drift or assertion failures, not
compile errors:
  1. hand-edit a generated Zig wire name  -> gate RED, guard RED
  2. swap wire ids 3<->4 in the live ABI, do not regenerate
                                          -> gate RED (3 of 3 files), guard RED
  3. add a 17th connector without naming it in Gen
                                          -> generator BUILD fails (not covering)
     then name it and regenerate          -> Zig comptime RED, Rust
                                             count_is_sixteen + from_id_round_trip RED
  4. generator replaced by a silent no-op exiting 0
                                          -> gate prints "connectors emitted: 0" /
                                             "files compared: 0 of 3" and fails
Positive control on a clean tree: gate rc=0 printing 16 / 3 of 3, guard 4/4,
zig 4/4, rust 5/5, `just abi-gen` byte-identical and idempotent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV
@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 96083ec5-4027-4b4c-81f0-eb110eb7acbc

📥 Commits

Reviewing files that changed from the base of the PR and between 0157a9b and 086998f.

⛔ Files ignored due to path filters (1)
  • .github/workflows/actions.lock is excluded by !**/*.lock
📒 Files selected for processing (14)
  • .github/workflows/abi-codegen-drift.yml
  • .github/workflows/zig.yml
  • Justfile
  • clients/rust/hypatia-client/src/connector.rs
  • clients/rust/hypatia-client/src/connector_generated.rs
  • clients/rust/hypatia-client/src/lib.rs
  • ffi/connectors.json
  • ffi/zig/build.zig
  • ffi/zig/src/connector_generated.zig
  • ffi/zig/src/unified-api-adapter.zig
  • src/Hypatia/ABI/Gen.idr
  • src/Hypatia/ABI/Types.idr
  • src/abi/hypatia-abi-gen.ipkg
  • test/unified-api-adapter-contract_test.exs
 _________________________________________________________________
< You sanitized... the wrong thing. That's impressively specific. >
 -----------------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • 🔴 Error committing to branch - (🔄 Check to retry)
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread .github/workflows/zig.yml
zig version
test "$(zig version)" = "${ZIG_VERSION}"

- name: Run the wire-contract tests
@hyperpolymath
hyperpolymath merged commit e0393c2 into main Sep 22, 2026
36 of 44 checks passed
@hyperpolymath
hyperpolymath deleted the feat/issue-120-abi-codegen branch September 22, 2026 09:39
@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

❌ Failed to create Coding Agent finishing-touch task. Please try again.

hyperpolymath added a commit that referenced this pull request Sep 22, 2026
…#817)

## The gate I shipped in #811 is red on `main`

#811 merged with its own new `ABI codegen drift` check failing. This
fixes it
forward. The gate is loudly red rather than silently green, so it is not
a
vacuous gate — but a gate that has never passed is not yet evidence of
anything, and left red it trains people to ignore it.

### The cause, measured

```
INTERNAL ERROR: Can't find data file chez/support.ss in any of
["/home/runner/.idris2/idris2-0.7.0/support/chez/support.ss",
 ".../base-0.7.0/data/chez/support.ss",
 .../contrib-0.7.0/data/chez/support.ss"]
```

Compiling an Idris2 **executable** needs the Chez support tree at
`$(idris2 --libdir)/support/chez/support.ss`. Type-checking and building
a
**library** does not.

`verify-proofs.yml` only ever does the latter — `idris2 --check` over
the proof
files, `idris2 --build` on two library packages — and never links a
binary. So
the cache this job deliberately shares with it (same key
`idris2-${IDRIS2_VERSION}-${runner.os}-2`, chosen so the gate would not
repeat
a 20-minute bootstrap) was populated **without** that tree.

**Sharing a warm cache imports its omissions as well as its contents.**
That is
the generalisable lesson here, and it is why my local build passed while
CI's
failed: locally the support tree exists under pack, so the local success
was
never evidence about CI.

### The fix

A step between "Verify Idris is on PATH" and "Build the generator" that:

1. resolves `$(idris2 --libdir)/support` and checks for
`chez/support.ss`;
2. if absent, shallow-clones the Idris2 source at the **pinned**
`IDRIS2_VERSION`
and copies `support/` into place — `support/` is plain data (Scheme and
C
   sources), needs no build, so this costs seconds, not a bootstrap;
3. **asserts the file is present by name** before continuing.

Step 3 matters independently of step 2. A missing support tree
previously
surfaced as an opaque `INTERNAL ERROR` three steps later, inside the
generator
build, where it read like a generator bug. Now it fails where it is
caused.

### Also in this PR

`zig.yml` extracted the test count with `grep … || true`, which the
`Hypatia`
workflow scanner flagged at `zig.yml:99` as swallowing a non-zero exit.

`set -o pipefail` already made a genuine `zig test` failure fatal, so
nothing
was actually masked — but a suppressed exit code is indistinguishable
from a
masked failure to a reader or a scanner, so this is a rewrite rather
than an
argument. `grep` exits 1 on no-match, which under `set -e` needs
suppressing;
**awk exits 0 whether or not it matched**, so the no-match case needs no
suppression at all and the count check below it becomes the only thing
that can
fail the step.

Measured, not assumed — the extraction was run against the real `zig
test`
output and three mutant logs:

| Input | Extracted | Gate |
|---|---|---|
| real run (`All 4 tests passed.`) | `4` | green |
| `All 0 tests passed.` | `0` | **red** |
| no summary line at all | `0` | **red** |
| empty log | `0` | **red** |
| `3 of 7 tests passed.` | `7` | green |

`awk` returns rc=0 on the no-match cases, which is the whole point of
the
change.

The one remaining `|| true` in `abi-codegen-drift.yml` — a diagnostic
`find`
inside an already-failing branch — is replaced by a `[ -d ]` guard for
the same
reason. There is now no `|| true` in either workflow.

### Not caused by this change

`governance / Actions lockfile verify` is red on every PR because #810
bumped 7
actions without refreshing `.github/workflows/actions.lock`
(`taiki-e/install-action` v2.87.13→v2.87.15, `ruby/setup-ruby`
v1.323.0→v1.324.0,
`docker/setup-buildx-action` v4.4.0→v4.4.1, and a `github/codeql-action`
SHA).
Every finding is on a workflow this PR does not touch. Filed separately
—
folding an unrelated repo-wide fault into this PR would make both harder
to
review.

The `Check` / `Test` / `Clippy` / `Cargo check + clippy + fmt` reds are
pre-existing on `main` (the unsatisfiable `testcontainers` constraint,
#814),
as are the three `governance /` reds.

### Acceptance

This PR is not done when it is merged; it is done when
`abi-codegen-drift`
reports **jobs > 0** and prints its denominator:

```
connectors emitted: 16
files compared:     3 of 3
```

Refs #120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Sep 22, 2026
… pass (#819)

## Why this exists

PR #817 was squash-merged at head `c82e9aa`, which carried only the
**first
half** of the fix. The second commit landed on the branch after the
merge and
is not in `main`. Measured, not assumed:

```
$ gh api repos/hyperpolymath/hypatia/commits/5060161.../check-runs
completed  failure  abi-codegen-drift
```

`abi-codegen-drift` — the gate #811 shipped — **is still red on
`main`**. This
PR carries the remainder.

## What was actually wrong

Running (not merely building) an Idris2 executable needs two support
artefacts,
and neither was present under `$(idris2 --libdir)` on the runner:

| artefact | when it is needed | symptom |
|---|---|---|
| `<libdir>/support/chez/support.ss` | compile | `INTERNAL ERROR: Can't
find data file chez/support.ss` |
| `<libdir>/lib/libidris2_support.so` | run | `(while loading
libidris2_support.so) cannot open shared object file` |

#817 cured the first by copying `support/` in as data. That is not
sufficient —
`libidris2_support.so` is a **built C library**, not data, and the Chez
backend
*copies it into the executable's `_app` directory at link time*. Fixing
an
incomplete installation file-by-file reveals one layer per CI round
trip, which
is exactly how the second failure happened. So this uses the tree's own
targets, which are complete by construction:

```sh
make -C "$src" support
make -C "$src" install-support PREFIX="$(idris2 --prefix)"
```

It builds `support/` only, never the compiler — seconds, not a
bootstrap.

### ⚠ It corrects the cause #817 recorded

#817 says library builds do not need the support tree, so the shared
cache
never had it. **That is false**, and the evidence was already in the
repo.
Idris2's top-level Makefile:

```
install: install-idris2 install-support install-libs
```

so `verify-proofs.yml`'s `sudo make install PREFIX=/usr/local` **does**
install
support — under `/usr/local/idris2-0.7.0/`. The binary resolves
`idris2 --libdir` to `/home/runner/.idris2/idris2-0.7.0`, and that job's
cache
`path:` list names *both* prefixes. It is a **prefix mismatch**. Library
work
(`--check`, `--build` on the two proof packages) never needs the
difference,
which is why only a job linking an executable ever noticed.

The fix is correct either way because it anchors on `$(idris2 --libdir)`
/
`$(idris2 --prefix)` rather than on a hard-coded prefix — but the
comment in
the workflow now says what was measured instead of the tidier story.

## Two further changes

**Assert on the consumer's artefact.** #817 asserted the *installer's*
output
and the job still died at run time. This asserts
`build/abi-gen/exec/hypatia-abi-gen_app/libidris2_support.so` — the
thing that
actually has to load it — because a build missing that copy still exits
0, and
the failure then surfaces one step later inside the comparison, where it
reads
as a drift failure rather than a toolchain fault. Plus a `--help` smoke
test:
the cheapest proof the binary runs, and it writes nothing, so it cannot
mask a
drift failure by leaving output behind.

**`Gen.idr`: `--help` exits 0.** It exited 1, so under `set -e` the
smoke test
would have failed and tempted an `|| true` — the exact pattern the
`Hypatia`
scanner flags and which is indistinguishable from a masked failure.
Missing
arguments are still a usage error and still exit 1.

## Verification done locally before pushing

- `--help` / `-h` → rc=0; no-args and one-arg → rc=1.
- Regeneration prints `connectors emitted: 16` / `files written: 3 of 3`
with
  **zero diff** against all three tracked targets.
- Mutants, against the gate's own comparison logic:

| case | compared | drifted | verdict |
|---|---|---|---|
| positive control (clean tree) | 3 of 3 | 0 | **green** |
| hand-edit generated Zig (`arrow_flight`→`arrowflight`) | 3 of 3 | 1 |
**red** |
| swap wire ids 3↔4 in the live `Types.idr`, no regen | 3 of 3 | 3 |
**red** |
| empty output dir | 0 of 3 | 3 | **red on the denominator** |

The third is the decisive one: it proves the gate reads the **normative
source**, not merely that the generated copies agree with each other.

## Note on caching

The repair is deliberately **not persisted** — `Post Cache` is skipped
on a
cache hit, so this job never mutates the shared entry and redoes the
clone+build
each run. That is the intended property and should not be "optimised"
away;
two workflows writing one cache key is how the poisoned `-1` cache
happened.

## Not blockers (pre-existing on `main`, per the standing ruling)

`governance / Actions lockfile verify` (#818, from #810) · `Check` /
`Test` /
`Clippy` / `Cargo check + clippy + fmt` (#814) · `governance /
{Code quality + docs, Validate Hypatia Baseline, Workflow security
linter}` ·
`CodeQL Security Analysis` startup failure. None are touched by this
diff.

Refs #120, #811, #817

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_0113HQM9LVGkNCzU1WwkJZSV

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants