translator: expand multi-architecture coverage from ecosystem discovery

[cpunion]

Summary

• Expand Plan 9 assembly lowering across all supported architectures (386, amd64, arm, arm64, and wasm) from Go 1.27 tables, with positive/negative form tests and LLVM 22 object checks.
• Make the Go module discovery result a durable, diff-friendly checkpoint: every inspected exact version is recorded, including versions with no assembly; matched versions retain every module-relative .s path for current and future platforms.
• Compile every applicable assembly file from all 661 discovered assembly-bearing versions. The final 32-shard run has zero failures.
• Keep the 28 issue/curated modules as a separate fast regression corpus, updated to exact current @latest versions.
• Run discovered-corpus CI in 32 shards on the Qiniu Linux runner for xgo-dev, using LLVM 22 only.

Discovery funnel

The committed checkpoint is one manifest.json plus 256 sha256(module)[0] JSONL record shards. Files are sorted by module path, Go semantic version, and record kind. There are no run directories, gzip files, or ZIPs in the final PR history.

Stage	Count	Meaning
Official index records consumed	320,400	Contiguous scanner input accumulated in this checkpoint
Successfully inspected exact module@version records	117,891	Includes versions with no assembly, so they are not downloaded again
Unique module paths	58,807	Every new path is resolved through module@latest before its exact-version seen check
Exact versions containing assembly	661	8,380 saved .s paths in 1,758 module-version/source directories
Retryable failure records	4,180	Resolution/download errors remain retryable and are never marked complete
Discovery corpus candidates selected	661	All assembly-bearing exact versions, split deterministically across 32 jobs
Passed candidates	510	At least one applicable target translated and compiled
Evidence-backed source N/A candidates	151	Current Go 1.27 rejects the source/package on every applicable supported target
Failed candidates	0	No unsupported instruction, parse, translation, LLVM, or infrastructure failure remains
Successful LLVM 22 translations	7,800	Per-file/per-target translations
Evidence-backed target N/A translations	283	Target-specific Go filename/build/ABI rejection, retained in JSON reports

All applicable discovered assembly passes translation and LLVM 22 compilation. N/A is accepted only with structured Go source/build/assembler/asmdecl evidence; it is not an unsupported-form escape hatch.

Scope of the public index

The separate census at the fixed 2026-09-13 cutoff counted 55,128,672 default-index module@version entries and 2,856,345 unique module paths. This PR's checkpoint covers:

Scope	Available at cutoff	Recorded here	Coverage
Default-index entries	55,128,672	320,400	0.581186%
Unique module paths	2,856,345	58,807	2.058820%

This measures the public default Go module index, not include=all, private modules, or modules never served by the Go services.

Future-platform replay

asm_files is the durable fact source; architectures is a derived query index. It includes all current Go architecture suffixes, not only architectures currently lowered by plan9asm. Unknown, unsuffixed, and custom-suffixed files are conservatively retained.

For a newly supported platform, PLAN9ASM_DISCOVERY_TARGETS=<goos>/<goarch> queries only the saved matched exact versions, derives candidate package directories from their saved paths, and then reapplies the current Go filename/build constraints before translation. It does not read the module index or revisit the 117,230 no-assembly inspected versions. A real linux/arm64 replay reduced 661 matched versions to 316 eligible candidates; shard 30 selected 11, passed 7, classified 4 with source evidence, and failed 0.

Independently rediscovered issue libraries

These are selected by the generic ledger path, not by the curated manifest:

Issue	Exact version	Saved assembly	Discovery shard result
llgo#2464	github.com/coder/websocket@v1.8.15	2 files	shard 7: passed, 6 translations
llgo#2552	github.com/klauspost/compress@v1.20.0	14 files	shard 28: passed, 42 translations
llgo#2576	github.com/tmthrgd/go-hex@v0.0.0-20190904060850-447a3041c3bc	2 files	shard 28: passed, 6 translations

The separate curated corpus contains 28 libraries and also passes across its 11-target matrix. github.com/phuslu/log is updated to v1.0.133 after the final @latest check.

Instruction and form coverage

Development followed red/green TDD: each external or official-corpus failure was reproduced first, then the complete Go 1.27 instruction family and legal/illegal operand rows were checked before lowering.

• 909 additional Go 1.27 opcode spellings are supported relative to main (895 added after the earlier PR head, with no supported-op regression).
• 4,596 official Go 1.27 form rows move from unsupported to supported relative to main.
• Go 1.20 through Go 1.27 coverage reports pass for every supported architecture, with zero parse failures and zero supported-form regressions.
• The Go 1.27 standard-library corpus passes all 44 GOOS/GOARCH/feature combinations with no unsupported instruction or operand form.
• No xfail was added. Corpus failures remain hard failures; only structured source/target N/A evidence is accepted.

Major completed families include:

• x86: vector move/arithmetic/logical/min-max, shifts/rotates, comparisons and masks, pack/unpack/shuffle/permute/blend/broadcast, gather/scatter, AES/VAES/CLMUL, conversion/rounding, x87, string and port I/O, scalar multiply/divide, ADC/SBB/carry, and 386-only BOUND.
• ARM: scalar integer and floating arithmetic, conversions, shifts/div/mod, comparisons and VMRS, general/F-register transfers, prefetch, and architecture-aware CFG lowering.
• ARM64: scalar/vector floating operations, comparisons, widening shifts/adds, bit count/extend, permutations, atomic pairs, negate/carry families, and return-width-safe tail calls.
• wasm: Go ABI/global handling, metadata directives, and complete F32/F64 unary lowering.
• Parser/driver: Go parenthesized ARM register expressions, historical declarations/frame slots, safe GOROOT includes, exact target selection, and opcode-map-aware support extraction.

The exhaustive encoder reports still classify currently unimplemented table rows explicitly rather than skipping them; this PR claims completeness for the discovered external corpus and the Go standard-library matrix, not every decoder-only/unused optab row.

Validation\n\nFinal CI: run 34897723166 completed 91/91 jobs successfully with no failed or cancelled job.\n\n- go test ./... -count=1, plus both nested command modules

• go test ./... -race -count=1
• root and nested go build, go vet, and go mod verify
• native Go amd64 and LLVM 22 executable semantic conformance
• all 32 discovery shards: 661 selected / 510 passed / 151 source N/A / 0 failed
• all 28 curated libraries across 11 targets
• Go 1.27 standard library across all 44 target/feature variants
• Go 1.20–1.27 official assembler regression matrix for 386, amd64, arm, arm64, and wasm
• ARM64 x/arch Plan 9 decoder corpus
• actionlint, shellcheck, gofmt, git diff --check, ledger integrity/dedup/semver/hash tests, and no committed .gz/.zip
• statement coverage: root 85.8%, cmd/plan9asm 31.7%, cmd/plan9asmll 51.4%; Codecov patch coverage 88.09% (target 87.59%)

CI and Codecov now pass. This PR intentionally remains Draft until review passes.

[codecov]

Codecov Report

❌ Patch coverage is 88.09050% with 837 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
amd64_lower_fp.go	64.98%	139 Missing ⚠️
amd64_ctx.go	78.31%	54 Missing ⚠️
amd64_lower_compare_vec.go	85.13%	33 Missing ⚠️
amd64_lower_broadcast_block.go	71.42%	28 Missing ⚠️
amd64_lower_compare_float.go	88.70%	27 Missing ⚠️
amd64_lower_move_sign_extend.go	81.42%	26 Missing ⚠️
amd64_lower_extract.go	74.48%	25 Missing ⚠️
amd64_lower_mul_dword_qword.go	84.10%	24 Missing ⚠️
amd64_lower_cmpxchg.go	90.04%	21 Missing ⚠️
amd64_lower_branch.go	74.68%	20 Missing ⚠️
... and 53 more

📢 Thoughts on this report? Let us know!

[fennoai]

Review summary

Large PR (~175k insertions across 518 files). The bulk is generated discovery testdata (testdata/discovery/ledger/**) and highly repetitive AMD64/ARM instruction-lowering tables, both of which I skimmed rather than reviewed line-by-line. I focused on the high-signal changes: the new discovery/scan tooling (cmd/plan9asmcorpus/discovery.go, cmd/plan9asmdiscover, cmd/plan9asmll, cmd/plan9asmscan), internal/discoverymeta, and the core translate/parser changes.

Overall this is solid, defensively-written code: file handles are consistently closed on error paths, zip extraction blocks zip-slip (proxy-prefix + path.Clean + filepath.Rel + O_EXCL), external commands use fixed argv with escaped module paths, and remote fetches are size-bounded. Doc/AGENTS.md claims (flags, scripts, committed manifest counts, test names) were verified accurate.

One inline correctness/consistency finding below, plus a few non-blocking notes.

Non-blocking notes:

• Exponential build-tag search — cmd/plan9asmcorpus/discovery.go (findDiscoveryBuildTags). The search iterates 1 << len(customTags) masks, each doing filesystem-backed ctx.MatchFile calls, and the outer wantedCount loop re-walks the mask space ~len(customTags)+1 times. With the 16-tag cap this is bounded but can reach millions of MatchFile calls for a package near the limit. Usually fine (tags are typically 0–2); consider iterating masks once (prioritizing by popcount) if large-tag modules become common.

• Symlink escape in local include expansion — cmd/plan9asmll/main.go (pathWithinRoot / expandAsmIncludes). Containment is checked lexically via filepath.Rel; a #include target that is a symlink pointing outside the source root would pass the check and be read (no filepath.EvalSymlinks). Blast radius is contained (read-only, inlined into a discarded IR artifact, ephemeral CI against public modules; the zip extractor already rejects non-regular files), so this is defense-in-depth only. Resolving symlinks before the containment check would close the gap.

• Readability — cmd/plan9asmll/main.go:988 (and :1023). op == "JMP" || op == "B" || op == "RET" && len(ins.Args) == 1 relies on && binding tighter than ||. The current behavior is correct, but explicit parentheses around (op == "RET" && len(ins.Args) == 1) would guard against a future || term silently changing the grouping.

[fennoai]

[P2] PSUBUSB/PABSD emit source IR before the loadX fall-through

loadXVecOperand(ins.Args[0]) emits IR into c.b before dst, err := c.loadX(ins.Args[1].Reg), whose error branch returns false, false, nil ("unhandled", fall through to other lowerers). If Args[1] is a non-X register, the already-emitted source-load IR is left orphaned in the buffer and a later lowerer may re-emit the same operand.

Contrast the PMINUB/min-max cases just below (line ~2446), which load dst first and are therefore safe. PTEST (line 2414) has the same ordering as here.

Suggest validating that Args[1].Reg is an X register before emitting any source IR (mirroring the min/max pattern), so the fall-through happens before IR is written. Impact is low in practice (the parser rarely yields a non-X second operand for these P-prefixed ops), but the ordering is inconsistent with the sibling cases.