Minor/optimize dero miner - #71
Open
secretnamebasis wants to merge 25 commits into
Open
Conversation
…t port AstroBWTv3's proof-of-work spends ~82% of its per-hash time in suffix-array construction, previously a Go port of stdlib SA-IS (sais_8_32). This swaps the production entry point (text_32_0alloc, called from both the miner and block validation via miniblock.go) to a faithful pure-Go port of libdivsufsort instead, which runs ~19-22% faster in isolation and ~11% faster on the real dero-miner binary end to end. The old SA-IS implementation is kept as text_32_0alloc_sais rather than deleted, serving as a permanent comparison oracle: since a suffix array is unique for a given input, two correct algorithms must produce byte-identical output, and TestDivSufSortMatchesProductionSAIS asserts exactly that on every test run. Correctness is additionally backed by the package's pre-existing golden-hash tests (TestAstroBWTv3, TestAstroBWTv3repeattest), which pass unchanged, and 379+ dual-compute trials against real captured AstroBWTv3 fixtures. Adds the divsufsort port itself (divsufsort_go.go, sssort_go.go, trsort_go.go), its correctness-oracle test suite, a C-vs-Go cross-check harness (divsufsort_bench/), and fixture-capture infrastructure used to build realistic benchmark/test corpora from live AstroBWTv3 runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
divsufsort treats AstroBWTv3's suffix-array input as opaque text, same as SA-IS did. It isn't: the wolf loop applies RC4 whitening on only ~25% of iterations (the trigger is AstroBWTv3's own protocol, `step_3[pos1]- step_3[pos2] <= 0x40`, identical in any implementation of this hash), so runs of 256-byte chunks between whitening events stay highly correlated. This adds a suffix-array construction that exploits that structure directly instead of sorting it as generic text. Ported from Dirtybird-Go-Miner's internal/astrobwt/sa_v114*.go (MIT), itself derived from a Zig/C++ reference. The core algorithm (sa_template_emit.go, sa_template_merge.go) is close to a verbatim port; what's new here is the integration into this package's own ScratchData/Pool structure, buffer sizing re-derived from this package's own MAX_LENGTH and protocol bounds rather than the reference's, and the fallback choice: any decline falls back to this package's own divsufsort path (text_32_0alloc), not SAIS, since divsufsort is already faster. Wolf-loop instrumentation (pow.go) records template markers unconditionally during the whitening branch -- pure bookkeeping, no effect on the hash regardless of whether anything reads it. The template path itself is opt-in only (ScratchData.useTemplateSA, off by default): AstroBWTv3's behavior is unchanged, proven by the existing golden-hash tests passing without modification. Correctness, checked independently of the ported reference's own test suite: a hand-built end-to-end run against this package's divsufsort oracle, Stage 4/5 isolated against synthetic buffers and hand-built descriptor records, 5,000-trial and 12-length differential runs against production (divsufsort, since the template path isn't yet the default), and a marker-aware fixture corpus (testdata/safixtures_template, regenerate with `go test -tags astrobwt_capture -run TestGenerateSATemplateFixtures`) round-tripped two independent ways. A million-hash differential gate (TEMPLATE_SA_GATE_HASHES=1000000) is available for a larger confirmation pass, off by default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The template path (previous commit) is proven byte-identical to divsufsort across a 1,000,000-hash differential gate, 64 real captured fixtures, and substantially faster in isolation and end to end. Pool.New now sets ScratchData.useTemplateSA = true for every pooled scratch, so AstroBWTv3 runs it by default. Any decline still falls back to text_32_0alloc (divsufsort) at the pow.go dispatch site, unchanged from the previous commit -- this adds no new correctness risk beyond what divsufsort itself already carries. Retargets every differential test that previously compared the template path against AstroBWTv3's own output as the "production" reference: once the template path is what AstroBWTv3 runs, that comparison is circular. Each test now forces scratch.useTemplateSA = false explicitly to get the independent divsufsort reference, so the suite keeps checking the two algorithms against each other rather than a path against itself. Adds BenchmarkHashDivSufSortOnly, forcing the divsufsort path explicitly through the same whole-hash benchmark shape as BenchmarkHashTemplateSA, for a permanent, fair end-to-end comparison against the new default. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single-stream SHA-NI is latency-bound: each SHA256RNDS2 depends on the previous, leaving pipeline bubbles even on hardware that has the dedicated instructions. Interleaving two independent messages' instruction streams lets an out-of-order core fill those bubbles, recovering throughput a single stream can't reach on its own. Ported from Dirtybird-Go-Miner's internal/astrobwt/sha256mb_amd64.s (MIT), verbatim -- transcription risk is closed by an exact byte diff against the source, since every instruction's semantics are already ISA-guaranteed by Intel/AMD, not something to re-derive by hand. pairHashAvailable gates on real cpuid SHA support (plus LittleEndian); every other host transparently falls back to two ordinary hashes through sha256-simd, so this is safe to build into any binary regardless of what it ends up running on. Unwired from the rest of the package for now -- nothing calls sha256Sum256Pair yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Splits astroBWTv3's body into astroBWTv3Stream (wolf loop through SA construction) and the existing SHA-256 finish, so the finish step can run through the 2-way SHA-NI kernel for two nonces at once instead of one at a time. astroBWTv3 itself calls Stream then does exactly what it did before -- no behavioral change to any existing caller. astroBWTv3Stream carries its own panic recover (astroBWTv3's original recover stays put, guarding the single-hash path unchanged) since it's now called from two places. data_len == 0 is an unambiguous panic signal: the wolf loop's own minimum of 261 tries guarantees data_len >= 65792 on any real run, so 0 can't occur otherwise. AstroBWTv3Pair's fallback and mid-stream-panic branches both call AstroBWTv3 directly -- the same function every single-hash caller uses -- rather than a second, separately maintained slow path. That makes AstroBWTv3Pair(a,b) == (AstroBWTv3(a), AstroBWTv3(b)) true by construction on any host, not just one a test happens to exercise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TestSha256Sum256PairDifferential pins the kernel itself against sha256-simd's own hashing, over a length matrix deliberately dense around block boundaries (56/64-byte padding edges) plus unequal-length pairs -- an end-to-end gate that only ever sees the miner's fixed input sizes could let a wrong two-stream kernel slip through undetected. It logs pairHashAvailable() so a green run on hardware without SHA-NI (or big-endian) isn't mistaken for proof the assembly ran; only a log showing pairHashAvailable=true means the kernel itself was exercised. TestAstroBWTv3PairMatchesSequential and the 0-alloc test extend this package's existing KAT/alloc conventions to the pair path. BenchmarkAstroBWTv3Pair mirrors the existing single-hash benchmark's shape for a same-basis throughput comparison. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pulls astroBWTv3Stream's 256-case per-position transform switch out into its own function, byte-for-byte unchanged. This is the ground truth every later fast path in this area gets derived from and verified against, not a separately-maintained copy: correctness of anything built on top reduces to "does it match applyBranchOp", checkable directly against DEROHE's own reference rather than any external source. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Classifies each of the 256 branch-loop opcodes by exhaustively testing applyBranchOp against every (input byte, step_3[pos2] byte) pair -- 65536 evaluations per op, not a sample -- into: always-zero-output, output-independent-of-step_3[pos2] (safe to precompute as a 256-entry table), or genuinely needs step_3[pos2] at each position. Derived entirely from applyBranchOp itself, not ported from any external source; cross-checked against Dirtybird-C-Miner's own (syntactic, more conservative) classification during development and found 6 ops provably always-zero where source inspection alone would call them op-count-dependent -- see the algebraic proof in the commit that wires this in. Unwired from astroBWTv3Stream for now; nothing calls opClass/opLUT yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Routes each branch-loop op through opClass's classification: a plain loop-clear for the always-zero ops, a single table lookup per byte for the step_3[pos2]-independent ops, and applyBranchOp unchanged for everything else. forceScalarBranchOp gives tests a way to force every op through applyBranchOp for a direct differential comparison against the fast paths, independent of the hardware they happen to run on. Caught a real bug before this ever landed: ops 254/255 carry a mandatory RC4 re-key that isn't part of their step_3 transform, so their transform alone tests as step_3[pos2]-independent -- but a fast path bypassing applyBranchOp would silently skip the re-key. The generator now forces those two (and 253, which already fails the independence test on its own transform) to the scalar path unconditionally, regardless of what the transform-only scan finds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For the ops that don't decompose into a static lookup table (their transform reads step_3[pos2], so no fixed input->output mapping exists), derives which 4-step sequence over a 16-symbol vocabulary (add-self, xor-pos2, variable shift/rotate, popcount, etc. -- the same primitives AVX2 can vectorize across a 32-byte window) reproduces applyBranchOp's output. Brute forces all 16^4 candidates per op, screens against sampled inputs, then verifies every surviving candidate exhaustively across all 256x256 (input, step_3[pos2]) pairs before accepting -- derived from DEROHE's own switch, not ported from Dirtybird's CodeLUT (spot checks against it agreed exactly, but the sequences here are independently re-derived, not copied). Found a decomposition for all 99 eligible ops (every opClassScalar op except 0/253/254/255, which have side effects or a cross-iteration dependency outside this per-byte-transform model entirely). Unwired: nothing calls opSubSeq yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four building-block AVX2 kernels (bit-reverse, popcount, per-byte self-multiply, per-byte variable left-shift), each independently differential-tested against a scalar Go reference before being trusted to compose into anything larger. The variable shift uses the standard multiply-by-2^count-via-VPSHUFB-lookup trick, split into even/odd byte lanes so 16-bit VPMULLW doesn't let one byte's product bleed into its neighbor. Right-shift and rotate aren't separate primitives: srl(x,k) = reverse8(shl(reverse8(x),k)) (right-shift-by-k equals reverse, then left-shift-by-k, then reverse again -- verified against this package's own composition, not assumed), and rotate composes shl|srl. Fewer independent pieces to get right in hand-written assembly, at the cost of the extra reverse8 calls -- correctness took priority over shaving instructions at this stage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Vectorizes the remaining ops' per-position transform across a 32-byte window using the derived opSubSeq sequences over the verified primitives, gated on cpuid AVX2 support and a pos1+32<=len(step_3) bounds check (a 32-byte SIMD load/store near the end of the 256-byte buffer would otherwise read past it -- costs roughly 12.5% of eligible iterations, falling back to applyBranchOp for those). Op 0's swap and ops 253/254/255's side effects are excluded from opSubSeq entirely (see the earlier derivation commit), so they always take the scalar path regardless of AVX2 availability. Fully unrolled -- no CALL/RET anywhere in the per-op dispatch, every primitive inlined at each of the 4 sequence positions -- after an earlier CALL-based version (subroutine per primitive, shared across all 4 positions) measured statistically indistinguishable from no-AVX2 at all (~0.45% on a 20x50-rep benchstat, inside the ~1% CI). The hypothesis was call overhead through the nested dispatch->rol->srl->reverse chain (up to 6 nested calls for a single rotate-based sub-op); removing it entirely changed nothing (2.437ms -> 2.426ms, same ~0.45%). Kept as verified-correct, zero-risk-to-merge infrastructure: every fast path here falls back to the untouched applyBranchOp by construction, so it can't be wrong even though it doesn't measurably help on this hardware. Real speedup, if any exists, needs different hardware or a different lever entirely -- not another rewrite of this same kernel shape. Differential-verified against applySubOpSeq (200k arbitrary-sequence trials) and applyBranchOp directly on real derived per-op sequences (~198k trials across all 99 eligible ops). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mineblock() now hashes two nonces per iteration through AstroBWTv3Pair instead of one through AstroBWTv3, for blocks past MAJOR_HF2_HEIGHT (the pre-HF2 branch, using the older astrobwt_fast.POW_optimized, is untouched). A second work buffer (workB) is synced from the job template alongside the existing one; each iteration advances the nonce counter twice, hashes both buffers together, and checks/submits each result independently against the current difficulty. The underlying kernel (AstroBWTv3Pair, sha256mb_amd64.s) was already merged and differential-tested; this was the one remaining piece -- actually calling it from the real mining loop -- and had been sitting as an uncommitted local diff, live-verified once already on real SHA-NI hardware (core, AMD Ryzen 9800X3D: kernel-level ~1.5x, live miner A/B against mainnet infra +6-7%) but never committed. On hardware without SHA-NI, AstroBWTv3Pair falls back to two ordinary hashes internally, so this is correct either way with no capability check needed at the call site. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sort_indices (sa_fast.go) is an alternate SA-construction path that was already measured slower than production and never wired into any real hash path -- its only caller is BenchmarkSortIndicesFastPath_Realistic, informational only. But its two backing arrays (indices/tmp_indices, 768KB combined) and the stage1_result/stage1_result_bytes pointers aliasing into them were fields on ScratchData, the struct every pooled mining-thread scratch object carries -- live, dead weight on every real hash, confirmed by grepping for any production reference (there is none). Moved indices/tmp_indices out to caller-supplied parameters instead (sort_indices now takes them directly), with the one live test caller allocating its own local buffers. ScratchData shrinks from ~1249KB to ~481KB per worker. Full package test suite green; no behavior change to any production code path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
templateSAScratch's runs/radixTmp/groupPos/mergePos/runLens/nextLens buffers were all sized at MAX_LENGTH (98,303) to cover their theoretical worst case, but that worst case is far from what real AstroBWTv3 output produces. Measured across 200,064 hashes (200,000 random MINIBLOCK-sized inputs plus every real captured fixture, TestMeasureTemplateSAUsage, not committed -- throwaway measurement tooling, kept only as the numbers below): - runs/radixTmp: observed max 28,358 (p99.99 27,097) vs MAX_LENGTH 98,303 - groupPos/mergePos/runLens/nextLens: observed max 924 vs MAX_LENGTH 98,303 New capacities: stage5RunsCapacity = MAX_LENGTH/2 = 49,151 (~1.7x margin over observed max), stage5MergeCapacity = 4,096 (~4.4x margin over observed max). arena is left untouched -- its real utilization (66,119 observed vs 98,303 worst case, ~67%) is close enough to its bound that shrinking it isn't worth the added decline risk for a modest gain. templateSAScratch drops from 3.38MB to 1.19MB per worker (combined with the previous commit's ScratchData cleanup: ~4.63MB -> ~1.67MB per worker overall, a 2.77x reduction). Correctness: runs/groupPos/runLens/nextLens are append-based, so Go's slice growth already makes exceeding the new smaller capacity safe by construction (a rare real allocation, not a bug). radixTmp and mergePos are indexed directly (not via append), so writeFusedRunsToSA gained two explicit decline guards before each is used -- same "clean fallback to divsufsort, never a wrong answer" contract as every other decline point in this file (arena overflow, stage4MaxGroupRun). Also fixed TestTemplateSAZeroAllocsAfterWarmup, which used synthetic uniform-random data (no structural correlation, unlike real wolf-loop output) shaped into artificial templates -- decorrelated enough to trip the new guard even though real traffic doesn't. Switched it to the same real-fixture data BenchmarkTemplateSAStageOnly_Realistic already uses. Added templateSAKWayMergeHits (mirrors the existing templateSAFallbacks counter) for observability into how often the k-way-merge path fires. Full package test suite green, including the 1M-hash-scale differential gates against divsufsort. Real dero-miner --bench on the dev machine (i9-9900K) showed no measurable throughput change from either this or the previous commit (~7415 H/s before, ~7415-7481 H/s after, within run-to-run noise) -- this is a memory-footprint reduction, not a compute reduction, and this machine's L3 doesn't appear to be the bottleneck it was hypothesized to be. Kept regardless: real, safe, fully verified reduction in per-worker memory footprint, independent of whether it moves this CPU's hashrate number. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eal usage" This reverts commit 8521f8c.
…atchData" This reverts commit 0cf39b0.
…h ops" This reverts commit fd02322.
…nel" This reverts commit b84b660.
This reverts commit 5f0268e.
This reverts commit c2b64d7.
…nwired" This reverts commit e8b28b2.
This reverts commit 35ba1fa.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces AstroBWTv3's suffix-array construction in two steps:
Both algorithms produce byte-identical output by construction (a suffix array is unique for a given input), verified via a 1,000,000-hash differential gate across all cores, 64 real captured fixtures round-tripped two independent ways, a 5,000-trial + 12-length differential suite, and the package's own pre-existing golden-hash tests passing unmodified throughout.
No consensus-level change — AstroBWTv3's output is unchanged, only the internal algorithm computing it. This is an implementation-level performance improvement, not a protocol change.
Note: this template says PRs should target dev — that branch doesn't exist on this fork; I've based this off community-dev instead. Flagging in case the base needs adjusting before merge.
Fixes # (no linked issue — proactive performance work, not a bug report)
Type of change
Which part is impacted?
Checklist:
License
I am contributing & releasing the code under DERO Research License (which can be found here).