From 82da80bfff1d76b852d286f6b3e1834a4df2a8f8 Mon Sep 17 00:00:00 2001 From: syntrust Date: Thu, 6 Aug 2026 18:53:39 +0800 Subject: [PATCH 1/6] tx execute doc --- L1/tx-exec.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 L1/tx-exec.md diff --git a/L1/tx-exec.md b/L1/tx-exec.md new file mode 100644 index 0000000..5f5cbe2 --- /dev/null +++ b/L1/tx-exec.md @@ -0,0 +1,153 @@ +# goshard Transaction Execution + +## Context + +For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute **byte-for-byte identical** state root / receipt root / gas used / xshard cursor / coinbase amount map / bloom. + +Today `qkc/` already has the static half: account leaf encoding, transactions and signatures, minor block header/meta, receipts, bloom, token balances, `CrossShardTransactionDeposit`, `XShardTxCursorInfo`, genesis ALLOC → state root (golden pinned down and passing). What's missing is the **dynamic half**: a mutable QKC StateDB, an EVM, and the `apply_transaction` / `apply_xshard_deposit` / `run_block` execution path. + +This plan delivers that complete path, covering both the intra-shard and cross-shard ends, and compares results against pyquarkchain block by block. + +## Goals / Non-goals + +**Goals**: make `qkc/core` a full Go counterpart of pyquarkchain's `evm/messages.py`, plus the **pure-execution** layer of `shard_state.py`: + +- `ValidateTransaction` + `ValidateTxForBlock` (`__validate_tx`), including the `version == 2` (EIP155) branch +- `ApplyTransaction`: intra-shard branch + `is_cross_shard` branch (producing deposits) +- `ApplyXShardDeposit` + cursor traversal +- `Process`: the counterpart of `run_block`, producing state root / receipts (regular + deposit) / `gas_used` / `xshard_receive_gas_used` / new cursor / coinbase map / bloom +- `ValidateBlockResult`: the counterpart of the **result** comparison in `add_block` +- **Reading the POSW `sender_disallow_map`**: walk back `WINDOW_SIZE` along the header chain, counting coinbase occurrences × `TOTAL_STAKE_PER_BLOCK`. This is the only gap that **does not fail loudly** — with an empty map, `transfer_failure_by_posw_balance_check` raises no error, it merely lets a transfer that should have failed succeed. On mainnet the **shard-level** `POSW_CONFIG` is in effect from genesis for chains 1–7 (chain 0 is off), so without this their replayable range is zero; the cost is just one header walk of `WINDOW_SIZE` (256/512), fully decoupled from difficulty adjustment / staking / boost +- **Registering QKC precompiles `01/02/03`**: they are enabled together with `ENABLE_EVM_TIMESTAMP`, fall inside the replay window, and are equally callable for the default QKC token (`PrecompiledContractsAfterEvmEnabled` in `qkc/params/evm_params.go` is already exactly these three addresses). Without registering them, geth treats them as empty accounts and CALLs silently succeed + +On the token side, coverage is full EVM semantics for transfers and contract CREATE/CALL of the **default token (QKC)**; `refund_rate` / `gas_token_id` are carried end-to-end as **pipeline fields** and participate in refund and burn calculations, but the exchange-rate conversion itself (`pay_native_token_as_gas`) is not implemented. + +**Non-goals (follow-up tasks)**: + +- Cluster/chain layer: intra-cluster broadcast and deposit of the xshard list, the gating checks of `add_root_block`, tip updates and fork choice, `create_block_to_mine`, tx pool +- **All of `validate_block`** — the structural checks that come before `run_block`: header version/height, prev block existence, gas limit, transaction count and size, merkle root, timestamp, difficulty, `hash_meta`. This plan only covers `run_block` + result comparison +- Multi-token transfers, the `GENERAL_NATIVE_TOKEN` system contract and `pay_native_token_as_gas`, the **MNT precompiles `0x…514b430004/05`** (`enable_ts` set to never-enabled) +- POSW **computation**: difficulty adjustment, staking, `BOOST_*`, decay. Only the disallow-map read described above +- Wiring into the `ShardChain` seam in `qkc/shard`, replacing `StubChainService` + +Non-goal items are rejected in code with an explicit `error` (rather than silently skipped), so that "unsupported" is never mistaken for "results match". + +## Architecture and reuse strategy + +Three new packages, zero modifications to geth source: + +``` +qkc/vm ← copied from geth core/vm and trimmed (Petersburg-only + QKC semantics) +qkc/state ← newly written, minimal mutable StateDB, backed by geth trie/triedb +qkc/core ← newly written, Validate / ApplyTransaction / ApplyXShardDeposit / Process +``` + +**Execution results are not determined by the parent state alone.** Three inputs live outside the state root, so they must be passed in explicitly through an `ExecutionContext` rather than having `qkc/core` reach back into node state: + +- the node's **root tip at the time** — `__validate_tx`'s "target shard already has genesis" check and `_is_neighbor` read this, not the block's `hash_prev_root_block` +- the ability to **look up minor headers by hash** — BLOCKHASH's 256 ancestors, plus the POSW window +- the **disallow map** computed from that window + +Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate` (internally `Process` + comparison, returning an error if any item mismatches, so callers can never get hold of a StateDB that has been committed but is already invalid). `ValidateBlockResult` is a separate function so replay can call it directly. + +Cross-shard data is fed in through an `XShardSource` interface. **It needs root block bodies (including the minor header list), and `qkc/types/rootblock.go` currently only has `RootBlockHeader`** — adding that type is a prerequisite and can be done in parallel with this plan. The real implementation (database reads/writes) belongs to the chain-layer task; the interface shape needs to be aligned with the owner of issue #1. + +The two validation layers are not a simple "validate then execute" sequence: `ApplyTransaction` calls `validate_transaction` **a second time** internally, while `ValidateTxForBlock` inside `run_block` returns early in the future-nonce range and skips it. Implementing this as "validate once" would behave differently on future-nonce transactions. + +## Step-by-step implementation + +### S0 — golden generator +Export three tiers of vectors — state level, message level, block level — from a pyquarkchain venv. The first case is fixed as a no-op genesis ALLOC whose post root must equal the two-network values in `minor_genesis_golden.json`, which calibrates the generator itself. + +### S1 — qkc/state mutable state layer +Today there is only the 33-line leaf struct in `account.go`. Balances are indexed by token id, `FullShardKey` goes into the leaf, and existence/deletion rules are their own thing — geth's `core/state` cannot be reused; estimated 1200+ lines on the Go side. + +### S2 — qkc/vm copy and trim +After copying geth `core/vm`, change four things: CREATE address derivation, `StateDB` carrying a token id, registering QKC precompiles 01–03, and removing post-Petersburg forks. The first is the one that must be changed in the copy and cannot be handled by injection. + +### S3 — intra-shard ApplyTransaction (pure transfers) +Implement all of `validate_transaction` except non-default gas token conversion, including the `version == 2` EIP155 branch — outside the mainnet replay window, but devnet enables it at genesis, so the branch is exercised from the first golden case onwards. + +### S4 — EVM integration (CREATE / CALL) +Assemble `BlockContext` / `TxContext` / message and hook up `qkc/vm`, feeding in `full_shard_key` for the address derivation from S2 to use. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. + +### S5 — cross-shard source side +The `is_cross_shard` branch: debit, produce the deposit, compute the address on the source shard for cross-shard deployments, and burn all gas on POSW failure. Charging and refunding the 9000 is gated by `ENABLE_EVM_TIMESTAMP`. + +### S6 — cross-shard target side +`RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. + +### S7 — Process(block) and block-level settlement +The three-level cursor traversal and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees — counting only the former misses part of it. + +### S8 — ported tests and differential random testing +Layers 2 and 3 of the verification plan, as a step of their own: port the behavior checklist from `test_shard_state.py`, then run randomly generated (alloc, tx sequence, deposit sequence) triples against both implementations. They need the whole path finished, but nothing from the chain layer — so they are the last step this plan owns. Layer 4 (historical replay) is not a step here; it lands with the chain-layer task. + +### Acceptance per step + +| Step | Acceptance | +|---|---| +| S0 | No-op genesis case, post root matches the golden for both networks | +| S1 | ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip | +| S2 | Compiles standalone; the Petersburg-and-earlier subset of geth's EVM unit tests passes | +| S3 | Transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase | +| S4 | Contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom) | +| S5 | Compared on root + receipt + `gas_used` + the produced deposit **field by field** | +| S6 | Must include three items: post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP ±1` boundary | +| S7 | Cut-and-resume of a single root block's deposit list consumed across two minor blocks; N consecutive chained blocks after genesis | +| S8 | The ported checklist passes; differential runs agree with pyquarkchain on root and cursor, and every divergence found is reduced to a golden case | + +## Ordering and parallelism + +``` +qkc/types root block body ─────┐ +S0 golden generator ───────────┤ +S1 state ──┐ │ +S2 vm ─────┴─> S3 ─> S4 ─┬─> S5 ─┐ + └─> S6 ─┴─> S7 ─> S8 ─> replay + ↑ + XShardSource real implementation (chain-layer task) +``` + +Parallelizable: root block body with S0; S1 with S2; S5 with S6 (source and target sides each rely on message-level golden and are independent of each other). +The earliest external unblock is S1 — once `qkc/state` lands, other tasks can depend on it. +**Person-day estimates TBD** (this plan gives dependency order only). + +## Hard-fork switches + +They are not "historical baggage a new chain can ignore" — replaying mainnet history requires keeping them, so it is worth writing against the switches from the start. This table doubles as the implementation checklist for S3–S7 and as the basis of the replayable range (timestamps and POSW config are in `qkc/config/singularity/mainnet.json`): + +| Switch | mainnet | devnet | Where it applies | +|---|---|---|---| +| genesis | 2019-04-30 | same | Start of the replayable range | +| `ENABLE_TX_TIMESTAMP` + `TX_WHITELIST_SENDERS` | 2019-06-29 | **0** | Before this, only whitelisted addresses could send transactions; devnet has no such phase | +| `ENABLE_EVM_TIMESTAMP` | 2019-09-27 | **0** | **Six places**: three differences in cross-shard gas settlement, `__validate_tx` forbidding contract transactions, whether cross-shard receiving takes the pre-EVM fixed-amount path or the EVM path, and enabling QKC precompiles 01–03. **It splits the mainnet window in two, so both sides need golden cases**; devnet is post-EVM throughout | +| shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same | Whether the disallow map is non-empty; without it these seven chains have a **zero replayable range** | +| `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same (both use the default) | The basis for determining starting gas on the target side | +| `configure_special_contract_ts` | **per precompile** | same | Gated by a strict `>` (`messages.py:672`), unlike the `<` used everywhere else — easy to get backwards | +| `ENABLE_NON_RESERVED` / `GENERAL_NATIVE_TOKEN` | **2020-05-01** | **0** | MNT / `pay_native_token_as_gas`, a non-goal — **end of the mainnet replayable range** | +| `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction`. Never hit inside the mainnet window; devnet needs it from block 1 | +| `ENABLE_POSW_STAKING_DECAY_TIMESTAMP` | 2020-05-01 | **absent = 0** | POSW staking decay (a non-goal, listed so it isn't overlooked) | + +Every chain-level switch on devnet is 0: post-EVM throughout, MNT permitted from genesis — **so devnet has no replayable range at all**. The column earns its place for golden vectors (S0 emits both networks) and for marking which branches devnet needs correct from block 1; `version == 2` is the obvious case. + +**Conclusion: the mainnet replayable range is ≈ 2019-04-30 → 2020-05-01 (applies to all eight chains).** Going past 2020-05-01 requires first moving the MNT group out of the non-goals. + +## Verification plan + +Four layers, cheapest to most expensive. Layer 1 is what the per-step acceptance table above runs on, one step at a time; layers 2 and 3 are S8; layer 4 is the final acceptance and falls outside this plan, landing with the chain-layer replay task: + +1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests. +2. **Porting pyquarkchain's shard state tests** — `test_shard_state.py` is a ready-made behavior checklist (~20 cross-shard-related spots). goquarkchain has already ported a large part of it to Go, usable directly as a checklist. +3. **Differential random testing** — randomly generate (alloc, tx sequence, deposit sequence), run both sides, and compare root and cursor. This is the only mechanism that can catch "the semantic that isn't on the checklist". +4. **Historical replay (final acceptance)** — join up with the paused replay task, replay real minor blocks in order and compare the seven items block by block. For the root tip, feed in the root block that confirmed the block; the two checks that read it are monotone, so a canonical block is never falsely rejected — and by the same token, replay does not verify those two checks. + +The replayable range is not "whatever is left once this is done" — it is fixed by the activation timestamps of the non-goal items; see Hard-fork switches above. + +## Risks and open questions + +- **triedb leaf decoding**: `hashdb.Update` decodes account leaves as geth's 4-field `types.StateAccount` to build account → storage-root references, which **QKC's 6-field leaves cannot satisfy**. `genesis_alloc.go` already works around it by committing each storage trie as a root of its own first; the mutable StateDB follows the same convention. + **Not a consensus issue** — the root is fixed at `trie.Hash()` and this code is only local GC bookkeeping, so a wrong decoder surfaces as a local `missing trie node`, never as a different root. The price is losing reference counting on this path; getting it back needs no geth change (the APIs are public, same shape as goquarkchain's onleaf callback). goshard runs on hashdb only (`triedb.HashDefaults`); **pathdb is neither used nor planned** — switching would need its own assessment, since it decodes accounts in three subsystems that sit on the read/rollback path. +- **Long-term cost of copying core/vm**: once the copy lands it diverges from upstream, and upstream security fixes will have to be tracked manually. +- **Legacy fork behavior in geth v1.17**: all fork-gated code is in principle still there, but 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners (EXP pricing, empty-account touching, CALL depth/balance check ordering). Layer 3 differential testing exists exactly for this; don't expect to enumerate them by reading code. +- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes V0. This must be filled in before reading old databases; it belongs to `qkc/types` and can be done in parallel. From db7e84a2cb71f143c585b18a32748ba6eaff2529 Mon Sep 17 00:00:00 2001 From: syntrust Date: Fri, 7 Aug 2026 11:24:10 +0800 Subject: [PATCH 2/6] fixes --- L1/tx-exec.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/L1/tx-exec.md b/L1/tx-exec.md index 5f5cbe2..64aef19 100644 --- a/L1/tx-exec.md +++ b/L1/tx-exec.md @@ -50,13 +50,13 @@ qkc/core ← newly written, Validate / ApplyTransaction / ApplyXShardDeposit Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate` (internally `Process` + comparison, returning an error if any item mismatches, so callers can never get hold of a StateDB that has been committed but is already invalid). `ValidateBlockResult` is a separate function so replay can call it directly. -Cross-shard data is fed in through an `XShardSource` interface. **It needs root block bodies (including the minor header list), and `qkc/types/rootblock.go` currently only has `RootBlockHeader`** — adding that type is a prerequisite and can be done in parallel with this plan. The real implementation (database reads/writes) belongs to the chain-layer task; the interface shape needs to be aligned with the owner of issue #1. +Cross-shard data is fed in through an `XShardSource` interface. **It needs root block bodies (including the minor header list), and `qkc/types/rootblock.go` currently only has `RootBlockHeader`** — adding that type is a prerequisite for the block-tier vectors and S7, and is being done in parallel with this plan. The code already exists on the `qkc-3-types-05-blocks` branch (https://github.com/QuarkChain/goshard/pull/36); it has to be merged before S7. The real implementation (database reads/writes) belongs to the chain-layer task; the interface shape needs to be aligned with the owner of issue #1. The two validation layers are not a simple "validate then execute" sequence: `ApplyTransaction` calls `validate_transaction` **a second time** internally, while `ValidateTxForBlock` inside `run_block` returns early in the future-nonce range and skips it. Implementing this as "validate once" would behave differently on future-nonce transactions. ## Step-by-step implementation -### S0 — golden generator +### S0 — golden generator — **state and message tiers landed; block tier blocked** Export three tiers of vectors — state level, message level, block level — from a pyquarkchain venv. The first case is fixed as a no-op genesis ALLOC whose post root must equal the two-network values in `minor_genesis_golden.json`, which calibrates the generator itself. ### S1 — qkc/state mutable state layer @@ -87,7 +87,7 @@ Layers 2 and 3 of the verification plan, as a step of their own: port the behavi | Step | Acceptance | |---|---| -| S0 | No-op genesis case, post root matches the golden for both networks | +| S0 | No-op genesis case, post root matches the golden for both networks — **met** (block tier still pending the root block body) | | S1 | ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip | | S2 | Compiles standalone; the Petersburg-and-earlier subset of geth's EVM unit tests passes | | S3 | Transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase | @@ -150,4 +150,4 @@ The replayable range is not "whatever is left once this is done" — it is fixed **Not a consensus issue** — the root is fixed at `trie.Hash()` and this code is only local GC bookkeeping, so a wrong decoder surfaces as a local `missing trie node`, never as a different root. The price is losing reference counting on this path; getting it back needs no geth change (the APIs are public, same shape as goquarkchain's onleaf callback). goshard runs on hashdb only (`triedb.HashDefaults`); **pathdb is neither used nor planned** — switching would need its own assessment, since it decodes accounts in three subsystems that sit on the read/rollback path. - **Long-term cost of copying core/vm**: once the copy lands it diverges from upstream, and upstream security fixes will have to be tracked manually. - **Legacy fork behavior in geth v1.17**: all fork-gated code is in principle still there, but 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners (EXP pricing, empty-account touching, CALL depth/balance check ordering). Layer 3 differential testing exists exactly for this; don't expect to enumerate them by reading code. -- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes V0. This must be filled in before reading old databases; it belongs to `qkc/types` and can be done in parallel. +- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes V1 (the other two are rejected outright). This must be filled in before reading old databases; it belongs to `qkc/types` and can be done in parallel. From 6048c3cdd257c58b594ed19d9f0a7c9af434b8b5 Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 18 Aug 2026 10:31:44 +0800 Subject: [PATCH 3/6] update --- L1/tx-exec.md | 169 +++++++++++++++++++++++++++++--------------------- 1 file changed, 98 insertions(+), 71 deletions(-) diff --git a/L1/tx-exec.md b/L1/tx-exec.md index 64aef19..f014ce1 100644 --- a/L1/tx-exec.md +++ b/L1/tx-exec.md @@ -4,7 +4,9 @@ For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute **byte-for-byte identical** state root / receipt root / gas used / xshard cursor / coinbase amount map / bloom. -Today `qkc/` already has the static half: account leaf encoding, transactions and signatures, minor block header/meta, receipts, bloom, token balances, `CrossShardTransactionDeposit`, `XShardTxCursorInfo`, genesis ALLOC → state root (golden pinned down and passing). What's missing is the **dynamic half**: a mutable QKC StateDB, an EVM, and the `apply_transaction` / `apply_xshard_deposit` / `run_block` execution path. +This work follows on from `feature/mnt-core-types` and `feature/mnt-state` — the six-field QKC account leaf (`core/types/state_account_qkc.go`) and the MNT balance layer inside geth's state layer are both landed. The static half has been in place for a while too: transactions and signatures, minor block header/meta, receipts, bloom, token balances, `CrossShardTransactionDeposit`, `XShardTxCursorInfo`, genesis ALLOC → state root (golden pinned down and passing). + +What's missing is **execution**: `qkc/state/evmstate.go`, which wraps the existing state layer into the shape of pyquarkchain's `State`; the QKC profile in `core/vm`; and the `apply_transaction` / `apply_xshard_deposit` / `run_block` path in `qkc/core`. This plan delivers that complete path, covering both the intra-shard and cross-shard ends, and compares results against pyquarkchain block by block. @@ -18,136 +20,161 @@ This plan delivers that complete path, covering both the intra-shard and cross-s - `Process`: the counterpart of `run_block`, producing state root / receipts (regular + deposit) / `gas_used` / `xshard_receive_gas_used` / new cursor / coinbase map / bloom - `ValidateBlockResult`: the counterpart of the **result** comparison in `add_block` - **Reading the POSW `sender_disallow_map`**: walk back `WINDOW_SIZE` along the header chain, counting coinbase occurrences × `TOTAL_STAKE_PER_BLOCK`. This is the only gap that **does not fail loudly** — with an empty map, `transfer_failure_by_posw_balance_check` raises no error, it merely lets a transfer that should have failed succeed. On mainnet the **shard-level** `POSW_CONFIG` is in effect from genesis for chains 1–7 (chain 0 is off), so without this their replayable range is zero; the cost is just one header walk of `WINDOW_SIZE` (256/512), fully decoupled from difficulty adjustment / staking / boost -- **Registering QKC precompiles `01/02/03`**: they are enabled together with `ENABLE_EVM_TIMESTAMP`, fall inside the replay window, and are equally callable for the default QKC token (`PrecompiledContractsAfterEvmEnabled` in `qkc/params/evm_params.go` is already exactly these three addresses). Without registering them, geth treats them as empty accounts and CALLs silently succeed +- **Five QKC precompiles**: `0x…514b430001/02/03` (`current_mnt_id` / `transfer_mnt` / `deploy_system_contract`) are enabled with `ENABLE_EVM_TIMESTAMP`; `0x…514b430004/05` (`mint_mnt` / `balance_of_mnt`) with `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP`. The gate is a strict `>` (`messages.py:672`). Without registering them, geth treats them as empty accounts and CALLs silently succeed +- **Full MNT consensus**: `pay_native_token_as_gas` / `get_gas_utility_info` call into the real general native token manager contract (`0x…514b430003`) to obtain `refund_rate` and the conversion price; the manager takes in the native token and pays out the genesis token; `refund_rate` participates both in the refund and in the burn to the zero address. The two system contract bytecodes, `NON_RESERVED_NATIVE_TOKEN` and `GENERAL_NATIVE_TOKEN`, are embedded in `qkc/core/syscontract_code.go` and deployed by `proc_deploy_system_contract` according to `SYSTEM_CONTRACT_SCOPE_MAP` (the former on chain 0 only) -On the token side, coverage is full EVM semantics for transfers and contract CREATE/CALL of the **default token (QKC)**; `refund_rate` / `gas_token_id` are carried end-to-end as **pipeline fields** and participate in refund and burn calculations, but the exchange-rate conversion itself (`pay_native_token_as_gas`) is not implemented. +On the token side, coverage extends to **all native tokens**: transfers of the default token and of any MNT, full EVM semantics for contract CREATE/CALL, and the conversion and settlement of paying gas in MNT — all aligned with pyquarkchain. **Non-goals (follow-up tasks)**: - Cluster/chain layer: intra-cluster broadcast and deposit of the xshard list, the gating checks of `add_root_block`, tip updates and fork choice, `create_block_to_mine`, tx pool -- **All of `validate_block`** — the structural checks that come before `run_block`: header version/height, prev block existence, gas limit, transaction count and size, merkle root, timestamp, difficulty, `hash_meta`. This plan only covers `run_block` + result comparison -- Multi-token transfers, the `GENERAL_NATIVE_TOKEN` system contract and `pay_native_token_as_gas`, the **MNT precompiles `0x…514b430004/05`** (`enable_ts` set to never-enabled) -- POSW **computation**: difficulty adjustment, staking, `BOOST_*`, decay. Only the disallow-map read described above +- **All of `validate_block`** — the structural checks in the half that comes before `run_block`: header version/height, prev block existence, gas limit, transaction count and size, merkle root, timestamp, difficulty, `hash_meta`. This plan only covers `run_block` + result comparison +- POSW **computation**: difficulty adjustment, staking, `BOOST_*`, decay (`_posw_info` only affects mining difficulty and does not enter `run_block`). Only the disallow-map read described above - Wiring into the `ShardChain` seam in `qkc/shard`, replacing `StubChainService` -Non-goal items are rejected in code with an explicit `error` (rather than silently skipped), so that "unsupported" is never mistaken for "results match". - ## Architecture and reuse strategy -Three new packages, zero modifications to geth source: +The overall approach is to hang QKC semantics onto geth's own execution stack, isolated behind a **nullable profile**: ``` -qkc/vm ← copied from geth core/vm and trimmed (Petersburg-only + QKC semantics) -qkc/state ← newly written, minimal mutable StateDB, backed by geth trie/triedb -qkc/core ← newly written, Validate / ApplyTransaction / ApplyXShardDeposit / Process +core/types, core/state ← six-field QKC account leaf + MNT balance layer + (the QKC side is concentrated in *_qkc.go) +core/vm ← QKCContext: two new files, qkc.go / contracts_qkc.go, + plus one `if evm.QKC != nil` branch each in + evm.go / instructions.go / gas_table.go +qkc/state/evmstate.go ← EvmState: the shape of pyquarkchain's State (block context, + receipts/logs, snapshot semantics) wrapped around geth's StateDB +qkc/core ← Validate / ApplyTransaction / ApplyXShardDeposit / + cursor traversal / Process / ValidateBlockResult ``` -**Execution results are not determined by the parent state alone.** Three inputs live outside the state root, so they must be passed in explicitly through an `ExecutionContext` rather than having `qkc/core` reach back into node state: +When `evm.QKC == nil`, geth's behavior is byte-for-byte unchanged (the existing `core/vm` and `core/state` unit tests all pass, which is the premise this shape rests on); when non-nil, it takes over CREATE address derivation, the token dimension of balances, SELFDESTRUCT semantics, the precompile table, and re-entry at the message layer. For the cost, see Risks and open questions. + +**Execution results are not determined by the parent state alone.** Several inputs live outside the state root and must be passed in explicitly through an `ExecutionContext`, rather than having `qkc/core` reach back into node state: -- the node's **root tip at the time** — `__validate_tx`'s "target shard already has genesis" check and `_is_neighbor` read this, not the block's `hash_prev_root_block` -- the ability to **look up minor headers by hash** — BLOCKHASH's 256 ancestors, plus the POSW window -- the **disallow map** computed from that window +- the node's **root tip at the time** (`ValidationRootTip`) — `__validate_tx`'s "target shard already has genesis" check and `_is_neighbor` read this, not the block's `hash_prev_root_block`. On replay you must feed in "the root block that confirmed this block", not the current root header +- **looking up minor headers by hash** (`MinorHeaderByHash`) — BLOCKHASH's 256 ancestors, plus the POSW window; the disallow map is computed on the fly over that chain by `SenderDisallowMap` +- **fetching the parent's meta by hash** (`MinorBlockMetaByHash`) — the cross-shard cursor resumes from there. Having it looked up rather than passed as a parameter prevents a caller from handing in a cursor that belongs to a different block -Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate` (internally `Process` + comparison, returning an error if any item mismatches, so callers can never get hold of a StateDB that has been committed but is already invalid). `ValidateBlockResult` is a separate function so replay can call it directly. +Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `TxSender` / `IntrinsicGas` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `SenderDisallowMap` / `CoinbaseAmountMap` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate` (internally `Process` + comparison, returning an error if any item mismatches, so callers can never get hold of a StateDB that has been committed but is already invalid). `ValidateBlockResult` is a separate function so replay can call it directly. -Cross-shard data is fed in through an `XShardSource` interface. **It needs root block bodies (including the minor header list), and `qkc/types/rootblock.go` currently only has `RootBlockHeader`** — adding that type is a prerequisite for the block-tier vectors and S7, and is being done in parallel with this plan. The code already exists on the `qkc-3-types-05-blocks` branch (https://github.com/QuarkChain/goshard/pull/36); it has to be merged before S7. The real implementation (database reads/writes) belongs to the chain-layer task; the interface shape needs to be aligned with the owner of issue #1. +Cross-shard data is fed in through the `XShardSource` interface (`RootBlockByHeight` / `RootHeaderByHash` / `DepositsByMinorBlockHash`, the last of which uses `found` to distinguish "empty list" from "no list" — pyquarkchain checks these two things in two different places, and a nil slice cannot serve both). It needs root block bodies; `qkc/types/rootblock.go` previously had only `RootBlockHeader` and has now been given a minimal implementation: header + minor header list + tracking data + `MinorHeaderMerkleRoot`, with goldens for both the serialization layout and the merkle root. The real implementation (database reads/writes) still belongs to the chain-layer task. The two validation layers are not a simple "validate then execute" sequence: `ApplyTransaction` calls `validate_transaction` **a second time** internally, while `ValidateTxForBlock` inside `run_block` returns early in the future-nonce range and skips it. Implementing this as "validate once" would behave differently on future-nonce transactions. ## Step-by-step implementation -### S0 — golden generator — **state and message tiers landed; block tier blocked** -Export three tiers of vectors — state level, message level, block level — from a pyquarkchain venv. The first case is fixed as a no-op genesis ALLOC whose post root must equal the two-network values in `minor_genesis_golden.json`, which calibrates the generator itself. +### S0 — golden generator + +`qkc/testdata/gen_exec_golden.py` exports three tiers of vectors — state level, message level, block level — from a pyquarkchain venv. All three tiers use pyquarkchain `75f8d7e166df0f5a2579ffe37ea7b4f5ba79db60` as the oracle: 17 state-level cases, 30 message-level cases, 11 block-level cases. The first case is fixed as a no-op genesis ALLOC whose post root must equal the two-network values in `minor_genesis_golden.json`, which calibrates the generator itself. + +**Acceptance**: the no-op genesis case, post root matching the golden for both networks; provenance (commit + module digest) written to disk alongside the vectors. + +### S1 — mutable state layer + +Six-field account leaves, balances indexed by token id, `FullShardKey` in the leaf, and existence/deletion rules that are their own thing. These extend geth's `core/state`: `core/types/state_account*.go` carries the leaf, `core/state/statedb_qkc.go` / `state_object_qkc.go` carry the MNT balance layer and the QKC-side journal entries, and `qkc/state/evmstate.go` (~550 lines) wraps it into the shape of pyquarkchain's `State` — block context, two receipt lists, `commit` one block at a time written through to disk, and snapshots that roll back the context along with the state. + +One concrete semantic difference: `Balance` is a scalar and cannot distinguish "a zero that was written" from "never held", yet those two serialize differently (`0x00c0` versus empty bytes) and are different leaves. `balanceUpdateCount` fills the gap by recording "does the balance map have this key", and it only ever increases — pyquarkchain's journal restores the **value** in the map (`state.py:166`), leaving the key in place; only `reset_balances` clears it. + +**Acceptance**: ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip. + +### S2 — the QKC profile in core/vm + +`core/vm` gains a `QKCContext`; once `evm.SetQKCContext` has attached it, it takes over CREATE address derivation (`keccak(rlp([sender, fullShardKey, nonce]))[12:]`, falling back to the old `rlp([sender, nonce])` derivation when there is no shard key), BALANCE and transfers using the chain's default token, QKC SELFDESTRUCT semantics, the five QKC precompiles, and re-entry at the message layer. + +`qkcApplyMsg` is the core of this layer: it reproduces the ordering of `_apply_msg` — `del_account` at the end of the message, truncating the suicide list along with everything else on revert, and the deferred `token_id_queried` check. The suicide list hangs off `QKCContext` rather than off the state, with a separate `QKCAdoptSuicides` to merge the self-destructs marked by the second EVM (the general native token manager call) back into this transaction — pyquarkchain's list lives on the state and is shared naturally. -### S1 — qkc/state mutable state layer -Today there is only the 33-line leaf struct in `account.go`. Balances are indexed by token id, `FullShardKey` goes into the leaf, and existence/deletion rules are their own thing — geth's `core/state` cannot be reused; estimated 1200+ lines on the Go side. +**Acceptance**: with `evm.QKC == nil`, the existing `core/vm` and `core/state` unit tests all pass. -### S2 — qkc/vm copy and trim -After copying geth `core/vm`, change four things: CREATE address derivation, `StateDB` carrying a token id, registering QKC precompiles 01–03, and removing post-Petersburg forks. The first is the one that must be changed in the copy and cannot be handled by injection. +### S3 — intra-shard ApplyTransaction -### S3 — intra-shard ApplyTransaction (pure transfers) -Implement all of `validate_transaction` except non-default gas token conversion, including the `version == 2` EIP155 branch — outside the mainnet replay window, but devnet enables it at genesis, so the branch is exercised from the first golden case onwards. +`validate_transaction` implemented in full, including conversion for non-default gas tokens (which goes through the manager for a quote but runs inside a snapshot that is then rolled back — `validate_transaction` only asks the price, it does not pay) and the `version == 2` EIP155 branch. Devnet enables EIP155 at genesis, so this branch is exercised from the very first golden case onwards. + +**Acceptance**: transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase. ### S4 — EVM integration (CREATE / CALL) -Assemble `BlockContext` / `TxContext` / message and hook up `qkc/vm`, feeding in `full_shard_key` for the address derivation from S2 to use. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. + +Assemble `BlockContext` / `TxContext` / message and hook up the S2 profile, feeding in `full_shard_key` for the address derivation to use; it is a `*uint32` rather than a `uint32`, because pyquarkchain's `None` and 0 derive different addresses, and `transfer_mnt` omits exactly this field when building its sub-message. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. + +**Acceptance**: contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom). ### S5 — cross-shard source side -The `is_cross_shard` branch: debit, produce the deposit, compute the address on the source shard for cross-shard deployments, and burn all gas on POSW failure. Charging and refunding the 9000 is gated by `ENABLE_EVM_TIMESTAMP`. + +The `is_cross_shard` branch: debit, produce the deposit, compute the address on the source shard for cross-shard deployments, and burn all gas on POSW failure; `refund_rate` and `gas_token_id` travel with the deposit, and the target side uses them to refund and to burn proportionally. Charging and refunding the 9000 is gated by `ENABLE_EVM_TIMESTAMP`. + +**Acceptance**: compared on root + receipt + `gas_used` + the produced deposit **field by field**. ### S6 — cross-shard target side + `RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. +**Acceptance**: must include three items — post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP ±1` boundary. + ### S7 — Process(block) and block-level settlement -The three-level cursor traversal and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees — counting only the former misses part of it. -### S8 — ported tests and differential random testing -Layers 2 and 3 of the verification plan, as a step of their own: port the behavior checklist from `test_shard_state.py`, then run randomly generated (alloc, tx sequence, deposit sequence) triples against both implementations. They need the whole path finished, but nothing from the chain layer — so they are the last step this plan owns. Layer 4 (historical replay) is not a step here; it lands with the chain-layer task. +The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token — counting only the former misses part of it. A cursor that cannot fetch a root block **within range** must raise `ErrMissingRootBlock` rather than treat it as end of stream — only a height beyond the root tip is EOF, and taking missing data for EOF computes a state root nobody else agrees with. + +**Acceptance**: block-level golden — the genesis root derived from ALLOC alone, then the seven result items compared block by block, plus the deposits consumed/produced and account read-back; and the cut-and-resume of a single root block's deposit list spanning two minor blocks. + +### S8 — ported tests -### Acceptance per step +Layer 2 of the verification plan, as a step of its own: port the behavior checklist from `test_shard_state.py` into `qkc/core/shardstate_port_test.go`. It requires the whole path to be finished but depends on nothing from the chain layer — so it is the last step this plan owns. Layer 3, historical replay, lands with the chain-layer task. -| Step | Acceptance | -|---|---| -| S0 | No-op genesis case, post root matches the golden for both networks — **met** (block tier still pending the root block body) | -| S1 | ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip | -| S2 | Compiles standalone; the Petersburg-and-earlier subset of geth's EVM unit tests passes | -| S3 | Transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase | -| S4 | Contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom) | -| S5 | Compared on root + receipt + `gas_used` + the produced deposit **field by field** | -| S6 | Must include three items: post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP ±1` boundary | -| S7 | Cut-and-resume of a single root block's deposit list consumed across two minor blocks; N consecutive chained blocks after genesis | -| S8 | The ported checklist passes; differential runs agree with pyquarkchain on root and cursor, and every divergence found is reduced to a golden case | +The ported checklist, plus the few gaps the goldens cannot cover: the cross-shard surcharge across the DDOS fix height, a cursor spanning two root blocks, a failed transaction consuming the entire allowance, the BLOCKHASH window, coinbase decay per epoch, each of the seven result items mismatching being rejected, EIP155 replay on another chain being rejected, the source side converting a native-token gas price, the cursor erroring on a missing in-range root block or an unknown parent, BALANCE reading the chain's default token, and the MNT precompiles following the non-reserved switch. The last two need a shard config where `DEFAULT_CHAIN_TOKEN` is not QKC and the two MNT switches differ — a situation the shipping configs cannot produce, so the goldens cannot catch it. + +**Acceptance**: the ported checklist passes; every new case is mutation-tested — revert the corresponding implementation to the wrong version and the case must turn red. ## Ordering and parallelism ``` qkc/types root block body ─────┐ S0 golden generator ───────────┤ -S1 state ──┐ │ -S2 vm ─────┴─> S3 ─> S4 ─┬─> S5 ─┐ - └─> S6 ─┴─> S7 ─> S8 ─> replay - ↑ - XShardSource real implementation (chain-layer task) +S1 state ────────┐ │ +S2 vm profile ───┴─> S3 ─> S4 ─┬─> S5 ─┐ + └─> S6 ─┴─> S7 ─> S8 ─> replay + ↑ + XShardSource real implementation (chain-layer task) ``` Parallelizable: root block body with S0; S1 with S2; S5 with S6 (source and target sides each rely on message-level golden and are independent of each other). -The earliest external unblock is S1 — once `qkc/state` lands, other tasks can depend on it. -**Person-day estimates TBD** (this plan gives dependency order only). +The earliest external unblock is S1 — once the state layer lands, other tasks can depend on it. ## Hard-fork switches -They are not "historical baggage a new chain can ignore" — replaying mainnet history requires keeping them, so it is worth writing against the switches from the start. This table doubles as the implementation checklist for S3–S7 and as the basis of the replayable range (timestamps and POSW config are in `qkc/config/singularity/mainnet.json`): - | Switch | mainnet | devnet | Where it applies | |---|---|---|---| | genesis | 2019-04-30 | same | Start of the replayable range | | `ENABLE_TX_TIMESTAMP` + `TX_WHITELIST_SENDERS` | 2019-06-29 | **0** | Before this, only whitelisted addresses could send transactions; devnet has no such phase | | `ENABLE_EVM_TIMESTAMP` | 2019-09-27 | **0** | **Six places**: three differences in cross-shard gas settlement, `__validate_tx` forbidding contract transactions, whether cross-shard receiving takes the pre-EVM fixed-amount path or the EVM path, and enabling QKC precompiles 01–03. **It splits the mainnet window in two, so both sides need golden cases**; devnet is post-EVM throughout | | shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same | Whether the disallow map is non-empty; without it these seven chains have a **zero replayable range** | -| `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same (both use the default) | The basis for determining starting gas on the target side | -| `configure_special_contract_ts` | **per precompile** | same | Gated by a strict `>` (`messages.py:672`), unlike the `<` used everywhere else — easy to get backwards | -| `ENABLE_NON_RESERVED` / `GENERAL_NATIVE_TOKEN` | **2020-05-01** | **0** | MNT / `pay_native_token_as_gas`, a non-goal — **end of the mainnet replayable range** | -| `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction`. Never hit inside the mainnet window; devnet needs it from block 1 | -| `ENABLE_POSW_STAKING_DECAY_TIMESTAMP` | 2020-05-01 | **absent = 0** | POSW staking decay (a non-goal, listed so it isn't overlooked) | +| `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same | The basis for determining starting gas on the target side | +| `configure_special_contract_ts` | **per precompile** | same | Gated by a strict `>` (`messages.py:672`), unlike the `<` used elsewhere — easy to get backwards. The gate on system contract **deployment** is the other way round and non-strict (it rejects when `block_timestamp < enable_ts`) | +| `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | The non-reserved token auction contract (chain 0 only) **and MNT precompiles `04/05`**. Those two precompiles listen to this switch alone (`env.py:63-76`); taking the min of the two switches is wrong | +| `ENABLE_GENERAL_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | Only determines when the general native token manager contract may be deployed; whether `pay_native_token_as_gas` takes effect depends on whether that address has code, with no separate time gate | +| `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction`. Devnet needs it from block 1; mainnet only has it after this point | -Every chain-level switch on devnet is 0: post-EVM throughout, MNT permitted from genesis — **so devnet has no replayable range at all**. The column earns its place for golden vectors (S0 emits both networks) and for marking which branches devnet needs correct from block 1; `version == 2` is the obvious case. +Every chain-level switch on devnet is 0: post-EVM throughout, MNT permitted from genesis. The column earns its place for golden vectors (S0 emits both networks) and for marking which branches devnet needs correct from block 1 — `version == 2` is the obvious case. -**Conclusion: the mainnet replayable range is ≈ 2019-04-30 → 2020-05-01 (applies to all eight chains).** Going past 2020-05-01 requires first moving the MNT group out of the non-goals. +**Conclusion: the execution layer no longer has an upper bound on the replayable range.** Every row in the table is implemented, including the MNT group that previously drew the line at 2020-05-01. ## Verification plan -Four layers, cheapest to most expensive. Layer 1 is what the per-step acceptance table above runs on, one step at a time; layers 2 and 3 are S8; layer 4 is the final acceptance and falls outside this plan, landing with the chain-layer replay task: - -1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests. -2. **Porting pyquarkchain's shard state tests** — `test_shard_state.py` is a ready-made behavior checklist (~20 cross-shard-related spots). goquarkchain has already ported a large part of it to Go, usable directly as a checklist. -3. **Differential random testing** — randomly generate (alloc, tx sequence, deposit sequence), run both sides, and compare root and cursor. This is the only mechanism that can catch "the semantic that isn't on the checklist". -4. **Historical replay (final acceptance)** — join up with the paused replay task, replay real minor blocks in order and compare the seven items block by block. For the root tip, feed in the root block that confirmed the block; the two checks that read it are monotone, so a canonical block is never falsely rejected — and by the same token, replay does not verify those two checks. +Three layers, cheapest to most expensive. Layer 1 is what the per-step "Acceptance" actually runs; layer 2 is S8; layer 3 is the final acceptance and falls outside this plan, landing with the chain-layer replay task: -The replayable range is not "whatever is left once this is done" — it is fixed by the activation timestamps of the non-goal items; see Hard-fork switches above. +1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests. Three tiers: 17 state-level cases (`qkc/state/statedb_test.go`), 30 message-level cases (`qkc/core/message_golden_test.go`), 11 block-level cases (`qkc/core/block_golden_test.go`). +2. **Porting pyquarkchain's shard state tests** — `test_shard_state.py` is a ready-made behavior checklist, landing as `qkc/core/shardstate_port_test.go`. Every new case has been mutation-tested: revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. +3. **Historical replay (final acceptance)** — join up with the paused replay task, replay real minor blocks in order and compare every committed value of `ValidateBlockResult` block by block. Replay only runs canonical blocks, so it only verifies the "what should pass does pass" half; rules that only take effect on the rejection path still depend on the first two layers for coverage. ## Risks and open questions -- **triedb leaf decoding**: `hashdb.Update` decodes account leaves as geth's 4-field `types.StateAccount` to build account → storage-root references, which **QKC's 6-field leaves cannot satisfy**. `genesis_alloc.go` already works around it by committing each storage trie as a root of its own first; the mutable StateDB follows the same convention. - **Not a consensus issue** — the root is fixed at `trie.Hash()` and this code is only local GC bookkeeping, so a wrong decoder surfaces as a local `missing trie node`, never as a different root. The price is losing reference counting on this path; getting it back needs no geth change (the APIs are public, same shape as goquarkchain's onleaf callback). goshard runs on hashdb only (`triedb.HashDefaults`); **pathdb is neither used nor planned** — switching would need its own assessment, since it decodes accounts in three subsystems that sit on the read/rollback path. -- **Long-term cost of copying core/vm**: once the copy lands it diverges from upstream, and upstream security fixes will have to be tracked manually. -- **Legacy fork behavior in geth v1.17**: all fork-gated code is in principle still there, but 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners (EXP pricing, empty-account touching, CALL depth/balance check ordering). Layer 3 differential testing exists exactly for this; don't expect to enumerate them by reading code. -- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes V1 (the other two are rejected outright). This must be filled in before reading old databases; it belongs to `qkc/types` and can be done in parallel. +Ordered by severity: unresolved divergences first, implemented boundaries and already-fixed defects last. + +- **Legacy fork behavior in geth v1.17**: 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners. The static layer — opcode table, constant gas, precompile pricing and set — turned up nothing; the dynamic layer — SSTORE's four Petersburg tiers, RETURNDATACOPY's out-of-bounds check, CREATE's handling of init code failure — has not been gone through case by case. +- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes the current version 1 and errors out on anything else. **This code may change along with [PR #47](https://github.com/QuarkChain/goshard/pull/47).** +- **Token trie boundary**: when an account has more than 16 non-zero token balances (`TokenTrieThreshold`), pyquarkchain switches the balance to `b"\x01" + secure trie root`; the Go side gives up on the whole block with `ErrUnsupportedNativeToken`. +- **pathdb not adapted**: pathdb mixes the slim and full formats in three places — **snapshot generation, state rollback, and account reads**. In geth the two are inter-derivable; once QKC changes the contents they no longer are. + - goshard runs hashdb only and the execution path never touches pathdb, so this does not affect the correctness of what this plan delivers. + - `triedb/pathdb`'s `TestDatabaseRollback` / `TestExecuteRollback` are red. + - Whether to enable pathdb later is a chain-layer decision and needs its own assessment at that point: all three places sit on read and rollback paths, so the failure mode is silently reading back a wrong account, rather than failing loudly the way the token trie boundary does. +- **CREATE2's negative gas (implemented as a boundary)**: pyquarkchain does not check the balance when charging CREATE2's per-word fee, and when memory happens not to need expanding, the frame keeps running with negative gas until `assert gas_remained >= 0` blows up on the spot. Such a transaction cannot be executed upstream at all, so it never appears in mainnet history and replay is unaffected. Rather than have goshard reproduce negative gas, it rejects outright. +- **Long-term cost of modifying files inside the geth tree**: there is no `qkc/vm` copy to maintain any more; the price is in-tree changes across `core/vm`, `core/state` and `core/types`, which will conflict when rebasing onto upstream. The mitigation is the shape itself: the QKC side lives in separate files (`*_qkc.go`) wherever possible, what gets inserted into existing files is only single-point branches like `if evm.QKC != nil`, and the nil path is guarded by geth's own unit tests. From d84510c57634b55d3c968b16372ad5c231ece3f9 Mon Sep 17 00:00:00 2001 From: syntrust Date: Wed, 19 Aug 2026 19:27:34 +0800 Subject: [PATCH 4/6] update --- L1/tx-exec.md | 55 +++++++++++++++++---------------------------------- 1 file changed, 18 insertions(+), 37 deletions(-) diff --git a/L1/tx-exec.md b/L1/tx-exec.md index f014ce1..41b7784 100644 --- a/L1/tx-exec.md +++ b/L1/tx-exec.md @@ -2,7 +2,7 @@ ## Context -For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute **byte-for-byte identical** state root / receipt root / gas used / xshard cursor / coinbase amount map / bloom. +For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute the same seven results **byte for byte**: state root / receipt root / gas used / xshard receive gas used / xshard cursor / coinbase amount map / bloom. This work follows on from `feature/mnt-core-types` and `feature/mnt-state` — the six-field QKC account leaf (`core/types/state_account_qkc.go`) and the MNT balance layer inside geth's state layer are both landed. The static half has been in place for a while too: transactions and signatures, minor block header/meta, receipts, bloom, token balances, `CrossShardTransactionDeposit`, `XShardTxCursorInfo`, genesis ALLOC → state root (golden pinned down and passing). @@ -56,11 +56,9 @@ When `evm.QKC == nil`, geth's behavior is byte-for-byte unchanged (the existing - **looking up minor headers by hash** (`MinorHeaderByHash`) — BLOCKHASH's 256 ancestors, plus the POSW window; the disallow map is computed on the fly over that chain by `SenderDisallowMap` - **fetching the parent's meta by hash** (`MinorBlockMetaByHash`) — the cross-shard cursor resumes from there. Having it looked up rather than passed as a parameter prevents a caller from handing in a cursor that belongs to a different block -Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `TxSender` / `IntrinsicGas` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `SenderDisallowMap` / `CoinbaseAmountMap` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate` (internally `Process` + comparison, returning an error if any item mismatches, so callers can never get hold of a StateDB that has been committed but is already invalid). `ValidateBlockResult` is a separate function so replay can call it directly. +Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `TxSender` / `IntrinsicGas` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `SenderDisallowMap` / `CoinbaseAmountMap` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate`. `ValidateBlockResult` is a separate function so replay can call it directly. -Cross-shard data is fed in through the `XShardSource` interface (`RootBlockByHeight` / `RootHeaderByHash` / `DepositsByMinorBlockHash`, the last of which uses `found` to distinguish "empty list" from "no list" — pyquarkchain checks these two things in two different places, and a nil slice cannot serve both). It needs root block bodies; `qkc/types/rootblock.go` previously had only `RootBlockHeader` and has now been given a minimal implementation: header + minor header list + tracking data + `MinorHeaderMerkleRoot`, with goldens for both the serialization layout and the merkle root. The real implementation (database reads/writes) still belongs to the chain-layer task. - -The two validation layers are not a simple "validate then execute" sequence: `ApplyTransaction` calls `validate_transaction` **a second time** internally, while `ValidateTxForBlock` inside `run_block` returns early in the future-nonce range and skips it. Implementing this as "validate once" would behave differently on future-nonce transactions. +Cross-shard data is fed in through the `XShardSource` interface (`RootBlockByHeight` / `RootHeaderByHash` / `DepositsByMinorBlockHash`). It needs root block bodies; `qkc/types/rootblock.go` previously had only `RootBlockHeader` and has now been given a minimal implementation: header + minor header list + tracking data + `MinorHeaderMerkleRoot`, with goldens for both the serialization layout and the merkle root. The real implementation (database reads/writes) still belongs to the chain-layer task. ## Step-by-step implementation @@ -74,8 +72,6 @@ The two validation layers are not a simple "validate then execute" sequence: `Ap Six-field account leaves, balances indexed by token id, `FullShardKey` in the leaf, and existence/deletion rules that are their own thing. These extend geth's `core/state`: `core/types/state_account*.go` carries the leaf, `core/state/statedb_qkc.go` / `state_object_qkc.go` carry the MNT balance layer and the QKC-side journal entries, and `qkc/state/evmstate.go` (~550 lines) wraps it into the shape of pyquarkchain's `State` — block context, two receipt lists, `commit` one block at a time written through to disk, and snapshots that roll back the context along with the state. -One concrete semantic difference: `Balance` is a scalar and cannot distinguish "a zero that was written" from "never held", yet those two serialize differently (`0x00c0` versus empty bytes) and are different leaves. `balanceUpdateCount` fills the gap by recording "does the balance map have this key", and it only ever increases — pyquarkchain's journal restores the **value** in the map (`state.py:166`), leaving the key in place; only `reset_balances` clears it. - **Acceptance**: ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip. ### S2 — the QKC profile in core/vm @@ -88,13 +84,13 @@ One concrete semantic difference: `Balance` is a scalar and cannot distinguish " ### S3 — intra-shard ApplyTransaction -`validate_transaction` implemented in full, including conversion for non-default gas tokens (which goes through the manager for a quote but runs inside a snapshot that is then rolled back — `validate_transaction` only asks the price, it does not pay) and the `version == 2` EIP155 branch. Devnet enables EIP155 at genesis, so this branch is exercised from the very first golden case onwards. +`validate_transaction` implemented in full, including conversion for non-default gas tokens and the `version == 2` EIP155 branch. Devnet enables EIP155 at genesis, so this branch is exercised from the very first golden case onwards. **Acceptance**: transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase. ### S4 — EVM integration (CREATE / CALL) -Assemble `BlockContext` / `TxContext` / message and hook up the S2 profile, feeding in `full_shard_key` for the address derivation to use; it is a `*uint32` rather than a `uint32`, because pyquarkchain's `None` and 0 derive different addresses, and `transfer_mnt` omits exactly this field when building its sub-message. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. +Assemble `BlockContext` / `TxContext` / message and hook up the S2 profile, feeding in `full_shard_key` for the address derivation to use. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. **Acceptance**: contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom). @@ -108,37 +104,20 @@ The `is_cross_shard` branch: debit, produce the deposit, compute the address on `RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. -**Acceptance**: must include three items — post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP ±1` boundary. +**Acceptance**: must include three items — post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP` ±1 boundary. ### S7 — Process(block) and block-level settlement -The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token — counting only the former misses part of it. A cursor that cannot fetch a root block **within range** must raise `ErrMissingRootBlock` rather than treat it as end of stream — only a height beyond the root tip is EOF, and taking missing data for EOF computes a state root nobody else agrees with. +The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token. **Acceptance**: block-level golden — the genesis root derived from ALLOC alone, then the seven result items compared block by block, plus the deposits consumed/produced and account read-back; and the cut-and-resume of a single root block's deposit list spanning two minor blocks. ### S8 — ported tests -Layer 2 of the verification plan, as a step of its own: port the behavior checklist from `test_shard_state.py` into `qkc/core/shardstate_port_test.go`. It requires the whole path to be finished but depends on nothing from the chain layer — so it is the last step this plan owns. Layer 3, historical replay, lands with the chain-layer task. - -The ported checklist, plus the few gaps the goldens cannot cover: the cross-shard surcharge across the DDOS fix height, a cursor spanning two root blocks, a failed transaction consuming the entire allowance, the BLOCKHASH window, coinbase decay per epoch, each of the seven result items mismatching being rejected, EIP155 replay on another chain being rejected, the source side converting a native-token gas price, the cursor erroring on a missing in-range root block or an unknown parent, BALANCE reading the chain's default token, and the MNT precompiles following the non-reserved switch. The last two need a shard config where `DEFAULT_CHAIN_TOKEN` is not QKC and the two MNT switches differ — a situation the shipping configs cannot produce, so the goldens cannot catch it. +Layer 2 of the verification plan, as a step of its own: port the behavior checklist from `test_shard_state.py` into `qkc/core/shardstate_port_test.go`. It requires the whole path to be finished but depends on nothing from the chain layer — so it is the last step this plan owns. **Acceptance**: the ported checklist passes; every new case is mutation-tested — revert the corresponding implementation to the wrong version and the case must turn red. -## Ordering and parallelism - -``` -qkc/types root block body ─────┐ -S0 golden generator ───────────┤ -S1 state ────────┐ │ -S2 vm profile ───┴─> S3 ─> S4 ─┬─> S5 ─┐ - └─> S6 ─┴─> S7 ─> S8 ─> replay - ↑ - XShardSource real implementation (chain-layer task) -``` - -Parallelizable: root block body with S0; S1 with S2; S5 with S6 (source and target sides each rely on message-level golden and are independent of each other). -The earliest external unblock is S1 — once the state layer lands, other tasks can depend on it. - ## Hard-fork switches | Switch | mainnet | devnet | Where it applies | @@ -155,15 +134,17 @@ The earliest external unblock is S1 — once the state layer lands, other tasks Every chain-level switch on devnet is 0: post-EVM throughout, MNT permitted from genesis. The column earns its place for golden vectors (S0 emits both networks) and for marking which branches devnet needs correct from block 1 — `version == 2` is the obvious case. -**Conclusion: the execution layer no longer has an upper bound on the replayable range.** Every row in the table is implemented, including the MNT group that previously drew the line at 2020-05-01. +**The execution layer no longer has an upper bound on the replayable range.** ## Verification plan -Three layers, cheapest to most expensive. Layer 1 is what the per-step "Acceptance" actually runs; layer 2 is S8; layer 3 is the final acceptance and falls outside this plan, landing with the chain-layer replay task: +Three layers, cheapest to most expensive. + +1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests; this is what the per-step "Acceptance" above actually runs. Three tiers: 17 state-level cases (`qkc/state/statedb_test.go`), 30 message-level cases (`qkc/core/message_golden_test.go`), 11 block-level cases (`qkc/core/block_golden_test.go`). +2. **Porting pyquarkchain's shard state tests** — S8, a step of its own: `test_shard_state.py` is a ready-made behavior checklist, landing as `qkc/core/shardstate_port_test.go`. Every new case has been mutation-tested: revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. +3. **Historical replay (final acceptance)** — falls outside this plan, landing with the chain-layer replay task: join up with the paused replay task, replay real minor blocks in order and compare every committed value of `ValidateBlockResult` block by block. -1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests. Three tiers: 17 state-level cases (`qkc/state/statedb_test.go`), 30 message-level cases (`qkc/core/message_golden_test.go`), 11 block-level cases (`qkc/core/block_golden_test.go`). -2. **Porting pyquarkchain's shard state tests** — `test_shard_state.py` is a ready-made behavior checklist, landing as `qkc/core/shardstate_port_test.go`. Every new case has been mutation-tested: revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. -3. **Historical replay (final acceptance)** — join up with the paused replay task, replay real minor blocks in order and compare every committed value of `ValidateBlockResult` block by block. Replay only runs canonical blocks, so it only verifies the "what should pass does pass" half; rules that only take effect on the rejection path still depend on the first two layers for coverage. +Replay only runs canonical blocks, so it only verifies the "what should pass does pass" half; rules that only take effect on the rejection path still depend on the first two layers for coverage. ## Risks and open questions @@ -173,8 +154,8 @@ Ordered by severity: unresolved divergences first, implemented boundaries and al - **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes the current version 1 and errors out on anything else. **This code may change along with [PR #47](https://github.com/QuarkChain/goshard/pull/47).** - **Token trie boundary**: when an account has more than 16 non-zero token balances (`TokenTrieThreshold`), pyquarkchain switches the balance to `b"\x01" + secure trie root`; the Go side gives up on the whole block with `ErrUnsupportedNativeToken`. - **pathdb not adapted**: pathdb mixes the slim and full formats in three places — **snapshot generation, state rollback, and account reads**. In geth the two are inter-derivable; once QKC changes the contents they no longer are. - - goshard runs hashdb only and the execution path never touches pathdb, so this does not affect the correctness of what this plan delivers. - - `triedb/pathdb`'s `TestDatabaseRollback` / `TestExecuteRollback` are red. - - Whether to enable pathdb later is a chain-layer decision and needs its own assessment at that point: all three places sit on read and rollback paths, so the failure mode is silently reading back a wrong account, rather than failing loudly the way the token trie boundary does. + - goshard runs hashdb only and the execution path never touches pathdb, so this does not affect the correctness of what this plan delivers. + - `triedb/pathdb`'s `TestDatabaseRollback` / `TestExecuteRollback` are red. + - Whether to enable pathdb later is a chain-layer decision and needs its own assessment at that point: all three places sit on read and rollback paths, so the failure mode is silently reading back a wrong account, rather than failing loudly the way the token trie boundary does. - **CREATE2's negative gas (implemented as a boundary)**: pyquarkchain does not check the balance when charging CREATE2's per-word fee, and when memory happens not to need expanding, the frame keeps running with negative gas until `assert gas_remained >= 0` blows up on the spot. Such a transaction cannot be executed upstream at all, so it never appears in mainnet history and replay is unaffected. Rather than have goshard reproduce negative gas, it rejects outright. - **Long-term cost of modifying files inside the geth tree**: there is no `qkc/vm` copy to maintain any more; the price is in-tree changes across `core/vm`, `core/state` and `core/types`, which will conflict when rebasing onto upstream. The mitigation is the shape itself: the QKC side lives in separate files (`*_qkc.go`) wherever possible, what gets inserted into existing files is only single-point branches like `if evm.QKC != nil`, and the nil path is guarded by geth's own unit tests. From 405e781b7a482758c6de25689d6a0cd90e0eebfc Mon Sep 17 00:00:00 2001 From: syntrust Date: Fri, 21 Aug 2026 19:18:20 +0800 Subject: [PATCH 5/6] rewrite --- L1/tx-exec.md | 180 ++++++++++++++++++++++++++++---------------------- 1 file changed, 102 insertions(+), 78 deletions(-) diff --git a/L1/tx-exec.md b/L1/tx-exec.md index 41b7784..66ed640 100644 --- a/L1/tx-exec.md +++ b/L1/tx-exec.md @@ -2,160 +2,184 @@ ## Context -For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute the same seven results **byte for byte**: state root / receipt root / gas used / xshard receive gas used / xshard cursor / coinbase amount map / bloom. +For goshard to become the new implementation of QuarkChain shards, it must eventually reach consensus with pyquarkchain on the same chain — given the same minor block, both sides must compute the same results **byte for byte**. -This work follows on from `feature/mnt-core-types` and `feature/mnt-state` — the six-field QKC account leaf (`core/types/state_account_qkc.go`) and the MNT balance layer inside geth's state layer are both landed. The static half has been in place for a while too: transactions and signatures, minor block header/meta, receipts, bloom, token balances, `CrossShardTransactionDeposit`, `XShardTxCursorInfo`, genesis ALLOC → state root (golden pinned down and passing). +The encoding and state layers are already in place, up to and including the genesis state root. What is missing is **execution**, and this design delivers it over the chain's default token, at both the intra-shard and the cross-shard end. -What's missing is **execution**: `qkc/state/evmstate.go`, which wraps the existing state layer into the shape of pyquarkchain's `State`; the QKC profile in `core/vm`; and the `apply_transaction` / `apply_xshard_deposit` / `run_block` path in `qkc/core`. +## Goals -This plan delivers that complete path, covering both the intra-shard and cross-shard ends, and compares results against pyquarkchain block by block. +A Go counterpart of pyquarkchain's message-level execution, plus the **pure-execution** layer of its shard state: -## Goals / Non-goals +- **Transaction validation**, both standalone and against the block being built, including the EIP155 signature branch +- **Applying a transaction**: the intra-shard path, and the cross-shard path that produces a deposit rather than executing at the destination +- **Applying a cross-shard deposit** at the destination, and the cursor traversal that decides which deposits a block consumes +- **Running a block**: the seven committed results — state root / receipt root / gas used / xshard receive gas used / xshard cursor / coinbase amount map / bloom — from a parent state and a set of transactions +- **Comparing results**: the check that a block's declared results match what execution produced, as its own entry point so the later replay task can call it directly -**Goals**: make `qkc/core` a full Go counterpart of pyquarkchain's `evm/messages.py`, plus the **pure-execution** layer of `shard_state.py`: +Running one block composes them into a single pass: -- `ValidateTransaction` + `ValidateTxForBlock` (`__validate_tx`), including the `version == 2` (EIP155) branch -- `ApplyTransaction`: intra-shard branch + `is_cross_shard` branch (producing deposits) -- `ApplyXShardDeposit` + cursor traversal -- `Process`: the counterpart of `run_block`, producing state root / receipts (regular + deposit) / `gas_used` / `xshard_receive_gas_used` / new cursor / coinbase map / bloom -- `ValidateBlockResult`: the counterpart of the **result** comparison in `add_block` -- **Reading the POSW `sender_disallow_map`**: walk back `WINDOW_SIZE` along the header chain, counting coinbase occurrences × `TOTAL_STAKE_PER_BLOCK`. This is the only gap that **does not fail loudly** — with an empty map, `transfer_failure_by_posw_balance_check` raises no error, it merely lets a transfer that should have failed succeed. On mainnet the **shard-level** `POSW_CONFIG` is in effect from genesis for chains 1–7 (chain 0 is off), so without this their replayable range is zero; the cost is just one header walk of `WINDOW_SIZE` (256/512), fully decoupled from difficulty adjustment / staking / boost -- **Five QKC precompiles**: `0x…514b430001/02/03` (`current_mnt_id` / `transfer_mnt` / `deploy_system_contract`) are enabled with `ENABLE_EVM_TIMESTAMP`; `0x…514b430004/05` (`mint_mnt` / `balance_of_mnt`) with `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP`. The gate is a strict `>` (`messages.py:672`). Without registering them, geth treats them as empty accounts and CALLs silently succeed -- **Full MNT consensus**: `pay_native_token_as_gas` / `get_gas_utility_info` call into the real general native token manager contract (`0x…514b430003`) to obtain `refund_rate` and the conversion price; the manager takes in the native token and pays out the genesis token; `refund_rate` participates both in the refund and in the burn to the zero address. The two system contract bytecodes, `NON_RESERVED_NATIVE_TOKEN` and `GENERAL_NATIVE_TOKEN`, are embedded in `qkc/core/syscontract_code.go` and deployed by `proc_deploy_system_contract` according to `SYSTEM_CONTRACT_SCOPE_MAP` (the former on chain 0 only) +1. **Bind a deterministic environment** — parent state, block header, fork configuration, root-chain view, ancestor lookup, disallow map, parent cursor, cross-shard source. +2. **Consume eligible cross-shard deposits first**, resuming from the parent cursor and stopping when the receive-gas budget or the available data runs out. The ordering is consensus, not convenience. +3. **Execute regular transactions in block order** — validate, run the transfer or EVM message, record receipts and logs, accumulate gas, and produce source-side deposits for cross-shard transactions. +4. **Settle and compare** — block reward plus fees, commit the state, derive receipts and bloom, finalize the cursor, and check every computed value against what the block declares. -On the token side, coverage extends to **all native tokens**: transfers of the default token and of any MNT, full EVM semantics for contract CREATE/CALL, and the conversion and settlement of paying gas in MNT — all aligned with pyquarkchain. +### Out of scope -**Non-goals (follow-up tasks)**: +- **Multi-native-token (MNT) consensus** — the manager contract that prices the conversion and sets the refund rate, the refund and burn that follow from it, the system contract bytecodes, and the minting and balance precompiles that go with them. Anything that reaches execution — a transaction paying gas in or transferring a non-default token, or a cross-shard deposit carrying one — abandons the **whole block** with a sentinel error +- **Historical replay** — running real minor blocks in order and comparing every committed value. It is what finally settles byte-for-byte equality, but it needs the chain layer underneath it and lands with its own task +- **Cluster and chain layer** — broadcasting and depositing the cross-shard list, the gating checks on accepting a root block, tip updates and fork choice, block assembly for mining, the transaction pool +- **Structural block validation** — everything checked before execution starts: header version and height, parent existence, gas limit, transaction count and size, merkle root, timestamp, difficulty, meta hash. This design covers execution and the result comparison +- **Proof-of-staked-work computation** — difficulty adjustment, staking, boost and decay, which feed mining difficulty rather than execution results +- **Trie-encoded balances** — above 16 non-zero token balances, pyquarkchain switches an account's balance encoding to a secure trie root. Same treatment: sentinel, whole block -- Cluster/chain layer: intra-cluster broadcast and deposit of the xshard list, the gating checks of `add_root_block`, tip updates and fork choice, `create_block_to_mine`, tx pool -- **All of `validate_block`** — the structural checks in the half that comes before `run_block`: header version/height, prev block existence, gas limit, transaction count and size, merkle root, timestamp, difficulty, `hash_meta`. This plan only covers `run_block` + result comparison -- POSW **computation**: difficulty adjustment, staking, `BOOST_*`, decay (`_posw_info` only affects mining difficulty and does not enter `run_block`). Only the disallow-map read described above -- Wiring into the `ShardChain` seam in `qkc/shard`, replacing `StubChainService` +## Design principles -## Architecture and reuse strategy +**The central design choice is to reuse geth with as little change as possible so that mechanism belongs to geth, policy belongs to QuarkChain.** -The overall approach is to hang QKC semantics onto geth's own execution stack, isolated behind a **nullable profile**: +- Reuse geth's mature state and virtual-machine machinery: the trie and database layers, the mutable state database with its journal and snapshots, the interpreter, opcodes, gas accounting and the contract-call lifecycle are reused unchanged. +- QuarkChain code adds only what geth cannot express: the account shape, full-shard-key semantics, token-aware balances, contract-address derivation, cross-shard messages, the QuarkChain precompiles, historical fork rules, and block settlement. +- Those differences are made in the geth tree rather than in a private copy of it, so upstream fixes arrive by rebase rather than by hand-porting and no mechanism ever gets a second implementation that can drift out of step with the first. +**That split creates four layers with clear responsibilities and boundaries: mechanism at the bottom and policy at the top.** + +```text + block processing and validation ordering, cursor traversal, rewards, comparison + ▼ +transaction and cross-shard execution validation, message application, deposits + ▼ + QuarkChain state and VM profile account shape, token balances, derivation + ▼ + geth trie / database / state / EVM state mutation, rollback, commitment, opcodes ``` + +- Each layer calls only the one beneath it. +- The top two never reach for storage on their own — everything they need arrives through the `ExecutionContext` described next. +- The bottom two never decide policy: not the ordering, not what counts as a valid result. + +## Architecture + +Where those principles land in the tree: + +```text +qkc/core ← validation / ApplyTransaction / ApplyXShardDeposit / + cursor traversal / Process / ValidateBlockResult, + over the ExecutionContext / XShardSource inputs +qkc/state/evmstate.go ← EvmState: pyquarkchain's State — block context, + receipts/logs, snapshots — wrapped around geth's StateDB +core/vm ← QKCContext: two new files, plus one `if evm.QKC != nil` + branch each in evm.go / instructions.go / gas_table.go core/types, core/state ← six-field QKC account leaf + MNT balance layer - (the QKC side is concentrated in *_qkc.go) -core/vm ← QKCContext: two new files, qkc.go / contracts_qkc.go, - plus one `if evm.QKC != nil` branch each in - evm.go / instructions.go / gas_table.go -qkc/state/evmstate.go ← EvmState: the shape of pyquarkchain's State (block context, - receipts/logs, snapshot semantics) wrapped around geth's StateDB -qkc/core ← Validate / ApplyTransaction / ApplyXShardDeposit / - cursor traversal / Process / ValidateBlockResult + (the QKC side concentrated in *_qkc.go) ``` -When `evm.QKC == nil`, geth's behavior is byte-for-byte unchanged (the existing `core/vm` and `core/state` unit tests all pass, which is the premise this shape rests on); when non-nil, it takes over CREATE address derivation, the token dimension of balances, SELFDESTRUCT semantics, the precompile table, and re-entry at the message layer. For the cost, see Risks and open questions. +What holds it together is four seams: one facing callers, two carrying data in, one where policy attaches to mechanism. -**Execution results are not determined by the parent state alone.** Several inputs live outside the state root and must be passed in explicitly through an `ExecutionContext`, rather than having `qkc/core` reach back into node state: +- **The public surface** — one entry point per goal above, plus an atomic `ExecuteAndValidate`; `ValidateBlockResult` stays separate so the later replay task can call it directly. -- the node's **root tip at the time** (`ValidationRootTip`) — `__validate_tx`'s "target shard already has genesis" check and `_is_neighbor` read this, not the block's `hash_prev_root_block`. On replay you must feed in "the root block that confirmed this block", not the current root header -- **looking up minor headers by hash** (`MinorHeaderByHash`) — BLOCKHASH's 256 ancestors, plus the POSW window; the disallow map is computed on the fly over that chain by `SenderDisallowMap` -- **fetching the parent's meta by hash** (`MinorBlockMetaByHash`) — the cross-shard cursor resumes from there. Having it looked up rather than passed as a parameter prevents a caller from handing in a cursor that belongs to a different block +- **Cross-shard input** — cross-shard data arrives through an `XShardSource` interface, which needs root block bodies — expected to arrive with [PR #36](https://github.com/QuarkChain/goshard/pull/36). -Conceptual public entry points: `ValidateTransaction` / `ValidateTxForBlock` / `TxSender` / `IntrinsicGas` / `ApplyTransaction` / `ApplyXShardDeposit` / `RunOneXShardTx` / `RunCrossShardTxWithCursor` / `SenderDisallowMap` / `CoinbaseAmountMap` / `Process` / `ValidateBlockResult`, plus an atomic `ExecuteAndValidate`. `ValidateBlockResult` is a separate function so replay can call it directly. +- **Inbound context** — `ExecutionContext` carries the out-of-band inputs. Three are worth naming: + - the node's **root tip at the time** — `__validate_tx`'s "target shard already has genesis" check and the neighbor test read this, not the block's `hash_prev_root_block`. + - **minor headers by hash** — BLOCKHASH's 256 ancestors and the POSW window, over which the disallow map is computed on the fly. + - **the parent's meta by hash** — where the cross-shard cursor resumes. Looking it up rather than taking it as a parameter stops a caller handing in a cursor that belongs to a different block. -Cross-shard data is fed in through the `XShardSource` interface (`RootBlockByHeight` / `RootHeaderByHash` / `DepositsByMinorBlockHash`). It needs root block bodies; `qkc/types/rootblock.go` previously had only `RootBlockHeader` and has now been given a minimal implementation: header + minor header list + tracking data + `MinorHeaderMerkleRoot`, with goldens for both the serialization layout and the merkle root. The real implementation (database reads/writes) still belongs to the chain-layer task. +- **The profile boundary** — one nullable field on the EVM. + - nil, and geth's own `core/vm` and `core/state` unit tests are the check that nothing moved. + - attached, it takes over CREATE address derivation, the token dimension of balances, SELFDESTRUCT semantics, the precompile table, and re-entry at the message layer. ## Step-by-step implementation ### S0 — golden generator -`qkc/testdata/gen_exec_golden.py` exports three tiers of vectors — state level, message level, block level — from a pyquarkchain venv. All three tiers use pyquarkchain `75f8d7e166df0f5a2579ffe37ea7b4f5ba79db60` as the oracle: 17 state-level cases, 30 message-level cases, 11 block-level cases. The first case is fixed as a no-op genesis ALLOC whose post root must equal the two-network values in `minor_genesis_golden.json`, which calibrates the generator itself. +**Everything downstream is checked against pyquarkchain's own output, so the oracle is built before any of it.** A generator exports three tiers of vectors — state level, message level, block level — from a pyquarkchain venv, against a pinned oracle commit. What it has to emit is set by the [Hard-fork switches](#hard-fork-switches) table. Every switch that branches the execution layer needs vectors on both sides, and no single network reaches both — which is why every tier is emitted twice, for mainnet and for devnet. -**Acceptance**: the no-op genesis case, post root matching the golden for both networks; provenance (commit + module digest) written to disk alongside the vectors. +**Acceptance**: provenance (commit + module digest) is written to disk alongside the vectors. The post root of the no-op genesis ALLOC — the genesis allocation table, the addresses and starting balances a shard's genesis state is built from — must match the minor genesis values already committed for both networks. That match is what calibrates the generator. ### S1 — mutable state layer -Six-field account leaves, balances indexed by token id, `FullShardKey` in the leaf, and existence/deletion rules that are their own thing. These extend geth's `core/state`: `core/types/state_account*.go` carries the leaf, `core/state/statedb_qkc.go` / `state_object_qkc.go` carry the MNT balance layer and the QKC-side journal entries, and `qkc/state/evmstate.go` (~550 lines) wraps it into the shape of pyquarkchain's `State` — block context, two receipt lists, `commit` one block at a time written through to disk, and snapshots that roll back the context along with the state. +**The account model diverges below the point where any execution logic starts, so the state layer is rebuilt first and everything else sits on it.** Six-field account leaves, balances indexed by token id, the full shard key in the leaf, and existence and deletion rules that follow pyquarkchain's, not geth's — all extending geth's `core/state`, with the QKC side kept in separate files. Above them, a wrapper presents the result in the shape of pyquarkchain's `State`: block context, two receipt lists, one commit per block written through to disk, and snapshots that roll back the context along with the state. -**Acceptance**: ALLOC for both networks read back unchanged with matching commit; state-level golden compared per case on post root; snapshot/revert round trip. +**Acceptance**: the genesis ALLOC for both networks written, committed, and read back field for field unchanged, with the commit root matching S0's golden; state-level golden compared per case on post root; snapshot/revert round trip. ### S2 — the QKC profile in core/vm -`core/vm` gains a `QKCContext`; once `evm.SetQKCContext` has attached it, it takes over CREATE address derivation (`keccak(rlp([sender, fullShardKey, nonce]))[12:]`, falling back to the old `rlp([sender, nonce])` derivation when there is no shard key), BALANCE and transfers using the chain's default token, QKC SELFDESTRUCT semantics, the five QKC precompiles, and re-entry at the message layer. - -`qkcApplyMsg` is the core of this layer: it reproduces the ordering of `_apply_msg` — `del_account` at the end of the message, truncating the suicide list along with everything else on revert, and the deferred `token_id_queried` check. The suicide list hangs off `QKCContext` rather than off the state, with a separate `QKCAdoptSuicides` to merge the self-destructs marked by the second EVM (the general native token manager call) back into this transaction — pyquarkchain's list lives on the state and is shared naturally. +**Every divergence inside the EVM is collected behind one attachable profile, so that with nothing attached geth's own behavior is provably untouched.** `core/vm` gains a `QKCContext`, attached at `evm.QKC` — the profile; `qkcApplyMsg` is the core of this layer, reproducing the ordering of `_apply_msg`. **Acceptance**: with `evm.QKC == nil`, the existing `core/vm` and `core/state` unit tests all pass. ### S3 — intra-shard ApplyTransaction -`validate_transaction` implemented in full, including conversion for non-default gas tokens and the `version == 2` EIP155 branch. Devnet enables EIP155 at genesis, so this branch is exercised from the very first golden case onwards. +**This step is admission: everything that decides whether a transaction may enter a block at all, before a single opcode runs.** `validate_transaction` implemented in full over the default token, including the `version == 2` EIP155 branch. -**Acceptance**: transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase. +**Acceptance**: transfer cases (nonce, balance, gas, block cap, network id, branch, each v2 failure) compared on root + receipt + `gas_used` + coinbase; a non-default-token transaction hits the sentinel and abandons the block. ### S4 — EVM integration (CREATE / CALL) -Assemble `BlockContext` / `TxContext` / message and hook up the S2 profile, feeding in `full_shard_key` for the address derivation to use. Several orderings differ from geth: nonce increment, selfdestruct refunds, and account deletion. +**With admission in place, the transaction is handed to the EVM and the S2 profile starts doing real work.** Assemble `BlockContext` / `TxContext` / message and hook up that profile, feeding in `full_shard_key` for the address derivation to use. **Acceptance**: contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom). ### S5 — cross-shard source side -The `is_cross_shard` branch: debit, produce the deposit, compute the address on the source shard for cross-shard deployments, and burn all gas on POSW failure; `refund_rate` and `gas_token_id` travel with the deposit, and the target side uses them to refund and to burn proportionally. Charging and refunding the 9000 is gated by `ENABLE_EVM_TIMESTAMP`. +**A cross-shard transaction does half its work on the shard it starts from: it debits and emits.** The `is_cross_shard` branch: debit, produce the deposit, compute the address on the source shard for cross-shard deployments, and burn all gas on POSW failure. **Acceptance**: compared on root + receipt + `gas_used` + the produced deposit **field by field**. ### S6 — cross-shard target side -`RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. +**The other half runs on the receiving shard, one deposit at a time.** `RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. **Acceptance**: must include three items — post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP` ±1 boundary. ### S7 — Process(block) and block-level settlement -The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token. +**Everything above gets lifted to a whole block: which deposits it may consume, in what order they run, and how the block settles.** The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token. -**Acceptance**: block-level golden — the genesis root derived from ALLOC alone, then the seven result items compared block by block, plus the deposits consumed/produced and account read-back; and the cut-and-resume of a single root block's deposit list spanning two minor blocks. +**Acceptance**: block-level golden — the genesis root derived from the genesis ALLOC alone, then the seven result items compared block by block, plus the deposits consumed/produced and account read-back; and the cut-and-resume of a single root block's deposit list spanning two minor blocks. ### S8 — ported tests -Layer 2 of the verification plan, as a step of its own: port the behavior checklist from `test_shard_state.py` into `qkc/core/shardstate_port_test.go`. It requires the whole path to be finished but depends on nothing from the chain layer — so it is the last step this plan owns. +**Golden vectors only cover what the generator was told to emit, so the last step brings in a checklist written by people who already knew where the corners were.** Layer 2 of the verification plan, as a step of its own: port the behavior checklist from pyquarkchain's shard state tests (`test_shard_state.py`). It needs the whole path finished but depends on nothing from the chain layer — so it is the last step this design owns. **Acceptance**: the ported checklist passes; every new case is mutation-tested — revert the corresponding implementation to the wrong version and the case must turn red. ## Hard-fork switches +The question that matters is not "when did this go live" but "which branch does the execution layer take, and which network exercises it". + | Switch | mainnet | devnet | Where it applies | |---|---|---|---| -| genesis | 2019-04-30 | same | Start of the replayable range | | `ENABLE_TX_TIMESTAMP` + `TX_WHITELIST_SENDERS` | 2019-06-29 | **0** | Before this, only whitelisted addresses could send transactions; devnet has no such phase | -| `ENABLE_EVM_TIMESTAMP` | 2019-09-27 | **0** | **Six places**: three differences in cross-shard gas settlement, `__validate_tx` forbidding contract transactions, whether cross-shard receiving takes the pre-EVM fixed-amount path or the EVM path, and enabling QKC precompiles 01–03. **It splits the mainnet window in two, so both sides need golden cases**; devnet is post-EVM throughout | -| shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same | Whether the disallow map is non-empty; without it these seven chains have a **zero replayable range** | +| `ENABLE_EVM_TIMESTAMP` | 2019-09-27 | **0** | **Five places in scope**: three differences in cross-shard gas settlement, contract transactions forbidden before it, and whether cross-shard receiving takes the pre-EVM fixed-amount path or the EVM path. **It splits mainnet in two, so both sides need golden cases**; devnet is post-EVM throughout | +| shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same | Whether the disallow map is non-empty — the one switch whose absence is silent | | `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same | The basis for determining starting gas on the target side | -| `configure_special_contract_ts` | **per precompile** | same | Gated by a strict `>` (`messages.py:672`), unlike the `<` used elsewhere — easy to get backwards. The gate on system contract **deployment** is the other way round and non-strict (it rejects when `block_timestamp < enable_ts`) | -| `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | The non-reserved token auction contract (chain 0 only) **and MNT precompiles `04/05`**. Those two precompiles listen to this switch alone (`env.py:63-76`); taking the min of the two switches is wrong | -| `ENABLE_GENERAL_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | Only determines when the general native token manager contract may be deployed; whether `pay_native_token_as_gas` takes effect depends on whether that address has code, with no separate time gate | -| `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction`. Devnet needs it from block 1; mainnet only has it after this point | - -Every chain-level switch on devnet is 0: post-EVM throughout, MNT permitted from genesis. The column earns its place for golden vectors (S0 emits both networks) and for marking which branches devnet needs correct from block 1 — `version == 2` is the obvious case. +| `configure_special_contract_ts` | **per precompile** | same | A strict `>`, unlike the `<` used elsewhere — easy to get backwards. Honored by the dependency precompiles, not by this layer | +| `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP` / `ENABLE_GENERAL_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | Out of scope. Marks where MNT becomes reachable and therefore where the sentinel starts firing | +| `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction` | -**The execution layer no longer has an upper bound on the replayable range.** +Every chain-level timestamp switch that gates this layer is 0 on devnet: post-EVM throughout, EIP155 from block 1. That is why the devnet column earns its place even though mainnet is the eventual consensus target — on mainnet the `version == 2` branch does not appear until 2021-09-14, long after the sentinel has taken over, so **devnet is the only cheap way to exercise it**. S0 emits both networks for that reason. ## Verification plan -Three layers, cheapest to most expensive. +Two layers in scope, cheapest first, plus the one that finally settles it. + +1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests; this is what the per-step "Acceptance" above actually runs, across all three tiers. +2. **Porting pyquarkchain's shard state tests** — S8: `test_shard_state.py` is a ready-made behavior checklist. Every ported case is mutation-tested — revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. +3. **Historical replay** — out of scope here, landing with the chain-layer task. Worth naming for what it cannot do: replay only runs canonical blocks, so it only ever verifies the "what should pass does pass" half. Rules that take effect on the rejection path depend on the two layers above no matter how much history gets replayed. + +## Dependencies -1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests; this is what the per-step "Acceptance" above actually runs. Three tiers: 17 state-level cases (`qkc/state/statedb_test.go`), 30 message-level cases (`qkc/core/message_golden_test.go`), 11 block-level cases (`qkc/core/block_golden_test.go`). -2. **Porting pyquarkchain's shard state tests** — S8, a step of its own: `test_shard_state.py` is a ready-made behavior checklist, landing as `qkc/core/shardstate_port_test.go`. Every new case has been mutation-tested: revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. -3. **Historical replay (final acceptance)** — falls outside this plan, landing with the chain-layer replay task: join up with the paused replay task, replay real minor blocks in order and compare every committed value of `ValidateBlockResult` block by block. +This design builds none of the following, and a change in any of them lands directly on it. -Replay only runs canonical blocks, so it only verifies the "what should pass does pass" half; rules that only take effect on the rejection path still depend on the first two layers for coverage. +- **The QuarkChain account layer** (`feature/mnt-core-types`, `feature/mnt-state`) — the six-field account leaf and the token-indexed balance layer. Both branches are still being rebased onto upstream geth, and two details in them are load-bearing: the bookkeeping that records whether a token has an entry in an account's balance map, which decides whether a zero-valued entry survives a revert and a read-back; and the threshold at which an account crosses from list-encoded balances to trie-encoded ones. Either one moving changes state roots, so the state-level golden vectors are the tripwire. +- **The three QuarkChain precompiles at `0x…514b430001/02/03`** — current native token id, native token transfer, and system contract deployment. Assumed to be available and gated by the EVM-enable switch under a strict `>`, the opposite of the comparison used elsewhere. +- **The root block body** ([PR #36](https://github.com/QuarkChain/goshard/pull/36), open) — `RootBlock` carrying the minor header list, the tracking data, and the merkle root computed over those headers. `XShardSource` reads it to find which cross-shard lists a block may consume, and the merkle root has to match pyquarkchain byte for byte. This design takes the type as given and adds nothing to it. +- **The pyquarkchain oracle** — every golden vector comes from a pinned pyquarkchain commit, so the generator has to stay reproducible from a clean tree. Re-pinning it means regenerating and re-diffing all three tiers. -## Risks and open questions +## Risks and considerations -Ordered by severity: unresolved divergences first, implemented boundaries and already-fixed defects last. +Ordered by severity: unresolved divergences first, deliberate boundaries last. -- **Legacy fork behavior in geth v1.17**: 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners. The static layer — opcode table, constant gas, precompile pricing and set — turned up nothing; the dynamic layer — SSTORE's four Petersburg tiers, RETURNDATACOPY's out-of-bounds check, CREATE's handling of init code failure — has not been gone through case by case. -- **Version compatibility of `CrossShardTransactionList`**: pyquarkchain has three versions with automatic upgrade, while `qkc/types` currently only recognizes the current version 1 and errors out on anything else. **This code may change along with [PR #47](https://github.com/QuarkChain/goshard/pull/47).** -- **Token trie boundary**: when an account has more than 16 non-zero token balances (`TokenTrieThreshold`), pyquarkchain switches the balance to `b"\x01" + secure trie root`; the Go side gives up on the whole block with `ErrUnsupportedNativeToken`. -- **pathdb not adapted**: pathdb mixes the slim and full formats in three places — **snapshot generation, state rollback, and account reads**. In geth the two are inter-derivable; once QKC changes the contents they no longer are. - - goshard runs hashdb only and the execution path never touches pathdb, so this does not affect the correctness of what this plan delivers. - - `triedb/pathdb`'s `TestDatabaseRollback` / `TestExecuteRollback` are red. - - Whether to enable pathdb later is a chain-layer decision and needs its own assessment at that point: all three places sit on read and rollback paths, so the failure mode is silently reading back a wrong account, rather than failing loudly the way the token trie boundary does. -- **CREATE2's negative gas (implemented as a boundary)**: pyquarkchain does not check the balance when charging CREATE2's per-word fee, and when memory happens not to need expanding, the frame keeps running with negative gas until `assert gas_remained >= 0` blows up on the spot. Such a transaction cannot be executed upstream at all, so it never appears in mainnet history and replay is unaffected. Rather than have goshard reproduce negative gas, it rejects outright. -- **Long-term cost of modifying files inside the geth tree**: there is no `qkc/vm` copy to maintain any more; the price is in-tree changes across `core/vm`, `core/state` and `core/types`, which will conflict when rebasing onto upstream. The mitigation is the shape itself: the QKC side lives in separate files (`*_qkc.go`) wherever possible, what gets inserted into existing files is only single-point branches like `if evm.QKC != nil`, and the nil path is guarded by geth's own unit tests. +- **Legacy fork behavior in geth v1.17**: 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners. A pass over the static layer — opcode table, constant gas, precompile pricing and set — turned up nothing; the dynamic layer — SSTORE's four Petersburg tiers, RETURNDATACOPY's out-of-bounds check, CREATE's handling of init code failure — has not been gone through case by case. +- **Deferring MNT is not free**: the MNT branches thread through `ApplyTransaction`, `validate_transaction` and the deposit path. Bringing them back later means reopening the same three functions and re-validating them against the same golden vectors, and reconstituting the pinned pyquarkchain oracle to generate the new cases. +- **pathdb not adapted**: pathdb mixes the slim and full account formats in three places — snapshot generation, state rollback, and account reads. The two formats are inter-derivable in stock geth, but stop being so once the QKC leaf changes the contents. goshard runs hashdb only and the execution path never touches pathdb, so this design is unaffected; enabling pathdb later needs its own assessment, because all three sit on read and rollback paths and so fail by silently returning a wrong account rather than loudly the way the profile boundary does. From b8eee127bbc8296305fc0862fff25a4d5935e808 Mon Sep 17 00:00:00 2001 From: syntrust Date: Mon, 24 Aug 2026 18:11:01 +0800 Subject: [PATCH 6/6] refine transaction execution design --- L1/tx-exec.md | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/L1/tx-exec.md b/L1/tx-exec.md index 66ed640..db7a11f 100644 --- a/L1/tx-exec.md +++ b/L1/tx-exec.md @@ -93,13 +93,13 @@ What holds it together is four seams: one facing callers, two carrying data in, **Everything downstream is checked against pyquarkchain's own output, so the oracle is built before any of it.** A generator exports three tiers of vectors — state level, message level, block level — from a pyquarkchain venv, against a pinned oracle commit. What it has to emit is set by the [Hard-fork switches](#hard-fork-switches) table. Every switch that branches the execution layer needs vectors on both sides, and no single network reaches both — which is why every tier is emitted twice, for mainnet and for devnet. -**Acceptance**: provenance (commit + module digest) is written to disk alongside the vectors. The post root of the no-op genesis ALLOC — the genesis allocation table, the addresses and starting balances a shard's genesis state is built from — must match the minor genesis values already committed for both networks. That match is what calibrates the generator. +**Acceptance**: provenance (commit + module digest) is written to disk alongside the vectors. The generator is calibrated by requiring the post-state root of the no-op genesis ALLOC to match the minor genesis values already committed for both networks. Here, ALLOC is the genesis allocation table containing the addresses and starting balances from which a shard's genesis state is built. ### S1 — mutable state layer -**The account model diverges below the point where any execution logic starts, so the state layer is rebuilt first and everything else sits on it.** Six-field account leaves, balances indexed by token id, the full shard key in the leaf, and existence and deletion rules that follow pyquarkchain's, not geth's — all extending geth's `core/state`, with the QKC side kept in separate files. Above them, a wrapper presents the result in the shape of pyquarkchain's `State`: block context, two receipt lists, one commit per block written through to disk, and snapshots that roll back the context along with the state. +**The account model diverges below the point where any execution logic starts, so the state layer is rebuilt first and everything else sits on it.** A wrapper presents the result in the shape of pyquarkchain's `State`: block context, two receipt lists, one commit per block written through to disk, and snapshots that roll back the context along with the state. -**Acceptance**: the genesis ALLOC for both networks written, committed, and read back field for field unchanged, with the commit root matching S0's golden; state-level golden compared per case on post root; snapshot/revert round trip. +**Acceptance**: the genesis ALLOC for both networks is written through the new account layer and read back field for field unchanged, with S0's golden root serving as the anchor; the post-state root is compared against the state-level golden for each case; snapshot/revert completes a round trip. ### S2 — the QKC profile in core/vm @@ -117,7 +117,7 @@ What holds it together is four seams: one facing callers, two carrying data in, **With admission in place, the transaction is handed to the EVM and the S2 profile starts doing real work.** Assemble `BlockContext` / `TxContext` / message and hook up that profile, feeding in `full_shard_key` for the address derivation to use. -**Acceptance**: contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom). +**Acceptance**: contract cases (deploy, call, revert, OOG, SELFDESTRUCT, CREATE2, nesting, logs/bloom), plus the six S0 vectors for the corners where a 2018 pyethereum fork and a 2026 geth can disagree — `sstore_legacy_pricing`, two `returndatacopy_*`, three `create_*`. ### S5 — cross-shard source side @@ -127,7 +127,7 @@ What holds it together is four seams: one facing callers, two carrying data in, ### S6 — cross-shard target side -**The other half runs on the receiving shard, one deposit at a time.** `RunOneXShardTx` is **two mutually exclusive paths**: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. +**The other half runs on the receiving shard, one deposit at a time.** `RunOneXShardTx` is **two mutually exclusive paths**, chosen by whether the block's timestamp has passed `ENABLE_EVM_TIMESTAMP`: pre-EVM credits the money directly to `to`, executes no code, and produces no deposit receipt; only post-EVM goes through `ApplyXShardDeposit`. **Acceptance**: must include three items — post-EVM failure leaving funds stranded, pre-EVM crediting an account with code directly, and the `ENABLE_EVM_TIMESTAMP` ±1 boundary. @@ -135,7 +135,7 @@ What holds it together is four seams: one facing callers, two carrying data in, **Everything above gets lifted to a whole block: which deposits it may consume, in what order they run, and how the block settles.** The three-level cursor traversal (root block height / minor block index / deposit index) and its skip rules; cross-shard receiving **must be ordered before regular transactions** (part of consensus); coinbase is the sum of the block reward and fees, split per token. -**Acceptance**: block-level golden — the genesis root derived from the genesis ALLOC alone, then the seven result items compared block by block, plus the deposits consumed/produced and account read-back; and the cut-and-resume of a single root block's deposit list spanning two minor blocks. +**Acceptance**: block-level golden — derive the genesis root, compare the seven result items block by block, check the deposits consumed and produced and the account read-back, and cut and resume a single root block's deposit list across two minor blocks. ### S8 — ported tests @@ -151,13 +151,13 @@ The question that matters is not "when did this go live" but "which branch does |---|---|---|---| | `ENABLE_TX_TIMESTAMP` + `TX_WHITELIST_SENDERS` | 2019-06-29 | **0** | Before this, only whitelisted addresses could send transactions; devnet has no such phase | | `ENABLE_EVM_TIMESTAMP` | 2019-09-27 | **0** | **Five places in scope**: three differences in cross-shard gas settlement, contract transactions forbidden before it, and whether cross-shard receiving takes the pre-EVM fixed-amount path or the EVM path. **It splits mainnet in two, so both sides need golden cases**; devnet is post-EVM throughout | -| shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same | Whether the disallow map is non-empty — the one switch whose absence is silent | -| `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same | The basis for determining starting gas on the target side | -| `configure_special_contract_ts` | **per precompile** | same | A strict `>`, unlike the `<` used elsewhere — easy to get backwards. Honored by the dependency precompiles, not by this layer | +| shard `POSW_CONFIG.ENABLE_TIMESTAMP` | chains 1–7 **in effect from genesis**, chain 0 off | same as mainnet | Whether the disallow map is non-empty — the one switch whose absence is silent | +| `XSHARD_GAS_DDOS_FIX_ROOT_HEIGHT` | root height 90000 | same as mainnet | The basis for determining starting gas on the target side | +| `configure_special_contract_ts` | **per precompile** | same as mainnet | A strict `>`, unlike the `<` used elsewhere — easy to get backwards. Honored by the dependency precompiles, not by this layer | | `ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP` / `ENABLE_GENERAL_NATIVE_TOKEN_TIMESTAMP` | **2020-05-01** | **0** | Out of scope. Marks where MNT becomes reachable and therefore where the sentinel starts firing | | `ENABLE_EIP155_SIGNER_TIMESTAMP` | 2021-09-14 | **0** | The `version == 2` guard branch in `validate_transaction` | -Every chain-level timestamp switch that gates this layer is 0 on devnet: post-EVM throughout, EIP155 from block 1. That is why the devnet column earns its place even though mainnet is the eventual consensus target — on mainnet the `version == 2` branch does not appear until 2021-09-14, long after the sentinel has taken over, so **devnet is the only cheap way to exercise it**. S0 emits both networks for that reason. +**Devnet is the only network in scope on which the `version == 2` branch is reachable**: on mainnet, it was disabled until 2021-09-14. ## Verification plan @@ -165,7 +165,9 @@ Two layers in scope, cheapest first, plus the one that finally settles it. 1. **Golden vectors** — the output of S0, asserted field by field in table-driven Go tests; this is what the per-step "Acceptance" above actually runs, across all three tiers. 2. **Porting pyquarkchain's shard state tests** — S8: `test_shard_state.py` is a ready-made behavior checklist. Every ported case is mutation-tested — revert the corresponding implementation to the wrong version and the case must turn red, otherwise it is not testing anything. -3. **Historical replay** — out of scope here, landing with the chain-layer task. Worth naming for what it cannot do: replay only runs canonical blocks, so it only ever verifies the "what should pass does pass" half. Rules that take effect on the rejection path depend on the two layers above no matter how much history gets replayed. +3. **Historical replay (out of scope)** — this will land with the chain-layer task. + - **What it cannot do**: replay only runs canonical blocks, so it only ever verifies the "what should pass does pass" half. Rules that take effect on the rejection path depend on the two layers above no matter how much history gets replayed. + - **What only it can do**: catch a pre-Constantinople gas divergence between 2018-era pyethereum and 2026-era geth that nobody thought to name. Hand-enumerated vectors cannot reach a corner nobody named, and replay sees the bytecode the chain actually ran. ## Dependencies @@ -178,8 +180,5 @@ This design builds none of the following, and a change in any of them lands dire ## Risks and considerations -Ordered by severity: unresolved divergences first, deliberate boundaries last. - -- **Legacy fork behavior in geth v1.17**: 2018-era pyethereum and 2026-era geth may differ historically on pre-Constantinople corners. A pass over the static layer — opcode table, constant gas, precompile pricing and set — turned up nothing; the dynamic layer — SSTORE's four Petersburg tiers, RETURNDATACOPY's out-of-bounds check, CREATE's handling of init code failure — has not been gone through case by case. - **Deferring MNT is not free**: the MNT branches thread through `ApplyTransaction`, `validate_transaction` and the deposit path. Bringing them back later means reopening the same three functions and re-validating them against the same golden vectors, and reconstituting the pinned pyquarkchain oracle to generate the new cases. - **pathdb not adapted**: pathdb mixes the slim and full account formats in three places — snapshot generation, state rollback, and account reads. The two formats are inter-derivable in stock geth, but stop being so once the QKC leaf changes the contents. goshard runs hashdb only and the execution path never touches pathdb, so this design is unaffected; enabling pathdb later needs its own assessment, because all three sit on read and rollback paths and so fail by silently returning a wrong account rather than loudly the way the profile boundary does.